Spectrum

The key architectural decision was made upfront: use the GPU (Metal) for rendering but the CPU (Accelerate/vDSP) for FFT analysis. Apple's vDSP is heavily SIMD-optimised and processes a 2048-sample FFT in microseconds. Running FFT on the GPU via Metal compute shaders would add complexity with data-transfer overhead that negates any speed benefit at these buffer sizes. The GPU's strength is rendering thousands of coloured triangles per frame — exactly what we need for the visualisation.

With clear requirements established, the full architecture was planned: four modes (Bars, Curve, Circular, Spectrogram), all rendered through a single Metal pipeline using coloured triangles. The audio engine would capture via AVAudioEngine, process with vDSP, and publish normalised data that the renderer reads each frame.

The FFT pipeline was built using Accelerate's C-level vDSP functions rather than the newer Swift overlay, giving precise control over the processing chain:

Hanning window → reduces spectral leakage at buffer boundaries
vDSP_ctoz → packs real audio data into split-complex format
vDSP_fft_zrip → performs the real-to-complex FFT in-place
vDSP_zvmags → computes squared magnitudes (power spectrum)
vDSP_vdbcon → converts power to decibels

Rather than using separate Metal pipelines for each visualisation mode, all four modes share a single vertex/fragment shader pair. The CPU builds coloured vertex data (position + RGBA) each frame, uploads it to a pre-allocated 200K-vertex Metal buffer, and issues a single draw call. This keeps the shader code trivial while giving full flexibility on the CPU side.

The spectrogram mode was initially considered with a Metal texture approach, but the vertex-based approach (128×128 coloured quads = ~98K vertices) proved simpler and well within performance budgets.

The root cause was a struct alignment mismatch between Metal and Swift. The Metal shader used packed_float2 + packed_float4 (24 bytes per vertex, no padding), but Swift's SIMD4<Float> requires 16-byte alignment, making SpectrumVertex 32 bytes (with 8 bytes of padding between position and colour). Every vertex after the first was read from the wrong offset, causing garbled positions and colours.

In Spectrogram mode, the ~98K triangles with garbled positions created degenerate geometry covering the entire screen many times over, overwhelming the GPU with fragment shader work — causing the freeze.

Rather than trying to tap simulator UI from the command line (which is unreliable), the app was given launch argument support. ProcessInfo.processInfo.arguments is parsed in ContentView to accept -mode bars|curve|circular|spectrogram. This enables fully automated visual testing: install, launch with a specific mode, screenshot, verify — all from the command line.

Pure decision logic was extracted from AudioEngine and MetalRenderer into internal static methods that can be tested without audio hardware or a Metal GPU. This included logarithmic band mapping, exponential smoothing, peak tracking, gradient colour calculation, and heatmap colour mapping. The test target uses Xcode 16's PBXFileSystemSynchronizedRootGroup for automatic test file discovery.

Three issues were identified working together to suppress the display:

1. Audio session mode: .measurement mode disables the iPhone's automatic gain control, resulting in very quiet raw mic input. Switching to .default mode enables AGC for maximum sensitivity.

2. FFT normalisation: The power spectrum was normalised by 1/N², but for a one-sided spectrum (discarding the symmetric negative frequencies) the correct factor is 4/N² — a +6dB boost.

3. Auto-level too conservative: The minimum ceiling of -30dB was too high, and the 60dB display range too wide. Changed to ceiling [-60, 0] with a 40dB window for much more visual sensitivity.

The fundamental issue was that smoothing happened at the audio callback rate (~21fps), not the display rate (60fps). Bars would jump 21 times per second with dead frames in between — visible as micro-stuttering.

The fix was to decouple the animation from the audio pipeline entirely. AudioEngine now publishes raw normalised spectrum data. MetalRenderer maintains its own displaySpectrum array and interpolates toward the target values every frame at 60fps using asymmetric lerp — fast attack (0.35) for responsive rises, slow decay (0.12) for smooth, professional-quality falls. Peak tracking also moved to 60fps with a gentle 0.006/frame decay (~3.5 seconds for a full fall).

An FPS counter was added by tracking frame times in the renderer and displaying via a SwiftUI overlay that polls the shared coordinator every 0.5 seconds.

The architecture document was transformed from a flat reference into a three-part technical article. A magazine-style introduction sets the scene — what a spectrum analyser does, why the architecture splits into CPU audio and GPU rendering, and the key insight about decoupled rates.

The math blocks were completely restyled with colour-coded syntax: cyan for variables, purple for functions, green for numbers, orange for operators, blue headings for visual hierarchy, and proper HTML tables replacing monospaced columns. Inline SVG diagrams were added throughout — a Hanning window visualisation, a time-to-frequency domain transform, a linear-vs-logarithmic band comparison, an auto-leveling before/after, a CPU/GPU pipeline split, gradient colour ramps, a memory alignment bug diagram, and a full-colour spectrogram heatmap simulation.

Music playback required a second audio path through AVAudioEngine: AVAudioPlayerNode → AVAudioMixerNode (with FFT tap) → mainMixerNode. The mixer tap feeds the same FFT pipeline as the mic, so all four visualisation modes work identically with music.

The music library is loaded via MPMediaQuery.songs() with DRM filtering. A MusicBrowserView shows the filtered track list with artist sections, and a TransportBarView provides play/pause/stop controls with a swipe-up gesture to reveal the browser.

What followed was a long and humbling debugging session. The core problem was deceptively simple — switching between mic mode (.record audio session) and music mode (.playAndRecord) corrupted the audio hardware format, making inputNode.outputFormat(forBus: 0) return 0 Hz. But each attempted fix introduced new problems:

DRM detection spiral: The initial AVAudioFile(forReading:) header check let DRM tracks through. Trying AVURLAsset.hasProtectedContent rejected everything — including playable tracks. Reading 4096 audio frames as a validation check passed for DRM tracks too. Checking for silent audio data didn't help either. Attempting to remove tracks at play time when they failed caused all tracks to be removed because the engine wasn't starting properly. Each "fix" broke something else.

Engine corruption spiral: The repeated engine creation/destruction and audio session category changes left the phone's audio subsystem in a corrupted state — 0 Hz sample rate on every new engine. Multiple phone reboots were needed during the session. At one point the app was crashing on startup from an installTap call with an invalid format — an Objective-C NSException that Swift's do/catch cannot intercept.

The research phase was thorough — examining Apple's documentation, AudioKit's source code, Apple developer forums, WWDC sessions, and the AVAudioEngine sample projects. Four key insights emerged:

1. Never recreate the engine. A single AVAudioEngine instance should live for the entire app lifetime. Creating fresh engines mid-lifecycle causes 0 Hz formats and RPC timeouts. Apple's own sample code and AudioKit both use a single persistent engine.

2. Never change the audio session category. Use .playAndRecord with .defaultToSpeaker from the very start. The mic quality and AGC behaviour are identical to .record when both use .default mode. There is no quality penalty.

3. Swap taps, not engines. installTap and removeTap are documented as safe while the engine is running. Source switching just means removing one tap and installing another.

4. Use hasProtectedAsset for DRM. MPMediaItem.hasProtectedAsset (iOS 9.2+) reliably identifies DRM-protected tracks. Since 2024, all iTunes Store purchases are DRM-free.

The second round of research, focused on the specific "player started when in a disconnected state" crash, found the critical missing piece (AudioKit Issue #2527):

All nodes must be connected BEFORE engine.start(). Calling engine.connect() on a running engine silently stops it and permanently marks the playerNode as "disconnected" at the CoreAudio C++ level. No amount of reconnecting, restarting, or resetting recovers it. The only solution: connect everything before the engine ever starts.

Additionally, nodes must never be disconnected. Calling engine.disconnectNodeOutput() permanently breaks the node. Idle connected nodes pass silence at zero CPU cost — confirmed by AudioKit's own architecture.

For simulator testing: use .playback category (not .playAndRecord) and skip engine.inputNode access entirely. This lets the simulator play files through AVAudioPlayerNode without the mic hardware that it lacks.

A diagnostic loop was added to loadLibrary() that printed every track's attributes: hasProtectedAsset, isCloudItem, mediaType, assetURL file extension, and an AVAudioFile(forReading:) readability check. Running on a real device with a mixed library (purchased, imported, Apple Music subscription) revealed the pattern immediately.

Five versions of a-ha's "Take On Me" told the whole story:

Playable: url=item.m4a — a 2007 iTunes Store purchase, protected=false, cloud=false, AVAudioFile=YES
Unplayable: url=item.movpkg — Apple Music subscription versions (2015–2022), protected=false, cloud=false, AVAudioFile=NO (error 2003334207)
No URL: url=nil — cloud-only with cloud=true, already filtered

The .movpkg extension is the key. These are Apple Music cached streaming packages — downloaded for offline playback but stored in a container format that AVAudioFile (and therefore AVAudioPlayerNode) cannot decode. They slip through both hasProtectedAsset and isCloudItem checks because from the system's perspective they are local and not DRM-protected in the traditional sense.

The problem is fundamental to how music works: commercial music typically follows a pink noise spectrum with energy rolling off at roughly -3dB per octave. Bass frequencies can be 30-40dB louder than treble. With a single auto-leveling window, the bass fills the display and the treble is invisible.

Two approaches were considered: a static linear tilt (fixed dB/octave boost) and per-octave auto-leveling (independent normalization per octave). The linear tilt was tried first as the simpler option — but even at 4.5dB/octave, then 6.5, then 9dB/octave, the upper frequencies remained stubbornly quiet. A linear correction simply couldn't keep up with the accelerating rolloff.

The breakthrough came when the user asked: "Do we need an exponential tilt?" Instead of a constant dB/octave boost, the correction uses rate × octavespower — accelerating the boost the further you get from the reference frequency. This matches the way music's spectral energy actually falls off.