Spectrum

The key architectural decision was made upfront: use the GPU (Metal) for rendering but the CPU (Accelerate/vDSP) for FFT analysis. Apple's vDSP is heavily SIMD-optimised and processes a 2048-sample FFT in microseconds. Running FFT on the GPU via Metal compute shaders would add complexity with data-transfer overhead that negates any speed benefit at these buffer sizes. The GPU's strength is rendering thousands of coloured triangles per frame — exactly what we need for the visualisation.

With clear requirements established, the full architecture was planned: four modes (Bars, Curve, Circular, Spectrogram), all rendered through a single Metal pipeline using coloured triangles. The audio engine would capture via AVAudioEngine, process with vDSP, and publish normalised data that the renderer reads each frame.

The FFT pipeline was built using Accelerate's C-level vDSP functions rather than the newer Swift overlay, giving precise control over the processing chain:

Hanning window → reduces spectral leakage at buffer boundaries
vDSP_ctoz → packs real audio data into split-complex format
vDSP_fft_zrip → performs the real-to-complex FFT in-place
vDSP_zvmags → computes squared magnitudes (power spectrum)
vDSP_vdbcon → converts power to decibels

Rather than using separate Metal pipelines for each visualisation mode, all four modes share a single vertex/fragment shader pair. The CPU builds coloured vertex data (position + RGBA) each frame, uploads it to a pre-allocated 200K-vertex Metal buffer, and issues a single draw call. This keeps the shader code trivial while giving full flexibility on the CPU side.

The spectrogram mode was initially considered with a Metal texture approach, but the vertex-based approach (128×128 coloured quads = ~98K vertices) proved simpler and well within performance budgets.

The root cause was a struct alignment mismatch between Metal and Swift. The Metal shader used packed_float2 + packed_float4 (24 bytes per vertex, no padding), but Swift's SIMD4<Float> requires 16-byte alignment, making SpectrumVertex 32 bytes (with 8 bytes of padding between position and colour). Every vertex after the first was read from the wrong offset, causing garbled positions and colours.

In Spectrogram mode, the ~98K triangles with garbled positions created degenerate geometry covering the entire screen many times over, overwhelming the GPU with fragment shader work — causing the freeze.

Rather than trying to tap simulator UI from the command line (which is unreliable), the app was given launch argument support. ProcessInfo.processInfo.arguments is parsed in ContentView to accept -mode bars|curve|circular|spectrogram. This enables fully automated visual testing: install, launch with a specific mode, screenshot, verify — all from the command line.

Pure decision logic was extracted from AudioEngine and MetalRenderer into internal static methods that can be tested without audio hardware or a Metal GPU. This included logarithmic band mapping, exponential smoothing, peak tracking, gradient colour calculation, and heatmap colour mapping. The test target uses Xcode 16's PBXFileSystemSynchronizedRootGroup for automatic test file discovery.

Three issues were identified working together to suppress the display:

1. Audio session mode: .measurement mode disables the iPhone's automatic gain control, resulting in very quiet raw mic input. Switching to .default mode enables AGC for maximum sensitivity.

2. FFT normalisation: The power spectrum was normalised by 1/N², but for a one-sided spectrum (discarding the symmetric negative frequencies) the correct factor is 4/N² — a +6dB boost.

3. Auto-level too conservative: The minimum ceiling of -30dB was too high, and the 60dB display range too wide. Changed to ceiling [-60, 0] with a 40dB window for much more visual sensitivity.

The fundamental issue was that smoothing happened at the audio callback rate (~21fps), not the display rate (60fps). Bars would jump 21 times per second with dead frames in between — visible as micro-stuttering.

The fix was to decouple the animation from the audio pipeline entirely. AudioEngine now publishes raw normalised spectrum data. MetalRenderer maintains its own displaySpectrum array and interpolates toward the target values every frame at 60fps using asymmetric lerp — fast attack (0.35) for responsive rises, slow decay (0.12) for smooth, professional-quality falls. Peak tracking also moved to 60fps with a gentle 0.006/frame decay (~3.5 seconds for a full fall).

An FPS counter was added by tracking frame times in the renderer and displaying via a SwiftUI overlay that polls the shared coordinator every 0.5 seconds.

The architecture document was transformed from a flat reference into a three-part technical article. A magazine-style introduction sets the scene — what a spectrum analyser does, why the architecture splits into CPU audio and GPU rendering, and the key insight about decoupled rates.

The math blocks were completely restyled with colour-coded syntax: cyan for variables, purple for functions, green for numbers, orange for operators, blue headings for visual hierarchy, and proper HTML tables replacing monospaced columns. Inline SVG diagrams were added throughout — a Hanning window visualisation, a time-to-frequency domain transform, a linear-vs-logarithmic band comparison, an auto-leveling before/after, a CPU/GPU pipeline split, gradient colour ramps, a memory alignment bug diagram, and a full-colour spectrogram heatmap simulation.

Music playback required a second audio path through AVAudioEngine: AVAudioPlayerNode → AVAudioMixerNode (with FFT tap) → mainMixerNode. The mixer tap feeds the same FFT pipeline as the mic, so all four visualisation modes work identically with music.

The music library is loaded via MPMediaQuery.songs() with DRM filtering. A MusicBrowserView shows the filtered track list with artist sections, and a TransportBarView provides play/pause/stop controls with a swipe-up gesture to reveal the browser.

What followed was a long and humbling debugging session. The core problem was deceptively simple — switching between mic mode (.record audio session) and music mode (.playAndRecord) corrupted the audio hardware format, making inputNode.outputFormat(forBus: 0) return 0 Hz. But each attempted fix introduced new problems:

DRM detection spiral: The initial AVAudioFile(forReading:) header check let DRM tracks through. Trying AVURLAsset.hasProtectedContent rejected everything — including playable tracks. Reading 4096 audio frames as a validation check passed for DRM tracks too. Checking for silent audio data didn't help either. Attempting to remove tracks at play time when they failed caused all tracks to be removed because the engine wasn't starting properly. Each "fix" broke something else.

Engine corruption spiral: The repeated engine creation/destruction and audio session category changes left the phone's audio subsystem in a corrupted state — 0 Hz sample rate on every new engine. Multiple phone reboots were needed during the session. At one point the app was crashing on startup from an installTap call with an invalid format — an Objective-C NSException that Swift's do/catch cannot intercept.

The research phase was thorough — examining Apple's documentation, AudioKit's source code, Apple developer forums, WWDC sessions, and the AVAudioEngine sample projects. Four key insights emerged:

1. Never recreate the engine. A single AVAudioEngine instance should live for the entire app lifetime. Creating fresh engines mid-lifecycle causes 0 Hz formats and RPC timeouts. Apple's own sample code and AudioKit both use a single persistent engine.

2. Never change the audio session category. Use .playAndRecord with .defaultToSpeaker from the very start. The mic quality and AGC behaviour are identical to .record when both use .default mode. There is no quality penalty.

3. Swap taps, not engines. installTap and removeTap are documented as safe while the engine is running. Source switching just means removing one tap and installing another.

4. Use hasProtectedAsset for DRM. MPMediaItem.hasProtectedAsset (iOS 9.2+) reliably identifies DRM-protected tracks. Since 2024, all iTunes Store purchases are DRM-free.

The second round of research, focused on the specific "player started when in a disconnected state" crash, found the critical missing piece (AudioKit Issue #2527):

All nodes must be connected BEFORE engine.start(). Calling engine.connect() on a running engine silently stops it and permanently marks the playerNode as "disconnected" at the CoreAudio C++ level. No amount of reconnecting, restarting, or resetting recovers it. The only solution: connect everything before the engine ever starts.

Additionally, nodes must never be disconnected. Calling engine.disconnectNodeOutput() permanently breaks the node. Idle connected nodes pass silence at zero CPU cost — confirmed by AudioKit's own architecture.

For simulator testing: use .playback category (not .playAndRecord) and skip engine.inputNode access entirely. This lets the simulator play files through AVAudioPlayerNode without the mic hardware that it lacks.

A diagnostic loop was added to loadLibrary() that printed every track's attributes: hasProtectedAsset, isCloudItem, mediaType, assetURL file extension, and an AVAudioFile(forReading:) readability check. Running on a real device with a mixed library (purchased, imported, Apple Music subscription) revealed the pattern immediately.

Five versions of a-ha's "Take On Me" told the whole story:

Playable: url=item.m4a — a 2007 iTunes Store purchase, protected=false, cloud=false, AVAudioFile=YES
Unplayable: url=item.movpkg — Apple Music subscription versions (2015–2022), protected=false, cloud=false, AVAudioFile=NO (error 2003334207)
No URL: url=nil — cloud-only with cloud=true, already filtered

The .movpkg extension is the key. These are Apple Music cached streaming packages — downloaded for offline playback but stored in a container format that AVAudioFile (and therefore AVAudioPlayerNode) cannot decode. They slip through both hasProtectedAsset and isCloudItem checks because from the system's perspective they are local and not DRM-protected in the traditional sense.

The problem is fundamental to how music works: commercial music typically follows a pink noise spectrum with energy rolling off at roughly -3dB per octave. Bass frequencies can be 30-40dB louder than treble. With a single auto-leveling window, the bass fills the display and the treble is invisible.

Two approaches were considered: a static linear tilt (fixed dB/octave boost) and per-octave auto-leveling (independent normalization per octave). The linear tilt was tried first as the simpler option — but even at 4.5dB/octave, then 6.5, then 9dB/octave, the upper frequencies remained stubbornly quiet. A linear correction simply couldn't keep up with the accelerating rolloff.

The breakthrough came when the user asked: "Do we need an exponential tilt?" Instead of a constant dB/octave boost, the correction uses rate × octavespower — accelerating the boost the further you get from the reference frequency. This matches the way music's spectral energy actually falls off.

Three features were designed together: @AppStorage persistence for all UI toggles, autocorrelation-based pitch detection, and spectral flux onset detection for BPM.

The first pitch detection approach used the Wiener-Khinchin theorem: since the autocorrelation of a signal equals the inverse Fourier transform of its power spectral density, the existing FFT power spectrum could be reused — just one additional inverse FFT call with kFFTDirection_Inverse. Parabolic interpolation around the peak gave sub-Hz accuracy, and a median filter over 5 detections removed jitter. The frequency was converted to the nearest musical note via 12*log2(freq/440), with the remainder expressed as cents. In the simulator with a clean 440Hz test tone, it locked onto A4 perfectly.

Then it was deployed to a real phone. With voice input, the detected pitch was wildly unstable — jumping between octaves every frame. Device logs showed the problem clearly: squaring the power spectrum (which the Wiener-Khinchin approach requires) exaggerates harmonics relative to the fundamental. For a pure sine wave this is fine, but a human voice has strong overtones that compete with — and often exceed — the fundamental after squaring.

The fix was to switch to time-domain autocorrelation using vDSP_dotpr on the raw audio samples, bypassing the power spectrum entirely. This was fundamentally more reliable for complex signals because it preserves the natural amplitude relationships between harmonics. But a new problem appeared: noise at short lags (corresponding to high frequencies) produced false detections. The standard energy normalisation was biased because at short lags, nearly all samples overlap — giving artificially high correlation values.

The solution was Pearson normalisation: instead of dividing by the total signal energy, each lag is normalised by the geometric mean of the energies of just the overlapping segments at that lag. This eliminates the short-lag bias completely. Incremental energy updates (subtracting the sample that leaves the window, adding the one that enters) keep the per-lag normalisation at O(1) rather than O(N).

Even with correct normalisation, the threshold for "confident detection" needed iterative tuning — done by sending device logs back and forth. The acPeak value (peak autocorrelation after normalisation) was logged for every frame. Singing into the phone at various pitches, the logs showed: speech hovered around 0.3-0.5, clear singing reached 0.6-0.8, and noise floor was below 0.2. The threshold was walked up from 0.25 (too many false positives on speech) to 0.5 (still flickering) to 0.6 (better but caught breath sounds) to 0.75 (clean detections only on sustained notes). The search range was also narrowed from 2000Hz to 1000Hz to skip the problematic very-short-lag region entirely.

For BPM detection, frame-to-frame energy changes in the low-frequency bands (20-200Hz, where kick drums live) create an "onset strength" signal. Peaks above an adaptive threshold (mean + 1.5 sigma) are tracked as beat onsets. The median inter-onset interval gives the tempo, but it's only displayed when confidence is high (>50% of intervals cluster within 20% of the median).

The mode picker was changed from a segmented text control to icon buttons (the text was getting truncated with the new toggle buttons), and long-press tooltips were added to all toolbar buttons for discoverability.

This required a fundamental extension to the Metal pipeline. The existing four modes all shared a single 2D pass-through shader — no transformation matrix, no depth buffer, no normals. Surface mode needed all three. Rather than inflating the 2D vertex struct with unused fields for 100K+ vertices per frame, a second MTLRenderPipelineState was created with its own shader pair.

The 3D vertex carries position (SIMD3), surface normal (SIMD3), and colour (SIMD4) — 48 bytes per vertex. The vertex shader transforms by an MVP matrix, and the fragment shader applies directional lighting (N dot L with configurable ambient). A depth buffer (depth32Float) handles occlusion.

The surface mesh is built from a 40-row circular buffer of spectrum history (~2 seconds at 20fps). Each frame, the CPU generates ~30K vertices as a triangulated grid — X for frequency, Y for amplitude, Z for time — with face normals computed via cross product. The gradient colour scheme matches curve mode (blue through red by frequency position).

Camera positioning proved surprisingly tricky. The view needed to work across three different Metal view heights — full mic mode, music with just the transport bar, and music with the full browser open. A hard threshold caused a jarring jump during the browser animation. The solution was smooth interpolation between two camera positions based on aspect ratio, blending across the 0.55-0.75 range. Peak height also scales adaptively — taller in the full portrait view (1.2x) to fill the vertical space, shorter in the compact browser view (0.6x).

Three changes came out of this conversation, each addressing a different aspect of the app:

Surface depth 40 → 60 rows: The original 40-row buffer gave ~2 seconds of history at the ~20fps update rate. With 60 rows, the waterfall extends to ~3 seconds — a deeper, more dramatic perspective. The vertex count rises from ~30K to ~45K for the solid mesh (and ~91K total for Surface+ with ridgelines), but this is still well under the 200K vertex budget. The surface vertex buffer allocation was bumped from 70K to 100K vertices to accommodate. This increase actually revealed a latent bug: Surface+ mode had been silently failing to render because its ~91K vertex payload exceeded the 70K buffer, causing a guard clause to bail out before drawing. The fix was straightforward once identified.

Bluetooth audio routing: The audio session was configured with .defaultToSpeaker to prevent music from playing through the tiny earpiece (the default for .playAndRecord). But this option also overrides Bluetooth routing. Adding .allowBluetooth and .allowBluetoothA2DP to the session options tells iOS to respect connected Bluetooth headphones for output while keeping the speaker as the fallback when no Bluetooth device is connected.

Tuning and BPM on surface modes: The overlay conditions simply needed the surface and surfaceLines cases added alongside bars and curve. The overlays render as SwiftUI text positioned over the Metal view, so they work identically regardless of the underlying rendering pipeline.

This began as a threshold tuning exercise but turned into a complete algorithmic rewrite — a journey through three fundamentally different approaches to BPM detection, with each failure revealing why the previous approach was structurally wrong for syncopated music.

Approach 1: Inter-onset intervals with median (original). The existing approach detected onset peaks in the spectral flux, recorded their timestamps, took the median interval between consecutive onsets, and converted to BPM. For straight 4/4 beats (like the bundled 120 BPM test tone), this worked perfectly. For "Missing" — a drum & bass track where kicks land on off-beat subdivisions — the intervals scattered wildly: 0.40s, 0.50s, 0.69s, 0.71s, 0.80s, 1.20s. The median landed on ~0.6s (100 BPM), not the true 0.48s (123 BPM).

Approach 2: Multi-candidate period scoring. Instead of trusting the median, this approach scanned every BPM from 60-200 and scored each by how many intervals aligned to integer multiples of that candidate's beat period. The theory: a 123 BPM candidate should match intervals at 0.48s (1 beat), 0.96s (2 beats), etc. In practice, 150 BPM scored higher because its period (0.40s) matched more intervals than 123 BPM (0.48s). The syncopated pattern genuinely has more energy at the subdivision level than the beat level — so the wrong tempo was objectively the better fit for this method.

Approach 3: Autocorrelation of the onset strength signal (Scheirer/Ellis). Research into the literature revealed the fundamental flaw: onset-based methods fail because they discard the continuous character of the rhythmic signal. The solution was to autocorrelate the spectral flux signal itself rather than individual onset timestamps. Even when kicks are syncopated, the overall rhythmic pattern repeats at the beat period — autocorrelation finds this periodicity directly.

Implementation required solving several sub-problems: the native audio callback rate of ~10fps gave far too coarse lag resolution (each lag step = ~15 BPM), so the flux signal is upsampled 4x via linear interpolation before autocorrelation. Parabolic interpolation around the autocorrelation peak gives sub-lag accuracy. And temporal smoothing (5-estimate median with 4/5 consensus) prevents brief harmonic-switching blips from flipping the display.

An overnight deep-research session produced a comprehensive 650-line report covering five open-source implementations, eight algorithm families, and specific parameter recommendations. The report identified the root causes of each problem and a priority-ordered implementation plan.

The key insight: three preprocessing steps transform the onset signal from unreliable to robust:

1. Log compression (log(1 + 10 * flux)): Raw spectral flux has enormous dynamic range — a loud kick produces flux 100x larger than a quiet hi-hat. Without compression, the autocorrelation is dominated by a few loud events. Log compression brings quiet onsets to comparable magnitude, making the full rhythmic pattern visible. This is what librosa's onset_strength does via power-to-dB conversion, and it proved to be the single most impactful change.

2. Detrending (subtract 2-second local mean): Slow energy variations — a crescendo, a quiet intro with harmonic content — create a DC component that produces positive autocorrelation at all lags. Subtracting the local mean removes these slow variations while preserving the fast rhythmic peaks. This eliminated the false 82-85 BPM readings during quiet intros.

3. Higher onset rate (~43fps via separate 1024-point FFTs): The original ~10fps rate gave only ~15 BPM resolution per lag step — so coarse that a Gaussian perceptual weighting distorted the peak. A separate 1024-point FFT runs at hop=1024 specifically for onset detection, giving 4 onset samples per audio callback and ~1.5 BPM resolution per lag. This completely eliminated the need for the 4x upsampling hack.

Additional improvements included harmonic checking at half-lag with a range constraint (resolves octave ambiguity without creating new errors), broadband spectral flux (all frequencies, not just bass — captures hi-hats and snares that carry the rhythmic pulse), reduced temporal smoothing (3 estimates instead of 5 for faster lock-on), and tempo-change detection that flushes the entire buffer when the autocorrelation consistently diverges from the locked tempo.

A suite of realistic test WAV files was generated: 128 BPM house (4-on-the-floor with offbeat hi-hats), 174 BPM D&B (syncopated kicks with steady hi-hats), a 123 BPM track with a 5-second quiet intro before the beat drops, and a 10-second extract from a real 85 BPM track. An -autoplay launch argument was added for automated device testing — it searches the music library by title substring, starts playback from 1/3 through the track (to skip intros), and enables fully automated BPM testing across the entire library.