Spectrum

Introduction

Every sound you hear is a cocktail of frequencies — bass, midrange, treble — mixed together into a single pressure wave. A spectrum analyser takes that wave apart, revealing its ingredients in real time. This article explains how Spectrum does it: capturing audio from the iPhone's microphone, decomposing it with a Fast Fourier Transform, and rendering the result at 60 frames per second using the GPU.

The architecture splits neatly into two halves. The audio pipeline runs on the CPU, using Apple's Accelerate framework to perform signal processing with SIMD instructions — processing four or eight numbers simultaneously on every clock cycle. The rendering pipeline runs on the GPU via Metal, turning frequency data into coloured triangles that the hardware rasterises in parallel across thousands of shader cores.

Between these two halves sits a crucial design decision: decoupled rates. Audio arrives at ~21 frames per second (every 2048 samples at 44.1kHz). The display runs at 60 frames per second. Rather than coupling them — which would mean either wasting GPU frames or stuttering the animation — the renderer maintains its own smoothed state and interpolates toward the latest audio data on every frame. This is what gives the display its fluid, professional feel.

What follows is a complete walkthrough of both pipelines, from microphone to pixel, with the mathematics behind each stage and how those operations map to the hardware.

Part 1: The Audio Pipeline

The journey from microphone to frequency data passes through eight stages, each performing a well-defined mathematical operation. What makes this fast isn't clever algorithms alone — it's that Apple's Accelerate framework maps each operation onto SIMD (Single Instruction, Multiple Data) instructions. ARM NEON, the SIMD engine on Apple Silicon, processes 4 floats simultaneously in a single clock cycle. The entire pipeline completes in about 10 microseconds — 0.02% of the available time budget.

1. Windowing — Reducing Spectral Leakage

The microphone delivers a continuous audio stream, but the FFT operates on a finite chunk (2048 samples). Abruptly cutting the signal at chunk boundaries creates artificial high-frequency artefacts called spectral leakage. A Hanning window tapers the signal to zero at both edges, eliminating this.

2. Packing — Real Data to Split Complex Format

The vDSP FFT requires input in split complex format: separate arrays for real and imaginary parts. Since our audio is purely real (no imaginary component), we pack it by treating consecutive pairs of real samples as (real, imaginary) pairs.

3. FFT — Time Domain to Frequency Domain

The Discrete Fourier Transform decomposes a time-domain signal into its constituent frequencies. For N samples, the DFT is:

Naively, this is O(N²) — each of the N output bins sums over all N input samples. The Fast Fourier Transform (Cooley-Tukey algorithm) exploits symmetries in the twiddle factors e^(−j2πkn/N) to reduce this to O(N log N). For N=2048, that's ~22,000 operations instead of ~4.2 million.

4. Squared Magnitudes — Complex to Power

The FFT output is 1024 complex values. We need the power (squared magnitude) at each frequency — this tells us how much energy is present at that frequency.

5. Power to Decibels — Logarithmic Scaling

Human hearing perceives loudness logarithmically — a 10× increase in power sounds roughly "twice as loud". Converting to decibels maps this perception to a linear scale.

Values are floored to 10⁻²⁰ before the log to avoid −∞ from silent bins.

6. Logarithmic Band Mapping — Perceptual Frequency Grouping

The 1024 FFT bins are linearly spaced in frequency (each bin = sampleRate/N ≈ 21.5 Hz). But human pitch perception is logarithmic — the jump from 100 Hz to 200 Hz (one octave) sounds the same as 1000 Hz to 2000 Hz (also one octave). We map bins to 128 logarithmically-spaced bands.

This is implemented as a scalar loop since each band maps to a different number of bins — not a uniform SIMD operation. At only 128 iterations with simple averaging, it completes in microseconds.

7. Auto-Leveling & Normalisation

Rather than mapping a fixed -80 to 0 dB range (which leaves quiet signals invisible), the display window adapts to the current signal level. The ceiling tracks the peak dB value with instant rise and slow decay.

8. 60fps Smoothing & Peak Tracking (in Renderer)

The audio engine publishes raw normalised data at ~21fps. The Metal renderer applies its own smoothing at 60fps for silky-smooth animation, completely decoupled from the audio update rate.

Asymmetric exponential smoothing — fast attack, slow decay:

Peak tracking at 60fps — instant rise, gentle fall:

Performance Summary

The entire audio pipeline uses roughly 0.02% of the available CPU budget per audio callback — demonstrating why vDSP's SIMD approach is more than adequate and GPU compute would add complexity for no benefit at this scale.

Part 3: System Architecture

The view hierarchy, data flow, rendering pipeline, and testing infrastructure that connect the audio and rendering halves into a working app.

Part 2: The Rendering Pipeline

At this point we have 128 normalised frequency values arriving ~21 times per second. The rendering pipeline's job is to turn these numbers into a beautiful, fluid visualisation at 60 frames per second — smoothing over the gaps between audio updates and giving the user four distinct ways to see their sound.

Why Metal, Not SwiftUI

SwiftUI is excellent for UI controls (our mode picker and label overlays use it), but for rendering thousands of animated shapes at 60fps it becomes a bottleneck. Each SwiftUI view carries overhead — identity tracking, diffing, layout computation. Drawing 128 animated bars means 128 view updates per frame through SwiftUI's reconciliation engine.

Metal bypasses all of this. We write vertex data directly into GPU memory and issue a single draw call. The GPU's massively parallel architecture rasterises all triangles simultaneously. For our spectrogram mode with ~98,000 triangles, this is the difference between fluid animation and a slideshow.

The Single-Pipeline Architecture

All four visualisation modes share one Metal render pipeline — a single vertex shader and a single fragment shader. Rather than writing mode-specific GPU code, we do the mode-specific work on the CPU: building an array of coloured triangles that the GPU simply renders. This is the key architectural choice.

This works because our bottleneck is fill rate (pixels), not vertex throughput. Modern mobile GPUs can handle millions of vertices per frame — our maximum of ~100K is trivial. The simplicity of having one shader pair means less GPU state switching, fewer pipeline objects, and dramatically simpler code.

Vertex Layout and the Alignment Lesson

Each vertex is a Swift struct with a 2D position and an RGBA colour:

The Metal shader must use matching types. Using packed_float4 (no alignment padding, 24-byte stride) caused a critical bug — every vertex after the first was read from the wrong offset, producing garbled geometry. The fix was to use non-packed float2/float4 in Metal, giving the same 32-byte stride with identical padding.

How Each Mode Builds Geometry

Bars Mode

The classic equaliser. Each of the 128 frequency bands becomes a vertical rectangle (quad), rendered as two triangles. Colour is determined by frequency position through a five-stop gradient ramp.

Curve Mode

A smooth filled area under the frequency curve, with a bright outline traced along the top edge.

Circular Mode

Frequency bands radiate outward from a central point, creating a pulsing ring. Each band is a tapered bar at a specific angle around the circle.

Spectrogram Mode

A scrolling heatmap where the x-axis is frequency, the y-axis is time, and colour represents intensity. This is the most vertex-heavy mode.

Waveform (All Modes)

The time-domain audio signal rendered as a cyan trace below the spectrum area. Drawn as a series of thin quads along the waveform curve, similar to the curve outline technique.

60fps Asymmetric Smoothing

A critical insight: the audio engine delivers new spectrum data at ~21fps (every 2048 samples at 44.1kHz), but the display runs at 60fps. If we used the audio data directly, bars would jump 21 times per second with dead frames in between — visible as micro-stuttering.

The renderer maintains its own displaySpectrum array and interpolates toward the target on every frame:

Why asymmetric? Equal attack and decay feels "mushy" — sounds seem to linger too long. Fast attack captures transients like drum hits instantly, while slow decay creates a smooth, natural fade that reads as "energy dissipating" rather than "signal dropping". The ~3:1 ratio between attack and decay is common in professional audio metering.

Peak tracking also runs at 60fps — peaks snap up instantly but fall at 0.006/frame, taking ~3.5 seconds to fully decay. This creates the characteristic "falling dot" effect seen in professional analysers.

The Draw Call

After building all vertices (spectrum + waveform), the renderer uploads them to the GPU and issues one draw call:

Rendering Performance

The GPU handles the heavy lifting — rasterising triangles, interpolating colours, and blending with alpha transparency. The CPU's job is just to decide what to draw and where. This CPU-for-logic, GPU-for-pixels split is the same principle behind every real-time graphics application, from games to medical imaging.