Sound is a wave — a continuous oscillation of air pressure over time. It seems like exactly the wrong shape of data for a transformer. The trick is not to feed it the wave. Feed it a picture of the wave instead.
A raw audio file samples the amplitude of a sound wave thousands of times per second — typically 16,000 or 44,100 samples per second. Thirty seconds of speech at 16,000 samples/second gives you 480,000 individual data points.
Like raw image pixels, this is far too long for efficient attention. And like individual pixels, individual audio samples aren't very meaningful in isolation. The information you care about — phonemes, words, intonation, speaker identity — emerges from patterns across many samples, not from any single value.
The solution used by almost every modern audio transformer is to first convert the raw waveform into a different representation — one that captures the meaningful structure of sound in a form that's much easier to tokenize.
A spectrogram is a visual representation of audio. The horizontal axis is time. The vertical axis is frequency. The brightness at each point represents how much energy is present at that frequency at that moment in time. It turns a one-dimensional wave into a two-dimensional image.
A mel spectrogram is a spectrogram where the frequency axis is scaled to match how human hearing works — our ears are more sensitive to changes in lower frequencies than higher ones, so the mel scale compresses high frequencies and expands low ones. This makes the representation richer in the information that matters most for understanding speech and music.
Once you have a mel spectrogram, you have an image. And once you have an image, you already know what to do with it: slice it into patches and tokenize each patch. The audio problem becomes the image problem, which is already solved.
Raw audio waveform → Mel spectrogram (image of sound) → Patch slices → Embedding vectors → Transformer. Whisper, OpenAI's speech recognition model, works exactly this way. The transformer itself never sees audio — it sees a picture of the audio, tokenized into patches.
OpenAI's Whisper is trained on 680,000 hours of audio paired with transcripts. During training, it receives 30-second audio chunks converted to 80-channel mel spectrograms. Those spectrograms are patched and tokenized into sequences of roughly 1,500 audio tokens.
The model is trained to predict the text transcript that corresponds to that audio sequence. Over enough examples across dozens of languages and acoustic conditions, it learns the statistical relationship between audio patterns and their text representations.
At inference time, you give it audio. It produces text. But underneath, it's running the same next-token prediction process as a language model — except the input tokens represent slices of a sound spectrogram rather than pieces of a sentence.
Models like Meta's AudioCraft and Google's MusicLM use a variant of this approach in reverse — they generate audio tokens from text descriptions ("upbeat jazz with piano and drums") and decode those tokens back into a waveform. The transformer generates the audio token sequence; a separate decoder turns it back into sound. Same architecture, opposite direction, different decoder.
Because the mel spectrogram captures frequency patterns over time, it implicitly encodes far more than just the words being spoken. The characteristic frequency signature of a voice — what makes your voice sound like yours — is present in the spectrogram. So is intonation, speaking rate, and the acoustic markers of emotional state.
Transformer models trained on audio can learn to identify speakers, detect emotional tone, classify accents, and distinguish genuine speech from synthetic speech — all from the same spectral token representation. None of this was hand-engineered; the model learned it from the statistical patterns in enough labeled audio data.
This is the same emergence we saw with text: train on enough data with a broad enough objective, and the model learns structure you didn't ask it to learn, because that structure helps it predict better.
Images and audio are the canonical examples of non-text transformers — they're well-understood, widely deployed, and relatively easy to explain. But they share a common property with text: they're data types humans naturally think of as rich, meaningful, and information-dense.
The deeper question — and the more interesting one — is what happens when you apply this to data that doesn't feel like language at all. Data that has sequence structure, but that humans don't normally think of as containing "meaning" in any narrative sense.
That's where this gets genuinely surprising.
The mel spectrogram trick converts audio into an image so that the image tokenization approach can apply. This kind of creative data transformation — finding a representation of your data that fits an existing pipeline — is a major part of how AI gets applied to new domains. What other kinds of data might benefit from being transformed into a visual or sequential representation before being tokenized?