Computational music analysis: pitch, rhythm, structure, and generation

Why this subject matters now

For most of its history, music information retrieval (MIR) was a discipline of hand-designed features: someone would decide that a chroma vector or a mel-frequency cepstrum captured what mattered, and the rest of the pipeline was classical signal processing and pattern recognition. That world produced the representations this page spends most of its time on, and those representations remain the right way to think, because they encode real facts about how pitch and time work. What changed in the last five years is the output side. A practitioner in 2020 could separate a mixture into stems, transcribe a melody, or estimate a key. A practitioner today is expected to generate a coherent three-minute track conditioned on a text prompt, at 44.1 kHz stereo, with structure that holds together across sections. The models that do this, MusicLM, MusicGen, and Stable Audio among them, are the same transformer and diffusion architectures that reshaped language and images, adapted to the peculiar demands of audio: a very high sample rate, long-range dependencies measured in millions of samples, and a listener whose ear is exquisitely sensitive to pitch and timing errors that a vision model would never be penalized for.

The consequence is that the field now has two halves that a strong practitioner must hold at once. The analysis half, chroma, onset envelopes, self-similarity matrices, is interpretable, cheap, and still the backbone of every content-based recommender and every automatic accompaniment system. The generative half is a neural stack that consumes tokens or latents and produces waveforms. The two are not separate: the tokenizers that feed a symbolic music transformer are built from the same note-event abstraction that a piano-roll analysis produces, and the neural audio codecs that feed MusicGen are trained to preserve exactly the spectral structure that harmonic-percussive separation exploits. Understanding the generation stack without understanding the analysis it is implicitly modelling leaves you unable to reason about why these systems fail, which they still do, on rhythm, on long-term form, and on precise pitch.

Pitch and the frequency-to-pitch mapping

The single most important fact about musical pitch is that it is logarithmic in frequency. Two tones an octave apart have a frequency ratio of exactly 2, and this ratio is perceptually constant: the octave from 220 Hz to 440 Hz and the octave from 440 Hz to 880 Hz sound like the same musical interval even though the second spans twice as many hertz. Any representation of pitch that is linear in frequency fights this fact; every representation MIR actually uses is linear in the logarithm of frequency.

The equal-tempered scale and the MIDI mapping

Twelve-tone equal temperament divides the octave into twelve equal ratios on the logarithmic axis. If the octave is a ratio of 2 and it is split into twelve equal steps, each semitone is a frequency ratio of \( 2^{1/12} \approx 1.05946 \). We anchor the whole grid with a single reference, the concert-pitch convention that the note A above middle C, MIDI number 69, has frequency 440 Hz. Every other note is then a fixed number of semitones away from that anchor.

Let \( m \) be a note's MIDI number and \( f \) its frequency. Moving up by \( m - 69 \) semitones from 440 Hz multiplies the frequency by \( 2^{1/12} \) that many times:

To go the other direction, solve for \( m \). Divide by 440 and take the base-2 logarithm:

That is the frequency-to-MIDI formula, derived rather than asserted. It is a straight line in \( \log_2 f \) with slope 12 (twelve MIDI numbers per octave) and an intercept fixed by the 440 Hz anchor. Nothing in it is special to 440; replacing the reference with 442 Hz, as some orchestras tune, shifts the whole line by \( 12\log_2(442/440)\approx 0.078 \) of a semitone.

Cents: the fine ruler

A semitone is often too coarse a unit for measuring tuning. The cent divides the semitone into 100 equal logarithmic parts, so an octave is 1200 cents. The interval in cents between two frequencies \( f_1 \) and \( f_2 \) is

which follows directly: one octave is \( \log_2 2 = 1 \) on the log-2 axis, and we declared that to be 1200 cents, so the conversion factor is 1200. The just-noticeable difference for pitch is roughly 5 to 10 cents for sustained tones, which is why a tuner reports cents and why quarter-tone deviations, 50 cents, are audible as clearly out of tune. The fractional part of \( m = 69 + 12\log_2(f/440) \), scaled by 100, is exactly the deviation in cents from the nearest equal-tempered note.

The harmonic series and why pitch is not one frequency

A single note from a violin, a voice, or a piano is not a single sinusoid. A vibrating string or air column supports a fundamental frequency \( f_0 \) and a series of overtones at integer multiples \( 2f_0, 3f_0, 4f_0, \dots \), the harmonic series. The perceived pitch corresponds to \( f_0 \), but most of the energy and all of the timbre live in the overtones. This is why pitch estimation is hard: the spectrum of a single note already looks like a comb of peaks, and the spectrum of a chord is several interleaved combs. The problem of recovering the set of active \( f_0 \) values from such a mixture is multi-pitch estimation, and it is genuinely ill-posed because harmonics collide. The third harmonic of C4 (3×261.63 = 784.9 Hz) is almost exactly G5 (784.0 Hz), so a C and a G an octave and a fifth apart share spectral energy, and no amount of peak-picking can unambiguously assign that energy. Klapuri's iterative spectral-subtraction estimators and, more recently, the deep transcription models trained on aligned audio-MIDI corpora both exist precisely because this assignment problem has no closed-form answer. The design of the frequency analysis that produces these spectra, the short-time Fourier transform, its window and hop choices, and the constant-Q transform that spaces bins logarithmically to match the pitch axis, is worked out on the audio signal processing page; here we take the magnitude spectrum \( |X(k)| \) as given and ask what musical structure to read off it.

The easier special case, a single melody line, is monophonic pitch tracking, and it is worth separating from the polyphonic problem because it is essentially solved. The autocorrelation of a quasi-periodic waveform peaks at the fundamental period, and the YIN estimator refines this by minimizing a cumulative-mean-normalized difference function that suppresses the octave errors plain autocorrelation makes; pYIN adds a probabilistic layer, running several YIN candidate pitches through an HMM to produce a smooth, gap-free melody contour. The learned counterpart, CREPE, trains a convolutional network directly on the waveform to classify pitch into fine-grained cents bins and beats the signal-processing estimators on noisy and expressive material. Multi-pitch estimation is harder by the harmonic-collision argument just given, and this is the point at which classical peak-picking gives way to the trained transcription models, whose most-cited instance, the Onsets and Frames piano transcriber, predicts note onsets and sustained frames with two coupled objectives so that the onset detector gates the frame detector, sharply cutting the spurious note fragments a frame-only model produces.

Chroma: folding the spectrum into twelve bins

The octave-equivalence fact, that notes an octave apart are musically the same pitch class, suggests collapsing the frequency axis modulo the octave. A chroma vector is a twelve-dimensional descriptor whose entries are the twelve pitch classes C, C#, D, ..., B, obtained by summing all spectral energy that maps to each pitch class regardless of octave. Deriving the mapping is a matter of composing the frequency-to-pitch line with a modulo.

A DFT bin \( k \) of an \( N \)-point transform at sample rate \( f_s \) sits at frequency \( f_k = k f_s / N \). Convert that to a real-valued MIDI-like pitch, \( p(k) = 69 + 12\log_2(f_k / 440) \), then reduce to a pitch class by rounding to the nearest semitone and taking the result modulo 12:

The chroma entry for pitch class \( q \) is the total magnitude of every bin that maps there,

usually restricted to a musically relevant band (roughly 50 Hz to 5 kHz) and normalized so the vector sums or maxes to one. Fujishima introduced this "pitch class profile" for real-time chord recognition in 1999, and it is the workhorse feature for every harmony task downstream.

Two invariances fall out of the construction and explain why chroma is the right feature for harmony. First, octave invariance: because the pitch class uses \( p(k) \bmod 12 \), a note and the same note an octave up land in the same bin, so chroma discards register while keeping harmonic identity, exactly what a chord is defined by. Second, key/transposition covariance: transposing a passage up by \( t \) semitones cyclically rotates the chroma vector by \( t \) positions, \( \text{chroma}'[q] = \text{chroma}[(q - t)\bmod 12] \), because every pitch class moves by the same amount. This turns "is this the same chord in a different key?" into "is this the same vector up to a cyclic shift?", a comparison a computer can do in twelve rotations. The cost of these invariances is that chroma cannot tell a root-position chord from its inversion, and it confuses harmonics with true notes; the third harmonic of the root falls a perfect twelfth above it, i.e. into the pitch class of the fifth, so a pure root note already deposits energy into the fifth's chroma bin. Practical chroma extractors dampen this with spectral whitening or by using a log-frequency constant-Q spectrogram whose bins align with semitones from the start.

Two refinements of the raw chroma are worth naming because production systems use them by default. The first is to compute chroma from a constant-Q transform rather than a linear DFT. A DFT spaces its bins linearly in hertz, so the low octaves, where semitones are only a few hertz apart, get too few bins to resolve, while the high octaves are oversampled; the constant-Q transform instead spaces bins geometrically so every semitone gets the same number of bins across the whole range, which is exactly the resolution chroma wants, and the mapping from constant-Q bins to pitch classes becomes a fixed integer grouping rather than the rounding of a logarithm. The second is CENS (chroma energy normalized statistics), Müller and colleagues' smoothed and quantized chroma that applies a logarithmic amplitude compression, quantizes each entry into a few levels, and averages over a window of frames; the compression tames dynamics, the quantization discards small differences, and the smoothing makes the feature robust to tempo and articulation, which is why CENS is the default feature for the cover-song and structure comparisons discussed below. Both refinements leave the twelve-bin, octave-folded, transposition-covariant structure of chroma intact; they only change how the energy is measured and normalized before folding.

Onset detection and the spectral flux

Rhythm analysis starts with finding when notes begin. An onset is a moment of sudden change, typically a rise in energy or a shift in spectral content as a new note or drum hit starts. The most durable onset detector is the spectral flux, which measures how much the magnitude spectrum changes from one STFT frame to the next. Let \( |X(k,n)| \) be the magnitude of bin \( k \) in frame \( n \). The spectral flux onset envelope is

The half-wave rectifier \( H \) is the crucial detail. Without it, energy rising in one band would be cancelled by energy falling in another, and the sum would stay near zero across a note change. By keeping only the positive differences, the flux responds to energy appearing, which is what an onset is, and ignores energy decaying, which is the tail of the previous note. Bello and colleagues' 2005 tutorial catalogues the variants: differencing the log magnitude (a relative rather than absolute change, so quiet passages are treated fairly), weighting by frequency, or using phase deviation for soft onsets that carry little energy. In practice one computes the flux, subtracts a local moving average to remove slow drift, half-wave rectifies again, and picks peaks above an adaptive threshold; each peak is an onset. The result, sampled at the STFT frame rate, is the onset envelope, the input to every tempo and beat algorithm below.

Tempo estimation and beat tracking

Given the onset envelope \( O(n) \), a real-valued signal that spikes at note onsets, two questions follow: how fast is the music (tempo), and where exactly are the beats (beat tracking). Tempo is a periodicity of the onset envelope; beats are the phase-aligned pulse train at that periodicity.

Tempo from autocorrelation of the onset envelope

If the music has a steady beat with period \( \tau \) frames, the onset envelope tends to repeat itself when shifted by \( \tau \), because onsets recur on the beat grid. The autocorrelation measures exactly this self-similarity as a function of lag \( \ell \):

A peak in \( r(\ell) \) at lag \( \ell^\star \) says the envelope is most similar to itself when shifted by \( \ell^\star \) frames, so \( \ell^\star \) is the beat period in frames. Converting to beats per minute, if the envelope is sampled at \( f_r \) frames per second, then a period of \( \ell^\star \) frames is \( \ell^\star / f_r \) seconds per beat, hence

One restricts the search to a musically plausible band, say 40 to 200 BPM, to avoid locking onto the trivial zero-lag peak or onto sub-beat periodicities. The autocorrelation has a characteristic ambiguity: it peaks not only at the true period but at its integer multiples and, less strongly, at rational fractions, so the octave error, reporting 60 BPM for a 120 BPM track or vice versa, is the dominant failure mode. A common refinement is the tempogram, a time-varying tempo representation that computes a local autocorrelation (or a bank of comb filters, or a short-time Fourier transform of the onset envelope) in a sliding window, giving a lag-versus-time image whose bright horizontal bands trace the tempo as it drifts. The tempogram is to tempo what the spectrogram is to pitch: a two-dimensional map trading time resolution for periodicity resolution.

Ellis's dynamic-programming beat tracker

Knowing the tempo fixes the beat spacing but not the beat positions. Ellis's 2007 formulation makes the positioning a global optimization with a clean objective, and it is worth deriving because the structure, a reward for hitting onsets plus a penalty for irregular spacing, recurs throughout MIR. Suppose we have already estimated a target beat period \( \tau_p \) frames from the tempogram. We seek a sequence of beat times \( t_1 < t_2 < \dots < t_N \) (frame indices) that simultaneously land on strong onsets and stay evenly spaced at roughly \( \tau_p \). Ellis writes the total score of a beat sequence as

where \( O(t_i) \) is the onset strength at beat \( i \), rewarding beats that coincide with detected onsets, and \( F \) is a penalty for the inter-beat interval deviating from the target period. Ellis chooses a log-domain penalty that is symmetric under doubling and halving,

which is zero when the spacing equals \( \tau_p \) and grows quadratically in the log-ratio, so a beat twice or half the target period is penalized equally, matching the perceptual symmetry of tempo. The weight \( \alpha \) trades onset fit against regularity: large \( \alpha \) forces a metronomic grid, small \( \alpha \) lets the beats chase every onset. Because \( C \) is a sum of terms each depending only on consecutive beats, it has optimal substructure and is maximized by dynamic programming. Define \( D^\star(t) \) as the best cumulative score of any beat sequence ending exactly at frame \( t \):

with the maximizing predecessor \( \tau \) stored in a back-pointer \( B(t) \). The search over \( \tau \) is restricted to a window around \( t - \tau_p \), so each state costs \( O(\tau_p) \) and the whole recursion is linear in the number of frames. After filling \( D^\star \) over all \( t \), the optimal final beat is \( t_N = \argmax_t D^\star(t) \) and the full beat sequence is recovered by following \( B \) backward. The elegance is that a single pass produces the globally optimal beat sequence under the objective, avoiding the drift that greedy phase-locking suffers when a few onsets are missing: the penalty term carries the grid across silent gaps, and the reward term snaps it back onto onsets when they resume. This is the algorithm behind librosa.beat.beat_track.

Chord recognition and key finding

Chroma turns harmony into a twelve-vector, and the two central harmony tasks, naming the chord at each instant and naming the key of the whole piece, are both nearest-template problems on chroma, differing mainly in time scale and in whether temporal smoothing is applied.

Chord recognition: template matching then HMM smoothing

Fujishima's original idea is disarmingly direct. For each chord type, write down a binary template over the twelve pitch classes: a major triad is 1 at the root, major third, and fifth and 0 elsewhere, so C major is \( [1,0,0,0,1,0,0,1,0,0,0,0] \). There are 12 roots times a handful of qualities (major, minor, seventh, diminished), giving a few dozen templates. At each frame, compute the chroma vector and score it against every template, for instance by the correlation or the inner product after normalization, and output the chord whose template scores highest. This works frame by frame but produces a jittery transcription, because chroma is noisy and a passing note can flip the estimate for a single frame.

The fix is to model the fact that chords persist: real music holds a chord for a beat or a bar, not a frame. This is exactly a hidden Markov model, with the hidden state being the current chord, the transition matrix encoding that chords change rarely and follow musical grammar (a dominant tends to resolve to its tonic), and the emission model being the template-match likelihood of the observed chroma given the chord. Decoding the most likely chord sequence with the Viterbi algorithm smooths out the frame-level jitter and enforces musically plausible transitions. The full derivation of the HMM, the forward-backward recursions, and Viterbi decoding lives on the probabilistic graphical models page and is applied to acoustic modelling on the speech and spoken language page; the only music-specific content is the choice of states (the chord vocabulary), the transition prior (from music theory or estimated from a labelled corpus), and the emission (chroma-to-template likelihood).

Key finding: the Krumhansl-Schmuckler correlation method

The key of a piece, C major, A minor, F# minor, is a global harmonic center. Krumhansl and Kessler measured, through listening experiments, how well each of the twelve pitch classes "fits" a given key, producing two 12-vectors, the major key profile and the minor key profile, that quantify tonal hierarchy. In a major key the tonic scores highest, followed by the fifth and the third, with non-scale tones scoring lowest. Krumhansl and Schmuckler's key-finding algorithm compares a piece's overall chroma against all 24 rotated profiles and picks the best match by correlation.

Concretely, aggregate the chroma over the whole piece into a single 12-vector \( \mathbf{x} \) (the total energy in each pitch class). Let \( \mathbf{k}^{\text{maj}} \) and \( \mathbf{k}^{\text{min}} \) be the major and minor profiles anchored at C. For each of the twelve tonics \( t \), form the candidate profile by cyclically rotating the profile by \( t \) semitones, and compute the Pearson correlation between \( \mathbf{x} \) and that rotated profile:

The estimated key is the \( (t, \text{mode}) \) maximizing \( \rho \), over all 24 candidates. Using the correlation rather than a raw dot product is what makes the method robust: subtracting the means removes any overall loudness offset, and normalizing by the standard deviations removes any overall scale, so only the shape of the tonal hierarchy is compared. The rotation-by-\(t\) exploits the transposition covariance of chroma derived earlier: the minor profile is stored once and reused for all twelve minor keys by rotation, and likewise for major.

Music structure analysis

Above the level of chords and beats sits musical form: verse, chorus, bridge, the repetition and contrast that give a piece its architecture. The tool for finding it is the self-similarity matrix (SSM). Extract a feature vector per frame, chroma for harmonic structure, MFCCs for timbral structure, and build the matrix

the pairwise similarity of every frame \( i \) with every frame \( j \), typically cosine similarity or a negated Euclidean distance. The SSM is symmetric with a bright main diagonal (every frame is identical to itself). Its off-diagonal structure encodes form in two readable patterns. A repeated section, the chorus returning, appears as a diagonal stripe parallel to the main diagonal, because a run of consecutive frames matches another run offset in time. A homogeneous section, a sustained texture, appears as a block of high similarity, because every frame within it resembles every other. Real SSMs mix both, and cleaning them, path enhancement along diagonals, smoothing within blocks, is the substance of structure analysis.

To turn an SSM into segment boundaries, Foote's novelty method correlates a small checkerboard kernel along the main diagonal. The kernel is a \( 2L\times 2L \) matrix that is positive on its diagonal quadrants and negative on its off-diagonal quadrants; sliding it along the diagonal of \( S \) and summing the elementwise product produces a novelty curve \( \nu(n) \) that peaks exactly where the texture changes, because at a boundary the past-past and future-future quadrants are self-similar (positive, aligned with the positive kernel) while the past-future quadrants are dissimilar (small, aligned with the negative kernel), maximizing the sum. Peaks in \( \nu(n) \) are the segment boundaries. Grouping the resulting segments into labelled sections (which are the same chorus) is then a clustering problem on the segment-level features, and modern systems, from the spectral-clustering method of McFee and Ellis to learned boundary detectors, build on exactly this SSM-plus-novelty foundation.

Source separation

Separating a mixture into its constituent sources, vocals, drums, bass, and other, is one of the oldest and most commercially important MIR problems. The classical approach exploits a structural asymmetry visible directly in the spectrogram; the modern approach learns the separation end to end. Both are worth understanding, because the classical method is still the right tool when no training data exists, and it reveals the structure the neural models learn to exploit.

Harmonic-percussive separation by median filtering

Look at a spectrogram \( S(f, t) \) (frequency by time) of typical music. Harmonic content, sustained pitched notes, forms horizontal ridges: a held note keeps the same frequency over many time frames. Percussive content, drum hits and transients, forms vertical ridges: a snare hit spreads energy across all frequencies in a single instant. Fitzgerald's 2010 insight is that a median filter separates these two geometries cleanly, and the derivation is short. Apply a median filter horizontally, along time, of length \( \ell \):

A horizontal ridge is nearly constant along time, so its median is that same value: the filter preserves harmonic content. A vertical ridge is a brief spike, present in only one or two of the \( \ell \) frames in the window, so it is a minority in the sample and the median rejects it. Thus \( H \) retains the horizontal structure and suppresses the vertical. Symmetrically, a median filter vertically, along frequency, of length \( m \),

preserves broadband vertical spikes and rejects narrow horizontal ridges, giving the percussive estimate. The two estimates are then turned into soft masks that partition the original spectrogram's energy. A common choice is the Wiener-like mask

and the separated spectrograms are \( M_H \odot S \) and \( M_P \odot S \), inverted back to audio with the mixture's phase. The whole method has no learned parameters, one filter length per axis, and runs in real time; it is the foundation of drum-versus-melody preprocessing and of the percussive-onset enhancement used before beat tracking.

The shift to deep separation

Median filtering separates by geometry, but it cannot tell a sung vowel from a bowed cello, both are harmonic and horizontal. Vocals-versus-accompaniment and four-stem separation need a model that has learned what each source looks like. Two lines dominate. Open-Unmix (Stöter and colleagues, 2019) is a bidirectional-LSTM network that operates on the magnitude spectrogram and predicts a soft mask per source, then multiplies it with the mixture spectrogram, the same masking idea as HPSS but with a learned, source-specific mask; it is the reference open baseline for the standard four-stem benchmark. Demucs (Défossez and colleagues, 2019, then the hybrid and transformer versions) instead works partly or wholly in the waveform domain with a U-Net-style encoder-decoder, sidestepping the phase-reconstruction problem that magnitude masking suffers, and its later hybrid variant combines spectral and waveform branches to get the best of both. Both are trained to minimize a reconstruction loss against ground-truth stems on the MUSDB18 dataset, and both are evaluated with signal-to-distortion ratio via the mir_eval / museval tooling. The masking-versus-waveform tension is the central design axis: masks are interpretable and guarantee the separated sources sum to the mixture, but inherit the mixture's phase; waveform models fix phase at the cost of a harder learning problem. The frequency-domain machinery, STFT, magnitude, phase, inversion, is derived on the audio signal processing page.

Symbolic music representation

Not all music analysis works on audio. A score or a MIDI file is a symbolic representation: an explicit list of notes with pitches, onsets, durations, and velocities, free of the acoustic ambiguity that makes transcription hard. Symbolic MIR and symbolic generation work on these discrete events, and the representation choice determines what a model can learn.

The piano roll is the most direct encoding: a binary (or velocity-valued) matrix with pitch on one axis and quantized time on the other, a 1 wherever a note is sounding. It is dense, image-like, and convenient for convolutional models, but it wastes capacity on silence and blurs the distinction between one long note and two repeated short ones. Sequence models instead consume a token stream, and how to tokenize music is a genuine design problem with two influential answers. A note-based tokenization emits, per note, a small tuple of tokens: pitch, duration, and a time-shift to the next event, so a melody becomes a flat sequence a transformer can autoregress over. The MIDI-derived REMI representation (Huang and Yang) adds an explicit metrical grid: it emits bar and position tokens that anchor each note to a beat within a bar, plus tempo and chord tokens, so the model is told where the downbeats are rather than having to infer absolute time from a running sum of time-shifts. That metrical scaffolding measurably improves rhythmic coherence, because the model can attend to "beat 1 of every bar" as an explicit position rather than an emergent one. The trade is vocabulary size and sequence length against inductive bias, the same trade that tokenization poses for language, worked in detail on the language models from scratch page. Music theory, scales, chord functions, voice-leading rules, can be encoded either as extra tokens (a chord token stream) or as constraints at sampling time (masking out-of-key pitches), and the design question of whether to bake theory into the representation or let the model learn it from data is unresolved and genre-dependent.

Music similarity and recommendation

"Find me more songs like this one" is content-based similarity; "people who liked this also liked that" is collaborative filtering; production recommenders blend both. Content-based similarity reduces a track to an embedding, a fixed vector summarizing timbre, harmony, and rhythm, and ranks candidates by cosine distance in that space. Classical embeddings averaged MFCC and chroma statistics; modern ones are the penultimate layer of a neural tagger or a self-supervised audio encoder, trained so that acoustically or semantically similar tracks land nearby. A pure-harmony instance of content similarity is cover-song detection: deciding whether two recordings are the same composition despite differing in instrumentation, tempo, and, crucially, key. The transposition covariance of chroma derived earlier is exactly what makes this tractable, because two performances of the same song in different keys have chroma sequences that are cyclic rotations of one another. A key-invariant comparison therefore scores the similarity under all twelve rotations and keeps the best, while a tempo-invariant comparison warps the two chroma sequences to absorb local tempo differences. The classic pipeline, from Ellis and Poliner and refined by Serrà and colleagues, builds a cross-similarity matrix between the two beat-synchronous chroma sequences and scores the length of the best diagonal alignment path through it, the same diagonal-stripe geometry the self-similarity matrix uses for structure, applied across two pieces rather than within one. That one property of chroma, its behavior under transposition, drives both key finding and cover detection is a good illustration of why the representation, not the classifier, is where the leverage lives.

Collaborative filtering ignores audio entirely and factorizes the user-item interaction matrix \( R \approx U V^\top \) into user and item latent vectors, so that a track's representation is inferred purely from co-listening patterns; the matrix-factorization and nearest-neighbor machinery, including the locality-sensitive hashing that makes similarity search scale to catalogs of tens of millions, is derived on the mining massive datasets page. The two signals are complementary in a precise way: collaborative filtering is accurate for popular items with dense interaction data but useless for a brand-new upload with no listens (the cold-start problem), whereas a content embedding is available the instant a track is uploaded but blind to the social signal of taste. The practical answer, pioneered in van den Oord and colleagues' work on predicting collaborative-filter latent vectors from audio, is to train an audio network to predict the collaborative embedding, giving cold-start items a usable vector from their audio alone.

Generative music

Music generation has passed through four eras, each strictly more capable and each still in use for the right problem: Markov chains, recurrent networks, transformers, and diffusion / latent audio models. Following the arc makes clear what each added.

Markov chains: melody as a transition matrix

The oldest statistical model of melody treats the next note as depending only on the current note (a first-order Markov chain). Estimate a transition matrix \( T \) where \( T_{ij} = \P(\text{next} = j \mid \text{current} = i) \) by counting note-to-note transitions in a corpus and normalizing each row to sum to one, then generate by sampling a walk through the states. The model is trivial to fit and interpretable, but its memory is one note deep, so it captures local pitch tendencies (which note tends to follow which) and nothing about phrase structure, meter, or long-range form; generated melodies wander without cadence. Higher-order chains (conditioning on the last \( k \) notes) extend the memory but suffer combinatorial state-space blowup and data sparsity, the exact pressure that motivated the move to neural models with continuous hidden state.

RNNs, then transformers, then the long-range problem

Recurrent networks (LSTM and GRU melody and performance models, of which Google's Magenta project shipped several) replaced the Markov chain's finite state with a continuous hidden vector that can, in principle, carry information arbitrarily far. In practice the hidden state forgets: the gradient signal linking a note to its context decays, and RNN-generated music holds a motif for a few bars then drifts. The transformer removed the recurrence in favor of attention, letting any position look directly at any other, which is exactly what music needs, because a chorus at 2:30 rhymes with the chorus at 0:30, a dependency of thousands of tokens that no recurrent state survives. The mechanics of self-attention, queries, keys, values, and the softmax, are derived at length on the NLP with deep learning and sequence models pages.

The Music Transformer and relative attention

Huang and colleagues' 2018 Music Transformer identified why a vanilla transformer, with absolute position embeddings, generates music that loses structure over long spans: what matters in music is relative position, the interval between two events, the number of beats separating them, not their absolute offsets from the start. A motif transposed to bar 32 relates to its answer four beats later by the same relative distance as the original at bar 1, and absolute embeddings make the model relearn that relationship at every offset. Relative-position self-attention, introduced by Shaw and colleagues and made efficient by Huang and colleagues, adds a term to the attention logits that depends on the distance \( i - j \) between the query at position \( i \) and the key at position \( j \):

where \( \mathbf{r}_{\,i-j} \) is a learned embedding of the relative distance. The first term is ordinary content-based attention; the second lets the query attend based purely on how far away a key is, so a learned "four beats back" relation applies identically everywhere in the piece. The obstacle is memory: naively, the relative term needs a distinct \( \mathbf{r} \) for every query-key pair, an \( O(L^2 d) \) tensor. Huang and colleagues' contribution is the skewing trick, an index-shifting reindexing of a compact \( O(Ld) \) tensor of relative embeddings into the \( L\times L \) logit positions using only a pad-and-reshape, cutting the intermediate memory from \( O(L^2 d) \) to \( O(Ld) \) and making relative attention feasible for the minute-long sequences that piano performances require. The payoff is qualitative: the Music Transformer generates piano performances with recognizable, repeating thematic material across a full minute, where the absolute baseline meanders.

Diffusion and latent audio models

The current frontier generates the audio itself, at full sample rate, conditioned on text. Three systems mark the trajectory. Google's MusicLM (Agostinelli and colleagues, 2023) casts music generation as hierarchical sequence modelling over discrete audio tokens, semantic tokens capturing long-term structure and acoustic tokens (from the SoundStream neural codec) capturing fine detail, conditioned on a text embedding from a joint music-text model. Meta's MusicGen (Copet and colleagues, 2023, in the AudioCraft repository) simplifies this to a single autoregressive transformer over the multiple codebooks of the EnCodec neural audio codec, with a token-interleaving pattern that lets one model predict all codebooks efficiently, and it conditions on text or on a melody. Stable Audio (Evans and colleagues, 2024) takes the diffusion route instead: it trains a latent diffusion model in the compressed latent space of an autoencoder, generating variable-length, full-band stereo audio conditioned on text and on timing embeddings that let the user request a specific duration. The common thread is a two-stage design, a neural codec or autoencoder that compresses waveform into a tractable discrete-token or continuous-latent space, then a powerful sequence or diffusion model over that compact space, because modelling 44.1 kHz audio sample by sample is computationally hopeless. The diffusion machinery, forward noising, the denoising objective, classifier-free guidance, and latent diffusion, is derived on the diffusion and large vision models page; the music-specific content is the codec that defines the latent space and the conditioning that ties audio to text and to musical structure.

Implementation

The analysis pipeline is compact enough to implement from first principles in NumPy, and doing so makes the derivations concrete. The first block folds a magnitude spectrum into chroma and runs the Krumhansl-Schmuckler key finder on an aggregated chroma vector; the printed correlations match Problem 4. It uses only elementwise operations and a small cyclic rotation, avoiding any linear-algebra factorization, so it is unaffected by the platform's historical LAPACK issue.

import numpy as np

NAMES = ['C','C#','D','D#','E','F','F#','G','G#','A','A#','B']

def hz_to_pitch_class(f):
    # p = 69 + 12 log2(f/440); pitch class = round(p) mod 12
    p = 69.0 + 12.0 * np.log2(f / 440.0)
    return int(round(p)) % 12

def fold_to_chroma(freqs, mags, fmin=50.0, fmax=5000.0):
    # freqs, mags: 1-D arrays of the same length (spectral peaks or bins)
    chroma = np.zeros(12)
    for f, m in zip(freqs, mags):
        if fmin <= f <= fmax:
            chroma[hz_to_pitch_class(f)] += m
    return chroma / (chroma.max() + 1e-12)

# Problem 2: four peaks -> a C-major triad chroma
freqs = np.array([261.6, 329.6, 392.0, 523.3])
mags  = np.array([1.0,   0.8,   0.9,   0.5])
chroma = fold_to_chroma(freqs, mags)
print({NAMES[i]: round(chroma[i], 3) for i in range(12) if chroma[i] > 0.01})
# -> {'C': 1.0, 'E': 0.533, 'G': 0.6}

# Krumhansl-Kessler key profiles (major, minor), anchored at C
MAJOR = np.array([6.35,2.23,3.48,2.33,4.38,4.09,2.52,5.19,2.39,3.66,2.29,2.88])
MINOR = np.array([6.33,2.68,3.52,5.38,2.60,3.53,2.54,4.75,3.98,2.69,3.34,3.17])

def pearson(a, b):
    a = a - a.mean(); b = b - b.mean()
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def find_key(x):
    scores = []
    for t in range(12):                       # 12 tonics
        scores.append((f'{NAMES[t]} major', pearson(x, np.roll(MAJOR, t))))
        scores.append((f'{NAMES[t]} minor', pearson(x, np.roll(MINOR, t))))
    scores.sort(key=lambda kv: -kv[1])
    return scores

# Problem 4: aggregated chroma of a C-major piece
x = np.array([0.9,0.1,0.5,0.1,0.7,0.55,0.15,0.8,0.12,0.45,0.15,0.35])
for name, r in find_key(x)[:4]:
    print(f'{name:9s} r = {r:+.4f}')
# C major   r = +0.9724   (winner)
# G major   r = +0.6805
# A minor   r = +0.6538
# E minor   r = +0.6397

The second block builds a spectral-flux onset envelope and recovers the tempo by autocorrelating it, reproducing Problem 3: a synthetic 120 BPM pulse train yields a top autocorrelation lag of 50 frames at a 100 Hz frame rate, i.e. 120.0 BPM.

import numpy as np

def spectral_flux(S):
    # S: (n_freq, n_frames) magnitude spectrogram
    diff = np.diff(S, axis=1)              # frame-to-frame change
    flux = np.maximum(diff, 0.0).sum(0)    # half-wave rectify, sum over freq
    flux = np.concatenate([[0.0], flux])   # align length to n_frames
    return flux

def tempo_from_envelope(env, fps, bpm_min=40, bpm_max=200):
    env = env - env.mean()
    n = len(env)
    ac = np.correlate(env, env, mode='full')[n - 1:]  # nonneg lags
    ac[0] = 0.0
    lo = int(fps * 60 / bpm_max)          # smallest lag we allow
    hi = int(fps * 60 / bpm_min)          # largest lag we allow
    lags = np.arange(lo, hi)
    best = lags[np.argmax(ac[lags])]
    return 60.0 * fps / best, best

# Synthetic onset envelope: strong beats at 120 BPM, weaker offbeats
fps, dur = 100, 8.0
n = int(dur * fps)
env = np.zeros(n)
for t in np.arange(0, dur, 0.50):         # on-beats  (0.5 s = 120 BPM)
    env[int(t * fps)] = 1.0
for t in np.arange(0.25, dur, 0.50):      # off-beats (weaker)
    env[int(t * fps)] = 0.4
env += 0.05 * np.random.RandomState(0).randn(n)

bpm, lag = tempo_from_envelope(env, fps)
print(f'best lag = {lag} frames -> {bpm:.1f} BPM')
# best lag = 50 frames -> 120.0 BPM

A learned separator is a small masking network: it consumes a magnitude spectrogram and predicts a soft mask per source, multiplied back onto the mixture, exactly the HPSS masking idea with the mask made learnable. The block below shows the forward pass of such a masker in PyTorch and JAX. It is a toy, a couple of convolutions and a sigmoid, but its shape is that of Open-Unmix's mask head, and it makes the masking-versus-waveform point concrete: the network never predicts audio, only a mask in \( [0,1] \) that partitions the mixture's energy.

import torch
import torch.nn as nn

class SpecMasker(nn.Module):
    """Predict one soft mask per source from a magnitude spectrogram.
    mix_mag: (B, 1, F, T)  ->  masks: (B, S, F, T) in [0, 1]."""
    def __init__(self, n_sources=2, ch=32):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(1, ch, 3, padding=1), nn.ReLU(),
            nn.Conv2d(ch, ch, 3, padding=1), nn.ReLU(),
            nn.Conv2d(ch, n_sources, 3, padding=1),
        )

    def forward(self, mix_mag):
        masks = torch.sigmoid(self.net(mix_mag))      # (B, S, F, T)
        # separated magnitudes: mask * mixture, so sources ~ partition energy
        sep = masks * mix_mag                          # broadcast over S
        return masks, sep

x = torch.rand(4, 1, 513, 128)          # (B, 1, F, T)
model = SpecMasker(n_sources=2)
masks, sep = model(x)
print(masks.shape, sep.shape)           # (4, 2, 513, 128) (4, 2, 513, 128)
# masks sum toward 1 across sources only if trained with that constraint;
# here they are independent sigmoids, as in a per-source mask head.

import jax, jax.numpy as jnp
import flax.linen as nn

class SpecMasker(nn.Module):
    n_sources: int = 2
    ch: int = 32

    @nn.compact
    def __call__(self, mix_mag):        # mix_mag: (B, F, T, 1)
        h = nn.relu(nn.Conv(self.ch, (3, 3), padding='SAME')(mix_mag))
        h = nn.relu(nn.Conv(self.ch, (3, 3), padding='SAME')(h))
        h = nn.Conv(self.n_sources, (3, 3), padding='SAME')(h)
        masks = jax.nn.sigmoid(h)                     # (B, F, T, S)
        sep = masks * mix_mag                          # broadcast over S
        return masks, sep

x = jnp.asarray(jax.random.uniform(jax.random.PRNGKey(0), (4, 513, 128, 1)))
model = SpecMasker(n_sources=2)
params = model.init(jax.random.PRNGKey(1), x)
masks, sep = model.apply(params, x)
print(masks.shape, sep.shape)           # (4, 513, 128, 2) (4, 513, 128, 2)

Finally, the core of the Music Transformer is the relative-position term in the attention logits. The block below computes one relative-attention head and implements Huang and colleagues' skewing trick, which reindexes a compact \( (L, L) \) matrix of query-times-relative-embedding scores into the correctly aligned relative positions with a pad-and-reshape, avoiding the \( O(L^2 d) \) intermediate.

import torch
import torch.nn.functional as F

def skew(qe):
    # qe: (B, H, L, L) = Q @ E_rel^T, columns indexed by relative position.
    # Reindex so entry (i, j) uses relative distance (i - j). Huang et al. 2018.
    B, H, L, _ = qe.shape
    qe = F.pad(qe, (1, 0))                 # pad one col on the left -> (B,H,L,L+1)
    qe = qe.reshape(B, H, L + 1, L)        # reshape shifts the diagonal
    return qe[:, :, 1:, :]                  # drop first row -> (B, H, L, L)

def relative_attention(Q, K, V, E_rel):
    # Q,K,V: (B, H, L, d); E_rel: (L, d) learned relative-position embeddings
    d = Q.shape[-1]
    content = Q @ K.transpose(-1, -2)       # (B, H, L, L) content-based logits
    rel = skew(Q @ E_rel.transpose(0, 1))   # (B, H, L, L) relative logits
    logits = (content + rel) / (d ** 0.5)
    mask = torch.triu(torch.ones(Q.shape[-2], Q.shape[-2]), 1).bool()
    logits = logits.masked_fill(mask, float('-inf'))   # causal
    return torch.softmax(logits, -1) @ V

B, H, L, d = 2, 4, 16, 32
Q = torch.randn(B, H, L, d); K = torch.randn(B, H, L, d); V = torch.randn(B, H, L, d)
E_rel = torch.randn(L, d)
print(relative_attention(Q, K, V, E_rel).shape)   # (2, 4, 16, 32)

import jax, jax.numpy as jnp

def skew(qe):
    # qe: (B, H, L, L). Pad-and-reshape reindexing (Huang et al. 2018).
    B, H, L, _ = qe.shape
    qe = jnp.pad(qe, ((0, 0), (0, 0), (0, 0), (1, 0)))  # (B,H,L,L+1)
    qe = qe.reshape(B, H, L + 1, L)
    return qe[:, :, 1:, :]                               # (B, H, L, L)

def relative_attention(Q, K, V, E_rel):
    d = Q.shape[-1]
    content = Q @ jnp.swapaxes(K, -1, -2)               # (B, H, L, L)
    rel = skew(Q @ E_rel.T)                              # (B, H, L, L)
    logits = (content + rel) / jnp.sqrt(d)
    L = Q.shape[-2]
    mask = jnp.triu(jnp.ones((L, L), bool), 1)
    logits = jnp.where(mask, -jnp.inf, logits)           # causal
    return jax.nn.softmax(logits, -1) @ V

B, H, L, d = 2, 4, 16, 32
k = jax.random.PRNGKey(0)
Q = jax.random.normal(k, (B, H, L, d)); K = jax.random.normal(k, (B, H, L, d))
V = jax.random.normal(k, (B, H, L, d)); E_rel = jax.random.normal(k, (L, d))
print(relative_attention(Q, K, V, E_rel).shape)          # (2, 4, 16, 32)

How it is done in practice

A deployed MIR analysis pipeline is almost never one model; it is a chain of the components above, each with a well-understood failure mode, arranged so that later stages can correct earlier ones. A typical automatic-accompaniment or DJ system computes a constant-Q spectrogram once, derives chroma and an onset envelope from it, runs tempo estimation and DP beat tracking to get a beat grid, then does beat-synchronous chroma, averaging chroma within each beat, so that chord recognition operates on one vector per beat rather than per frame. Beat synchronization is the single most effective practical trick in the field: it collapses hundreds of noisy frames into a handful of stable beat-level features, cutting both the compute and the jitter that the downstream HMM would otherwise have to smooth away. The libraries that ship these pipelines, librosa for the classical stack, essentia for the C++ production stack, madmom for state-of-the-art beat and downbeat tracking, are the reference implementations, and reading their source is the fastest way to see the engineering details, the exact window lengths, the reference-frequency conventions, the normalization, that the derivations gloss.

On the generative side the engineering gap is dominated by the audio codec, not the sequence model. Generating raw 44.1 kHz stereo is out of reach, so every production system, MusicLM, MusicGen, Stable Audio, first trains a neural codec (SoundStream, EnCodec) or a continuous autoencoder that compresses audio by one to two orders of magnitude into discrete tokens or a latent, and the quality ceiling of the whole system is the codec's reconstruction quality: no downstream model can recover detail the codec threw away. Latency and streaming are the other production constraints, an autoregressive codec-token model like MusicGen generates left to right and can stream, whereas a diffusion model like Stable Audio denoises the whole clip jointly and cannot emit the first second before finishing the last, which shapes which architecture a given product chooses. Evaluation remains unsolved: Fréchet Audio Distance and CLAP-based text-audio alignment scores are the standard automatic metrics, but they correlate only loosely with human judgments of musicality, so listening tests remain the ground truth, and the objective metric numbers reported in the generation papers should be read as relative comparisons within a paper, not absolute quality claims.

The current research frontier

Four threads are moving quickly and pull from different groups. The first is controllability: text prompts are a blunt instrument, and recent work adds melody conditioning (MusicGen's melody variant, from Meta AI), explicit chord and structure conditioning, and inpainting or continuation so a musician can fix a bar without regenerating a track. The second is codec and latent quality: the entire generation stack is bottlenecked by its compressor, and descendants of SoundStream and EnCodec (Google, Meta) plus the Descript Audio Codec and RVQ variants keep pushing bitrate down and fidelity up, with the residual-vector-quantization design now standard. The third is real-time and streaming generation, where the autoregressive-versus-diffusion tension is being resolved by consistency and flow-matching models that denoise in a few steps, and by architectures that can emit audio causally; Stability AI's and others' fast-sampling work sits here. The fourth is transcription and analysis at scale, where large models trained on weakly-aligned or self-supervised audio, the descendants of the piano-transcription systems and of contrastive music-text models like CLAP and MuLan, are closing the gap between what a human annotator and a machine can read off a recording. Across all four, the open question the field has not answered is long-term form: current systems produce locally convincing music that still lacks the deliberate large-scale architecture, the setup and payoff across minutes, that distinguishes a composition from a convincing texture, and relative attention, structure conditioning, and hierarchical generation are all partial attacks on exactly this.

Open source to read

• librosa/librosa is the reference classical MIR stack in Python. Open librosa/feature/spectral.py for the chroma implementation and librosa/beat.py for the dynamic-programming beat tracker; both match the derivations here almost line for line.
• facebookresearch/demucs is the waveform-and-hybrid source separator. Start with demucs/htdemucs.py to see how the spectral and waveform branches are combined.
• sigsep/open-unmix-pytorch is the clean spectrogram-masking baseline. Read openunmix/model.py to see the BiLSTM mask head, the learned version of the HPSS mask.
• facebookresearch/audiocraft holds MusicGen and EnCodec. Read audiocraft/models/musicgen.py for the single-transformer codec-token model and the codebook-interleaving pattern.
• magenta/magenta and magenta/note-seq are the reference for symbolic generation and the NoteSequence abstraction that underlies most MIDI tokenizers; open note_seq/sequences_lib.py.
• craffel/mir_eval is the standard evaluation library; mir_eval/beat.py and mir_eval/chord.py define exactly how beat and chord accuracy are scored, which is worth reading before trusting any reported number.
• MTG/essentia is the C++ production MIR library with Python bindings; its key, tempo, and onset algorithms are what many commercial systems actually run.

Common misconceptions

"Chroma tells you the chord, including its inversion and bass note." Chroma is octave-invariant by construction, so it discards register entirely; a C major chord in root position and its first inversion (E in the bass) produce the same chroma vector. Recovering the bass note or inversion requires a separate bass-chroma feature computed over a low frequency band, not the full chroma.

"Autocorrelation gives the tempo, unambiguously." Autocorrelation peaks at the true beat period and at every integer multiple and simple fraction of it, so it cannot by itself distinguish 120 BPM from 60 or 240 BPM. Resolving the metrical level requires phase and accent information, which is why beat trackers score candidate beat positions against onset strength rather than reading the tempo off the autocorrelation alone.

"Harmonic-percussive separation can pull the vocals out of a mix." Median-filter separation splits a spectrogram by geometry, horizontal ridges versus vertical spikes, so it separates sustained pitched content from transients, but a sung voice is horizontal and pitched exactly like a held cello note. Isolating vocals needs a model that has learned what a voice is, which is what Open-Unmix and Demucs provide and what median filtering cannot.

"MIDI is audio." MIDI is a symbolic score, a list of note events with pitch, timing, and velocity, and it contains no sound at all; turning it into audio requires a synthesizer or sample-based instrument, and two renderings of the same MIDI file can sound completely different. Symbolic generation models output MIDI-like events, which is why their output must be synthesized before it can be heard, and why they sidestep the acoustic modelling that audio generators must solve.

"A bigger transformer will fix long-term musical structure on its own." The Music Transformer's improvement came not from scale but from changing the inductive bias, replacing absolute with relative position so that musical relationships generalize across the piece. Scale helps, but the representation of position and structure is a first-order design choice, and current models still fail on minutes-long form despite large parameter counts.

"Equal temperament is how instruments are actually tuned." Equal temperament is a convenient compromise that makes all keys equally usable by making every semitone the same ratio, but it puts every interval except the octave slightly out of just intonation; a violinist or singer adjusts pitches by several cents toward pure ratios in performance, which is why real recordings deviate systematically from the equal-tempered grid and why cents, not semitones, are the right unit for measuring intonation.

Self-check

References

1. M. Müller. Fundamentals of Music Processing: Audio, Analysis, Algorithms, Applications, 2nd ed. Springer, 2021. The central MIR reference; chroma, tempo, structure, and separation are all developed here. doi.org/10.1007/978-3-030-69808-9
2. A. Klapuri and M. Davy (eds.). Signal Processing Methods for Music Transcription. Springer, 2006. doi.org/10.1007/0-387-32845-9
3. C. L. Krumhansl. Cognitive Foundations of Musical Pitch. Oxford University Press, 1990. The source of the tonal-hierarchy key profiles.
4. T. Fujishima. "Realtime chord recognition of musical sound: a system using Common Lisp Music." Proc. International Computer Music Conference (ICMC), 1999. Introduces the pitch-class profile (chroma).
5. C. L. Krumhansl and E. J. Kessler. "Tracing the dynamic changes in perceived tonal organization in a spatial representation of musical keys." Psychological Review 89(4), 1982. doi.org/10.1037/0033-295X.89.4.334
6. D. P. W. Ellis. "Beat tracking by dynamic programming." Journal of New Music Research 36(1), 2007. doi.org/10.1080/09298210701653344
7. J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury, M. Davies, and M. B. Sandler. "A tutorial on onset detection in music signals." IEEE Transactions on Speech and Audio Processing 13(5), 2005. doi.org/10.1109/TSA.2005.851998
8. J. Foote. "Automatic audio segmentation using a measure of audio novelty." Proc. IEEE International Conference on Multimedia and Expo (ICME), 2000. doi.org/10.1109/ICME.2000.869637
9. D. FitzGerald. "Harmonic/percussive separation using median filtering." Proc. Digital Audio Effects (DAFx), 2010. arxiv.org/abs/1811.03917
10. A. Défossez, N. Usunier, L. Bottou, and F. Bach. "Music source separation in the waveform domain." arXiv:1911.13254, 2019 (Demucs). arxiv.org/abs/1911.13254
11. F.-R. Stöter, S. Uhlich, A. Liutkus, and Y. Mitsufuji. "Open-Unmix: a reference implementation for music source separation." Journal of Open Source Software 4(41), 2019. doi.org/10.21105/joss.01667
12. C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, N. Shazeer, I. Simon, C. Hawthorne, A. M. Dai, M. D. Hoffman, M. Dinculescu, and D. Eck. "Music Transformer." arXiv:1809.04281, 2018. arxiv.org/abs/1809.04281
13. Y.-S. Huang and Y.-H. Yang. "Pop Music Transformer: beat-based modeling and generation of expressive pop piano compositions." Proc. ACM Multimedia, 2020 (the REMI representation). arxiv.org/abs/2002.00212
14. A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank. "MusicLM: generating music from text." arXiv:2301.11325, 2023. arxiv.org/abs/2301.11325 The relative-position self-attention it builds on was introduced by P. Shaw, J. Uszkoreit, and A. Vaswani, "Self-attention with relative position representations," Proc. NAACL 2018, arxiv.org/abs/1803.02155.
15. J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez. "Simple and controllable music generation." arXiv:2306.05284, 2023 (MusicGen). arxiv.org/abs/2306.05284
16. Z. Evans, C. J. Carr, J. Taylor, S. H. Hawley, and J. Pons. "Fast timing-conditioned latent audio diffusion." arXiv:2402.04825, 2024 (Stable Audio). arxiv.org/abs/2402.04825
17. N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. "SoundStream: an end-to-end neural audio codec." IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, 2022. arxiv.org/abs/2107.03312
18. A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. "High fidelity neural audio compression." arXiv:2210.13438, 2022 (EnCodec). arxiv.org/abs/2210.13438
19. A. van den Oord, S. Dieleman, and B. Schrauwen. "Deep content-based music recommendation." Advances in Neural Information Processing Systems (NeurIPS), 2013. papers.nips.cc
20. B. McFee, C. Raffel, D. Liang, D. P. W. Ellis, M. McVicar, E. Battenberg, and O. Nieto. "librosa: audio and music signal analysis in Python." Proc. Python in Science Conference (SciPy), 2015. doi.org/10.25080/Majora-7b98e3ed-003
21. C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis. "mir_eval: a transparent implementation of common MIR metrics." Proc. International Society for Music Information Retrieval (ISMIR), 2014. ismir.net
22. C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck. "Onsets and Frames: dual-objective piano transcription." Proc. ISMIR, 2018. arxiv.org/abs/1710.11153
23. J. Serrà, E. Gómez, P. Herrera, and X. Serra. "Chroma binary similarity and local alignment applied to cover song identification." IEEE Transactions on Audio, Speech, and Language Processing 16(6), 2008. doi.org/10.1109/TASL.2008.924595
24. A. de Cheveigné and H. Kawahara. "YIN, a fundamental frequency estimator for speech and music." Journal of the Acoustical Society of America 111(4), 2002. doi.org/10.1121/1.1458024
25. J. W. Kim, J. Salamon, P. Li, and J. P. Bello. "CREPE: a convolutional representation for pitch estimation." Proc. IEEE ICASSP, 2018. arxiv.org/abs/1802.06182