Continuous-Time Audio Flow Matching: Vocal Seed Infilling & Phase Coherence Across Long Tracks
In Flow Matching & Diffusion Transformers (DiT), we derived why straight Optimal Transport trajectories $x_t = (1 - t)x_0 + t x_1$ dramatically outperform curved DDPM schedules. When you apply that exact same continuous-time transport PDE to music and vocal generation (Voicebox, Le et al., arXiv:2306.15687; Stable Audio Open, Evans et al., arXiv:2407.14358), a unique domain problem arises: how do you maintain vocal timbre consistency, formant phrasing, and phase-locked BPM across a 3-minute track when the underlying DiT window spans only 15 to 47 seconds?
Autoregressive audio tokenizers (like MusicGen or VALL-E) suffer from progressive acoustic drift: small sampling errors accumulate over thousands of discrete RVQ tokens, causing a vocalist's formant profile or a synthwave bassline's stereo phase to mutate mid-song. Non-autoregressive Masked Conditional Flow Matching (Infilling CFM) solves this by conditioning the velocity field $v_\theta(x_t, t \mid x_{\text{ctx}}, m)$ directly on an uncorrupted 15-second master reference seed.
0. Acoustic Drift in Autoregressive RVQ vs. Context-Conditioned Flow Infilling
Let $x_1 \in \mathbb{R}^{T \times D}$ denote a sequence of continuous stereo VAE latents (e.g., $44.1\,\text{kHz}$ audio compressed $2048\times$ to $21.5\,\text{Hz}$ latent frames in Stable Audio Open) or log-mel spectrogram frames (Voicebox). Given a binary temporal context mask $m \in \{0, 1\}^{T \times 1}$ where $m_\tau = 1$ marks locked vocal/master seed frames and $m_\tau = 0$ marks frames to be synthesized:
Crucially, self-attention inside the Diffusion Transformer attends globally across both masked ($m_\tau=0$) and unmasked ($m_\tau=1$) tokens at every ODE step, locking vocal harmonics, vibrato cadence, and room reverb tail to the reference seed.
s, t, hi-hats); rescale guided velocities using standard-deviation clamping.
1. Runnable PyTorch Reference: Masked Audio Flow Matching & Cosine-Blended Heun Solver
Below is a self-contained implementation of the masked CFM training loss and the boundary-smoothed Heun 2nd-order ODE sampler for extending a locked vocal/instrumental seed:
💻 Python: Masked Audio Flow Matching Loss & Phase-Locked Heun InfillerPyTorch
import math
import torch
import torch.nn as nn
class AudioInfillCFM(nn.Module):
"""Minimal DiT-style Masked Conditional Flow Matching module for audio latents."""
def __init__(self, d_latent: int = 16, d_model: int = 64) -> None:
super().__init__()
# Input concatenates noisy state x_t [D], locked context m * x_1 [D], and mask m [1]
self.in_proj = nn.Linear(2 * d_latent + 1, d_model)
self.attn = nn.MultiheadAttention(d_model, num_heads=4, batch_first=True)
self.out_proj = nn.Linear(d_model, d_latent)
def forward(self, x_t: torch.Tensor, x_ctx: torch.Tensor, mask: torch.Tensor, t: torch.Tensor) -> torch.Tensor:
# x_t, x_ctx: [B, T, D], mask: [B, T, 1], t: [B, 1, 1]
h = self.in_proj(torch.cat([x_t, x_ctx * mask, mask], dim=-1)) + torch.sin(t * math.pi)
h_attn, _ = self.attn(h, h, h, need_weights=False)
return self.out_proj(h + h_attn) # Predicted velocity v_theta: [B, T, D]
def masked_cfm_loss(model: nn.Module, x_1: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
# x_1: [B, T, D] target audio latents, mask: [B, T, 1] (1 = seed context, 0 = target infill)
x_0 = torch.randn_like(x_1)
t = torch.sigmoid(torch.randn(x_1.size(0), 1, 1, device=x_1.device)) # Logit-normal t in (0, 1)
x_t = (1.0 - t) * x_0 + t * x_1
v_pred = model(x_t, x_1, mask, t)
v_target = x_1 - x_0
target_region = 1.0 - mask # Evaluate MSE strictly on synthesized frames
return ((v_pred - v_target).pow(2) * target_region).sum() / (target_region.sum() * x_1.size(-1))
@torch.no_grad()
def heun_infill_sample(
model: nn.Module, x_seed: torch.Tensor, mask: torch.Tensor, steps: int = 8
) -> torch.Tensor:
"""2nd-order Heun ODE sampler with raised-cosine boundary crossfade on final 25% steps."""
B, T, _ = x_seed.shape
seed_end = int(mask[0, :, 0].sum().item())
fade_len = min(4, max(0, T - seed_end))
soft_mask = mask.clone()
if fade_len > 0:
w = 0.5 * (1.0 + torch.cos(torch.linspace(0.0, math.pi, fade_len + 2, device=x_seed.device)[1:-1]))
soft_mask[:, seed_end : seed_end + fade_len, 0] = w.unsqueeze(0)
x_t = torch.randn_like(x_seed)
dt = 1.0 / steps
for step in range(steps):
t = torch.full((B, 1, 1), step * dt, device=x_seed.device)
t_next = torch.full((B, 1, 1), min(1.0, (step + 1) * dt), device=x_seed.device)
m_blend = soft_mask if step >= int(0.75 * steps) else mask
x_in = m_blend * x_seed + (1.0 - m_blend) * x_t
v1 = model(x_in, x_seed, mask, t)
x_pred = x_in + dt * v1
v2 = model(m_blend * x_seed + (1.0 - m_blend) * x_pred, x_seed, mask, t_next)
x_t = x_in + 0.5 * dt * (v1 + v2)
return mask * x_seed + (1.0 - mask) * x_t
if __name__ == "__main__":
torch.manual_seed(42)
B, T_frames, D_lat = 2, 64, 16
x_master = torch.sin(torch.linspace(0, 4 * math.pi, T_frames)).view(1, T_frames, 1).repeat(B, 1, D_lat)
seed_mask = torch.zeros(B, T_frames, 1)
seed_mask[:, :16, :] = 1.0 # First 16 frames locked as master vocal seed
net = AudioInfillCFM(d_latent=D_lat, d_model=64)
loss = masked_cfm_loss(net, x_master, seed_mask)
infill = heun_infill_sample(net, x_master, seed_mask, steps=8)
seed_drift = (infill[:, :16] - x_master[:, :16]).abs().max().item()
print(f"[PASS] Locked Seed Frames: {int(seed_mask[0].sum().item())}/{T_frames} | Masked Infill CFM Loss: {loss.item():.4f}")
print(f"[PASS] Heun Infill Shape : {tuple(infill.shape)} | Seed Drift Max Error: {seed_drift:.2e}")
# Output:
# [PASS] Locked Seed Frames: 16/64 | Masked Infill CFM Loss: 1.7610
# [PASS] Heun Infill Shape : (2, 64, 16) | Seed Drift Max Error: 0.00e+00
2. Generative Audio Architectures: Head-to-Head
| Dimension | Autoregressive RVQ (MusicGen / VALL-E) | Score Diffusion (AudioLDM 2) | Masked Audio Flow Matching (Voicebox / Stable Audio) |
|---|---|---|---|
| Latent Representation | Discrete EnCodec / SoundStream RVQ tokens | Continuous mel-VAE latents (curved SDE) | Continuous stereo VAE / mel frames (straight OT ODE) |
| Vocal Seed Extension | Prefix-only; drifts over $\gt 30\,\text{s}$ rollouts | Requires approximate repaint score guidance | Exact bidirectional prefix + suffix infilling ($m \in \{0,1\}$) |
| Sampling Latency | $O(T_{\text{tokens}})$ sequential decoding steps | $50\text{–}100$ U-Net / DiT NFE | $16\text{–}24$ DiT Heun ODE steps across full window |