AI/ML Audio Gen Flow Matching Diffusion Transformers PyTorch

Continuous-Time Audio Flow Matching: Vocal Seed Infilling & Phase Coherence Across Long Tracks


In Flow Matching & Diffusion Transformers (DiT), we derived why straight Optimal Transport trajectories $x_t = (1 - t)x_0 + t x_1$ dramatically outperform curved DDPM schedules. When you apply that exact same continuous-time transport PDE to music and vocal generation (Voicebox, Le et al., arXiv:2306.15687; Stable Audio Open, Evans et al., arXiv:2407.14358), a unique domain problem arises: how do you maintain vocal timbre consistency, formant phrasing, and phase-locked BPM across a 3-minute track when the underlying DiT window spans only 15 to 47 seconds?

Autoregressive audio tokenizers (like MusicGen or VALL-E) suffer from progressive acoustic drift: small sampling errors accumulate over thousands of discrete RVQ tokens, causing a vocalist's formant profile or a synthwave bassline's stereo phase to mutate mid-song. Non-autoregressive Masked Conditional Flow Matching (Infilling CFM) solves this by conditioning the velocity field $v_\theta(x_t, t \mid x_{\text{ctx}}, m)$ directly on an uncorrupted 15-second master reference seed.


0. Acoustic Drift in Autoregressive RVQ vs. Context-Conditioned Flow Infilling

Let $x_1 \in \mathbb{R}^{T \times D}$ denote a sequence of continuous stereo VAE latents (e.g., $44.1\,\text{kHz}$ audio compressed $2048\times$ to $21.5\,\text{Hz}$ latent frames in Stable Audio Open) or log-mel spectrogram frames (Voicebox). Given a binary temporal context mask $m \in \{0, 1\}^{T \times 1}$ where $m_\tau = 1$ marks locked vocal/master seed frames and $m_\tau = 0$ marks frames to be synthesized:

Context-Preserving Masked Conditional Flow Matching Objective (Le et al., 2023)
$$x_t = (1 - t)\, x_0 + t\, x_1, \qquad x_{\text{cond}} = m \odot x_1, \qquad u_t(x_t \mid x_0, x_1) = x_1 - x_0$$ $$\mathcal{L}_{\text{Infill-CFM}}(\theta) = \mathbb{E}_{t, x_0, x_1, m}\!\left[ \frac{1}{\sum_{\tau=1}^T (1 - m_\tau) D} \sum_{\tau=1}^{T} (1 - m_\tau)\, \Bigl\| v_\theta\bigl([x_t;\, m \odot x_1;\, m],\, t,\, c_{\text{text}}\bigr)_\tau - (x_{1,\tau} - x_{0,\tau}) \Bigr\|_2^2 \right]$$

Crucially, self-attention inside the Diffusion Transformer attends globally across both masked ($m_\tau=0$) and unmasked ($m_\tau=1$) tokens at every ODE step, locking vocal harmonics, vibrato cadence, and room reverb tail to the reference seed.

Locked Master Seed (m = 1) 0.0s – 15.0s Reference Anchor Fixed Formant & Synth Timbre Active ODE Infill Window (m = 0) dx_t / dt = v_θ([x_t, m⊙x₁, m], t, c_lyrics) Cross-attends to Seed (m=1) + Lyric Phonemes Phase-Aligned Suffix (m = 1) Optional Beat/Hook Outbound Lock Zero Boundary Click / Tempo Jump
Figure 1: Anchor-seeded audio flow matching. Keeping the 15-second master vocal/synth reference unmasked ($m=1$) forces the DiT velocity field to transport Gaussian noise in the $m=0$ region onto the exact same harmonic manifold.
Audio Flow Matching Boundary Footguns: (1) Hard-Mask Boundary Transients: Replacing $x_t$ with $m \odot x_1 + (1 - m) \odot x_t$ via a step-function binary mask at inference creates a 1-frame phase discontinuity at the $15.0\,\text{s}$ splice point – apply a raised-cosine crossfade mask $m_\tau \in [0, 1]$ over 4–8 latent frames ($\sim 200\,\text{ms}$) during the final 25% of Heun ODE steps. (2) CFG Over-Saturation in Mel Space: Classifier-Free Guidance scales $w_{\text{cfg}} \gt 4.0$ in audio flow matching clip high-frequency sibilants (s, t, hi-hats); rescale guided velocities using standard-deviation clamping.

1. Runnable PyTorch Reference: Masked Audio Flow Matching & Cosine-Blended Heun Solver

Below is a self-contained implementation of the masked CFM training loss and the boundary-smoothed Heun 2nd-order ODE sampler for extending a locked vocal/instrumental seed:

💻 Python: Masked Audio Flow Matching Loss & Phase-Locked Heun InfillerPyTorch
import math
import torch
import torch.nn as nn

class AudioInfillCFM(nn.Module):
    """Minimal DiT-style Masked Conditional Flow Matching module for audio latents."""
    def __init__(self, d_latent: int = 16, d_model: int = 64) -> None:
        super().__init__()
        # Input concatenates noisy state x_t [D], locked context m * x_1 [D], and mask m [1]
        self.in_proj = nn.Linear(2 * d_latent + 1, d_model)
        self.attn = nn.MultiheadAttention(d_model, num_heads=4, batch_first=True)
        self.out_proj = nn.Linear(d_model, d_latent)

    def forward(self, x_t: torch.Tensor, x_ctx: torch.Tensor, mask: torch.Tensor, t: torch.Tensor) -> torch.Tensor:
        # x_t, x_ctx: [B, T, D], mask: [B, T, 1], t: [B, 1, 1]
        h = self.in_proj(torch.cat([x_t, x_ctx * mask, mask], dim=-1)) + torch.sin(t * math.pi)
        h_attn, _ = self.attn(h, h, h, need_weights=False)
        return self.out_proj(h + h_attn)  # Predicted velocity v_theta: [B, T, D]

def masked_cfm_loss(model: nn.Module, x_1: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
    # x_1: [B, T, D] target audio latents, mask: [B, T, 1] (1 = seed context, 0 = target infill)
    x_0 = torch.randn_like(x_1)
    t = torch.sigmoid(torch.randn(x_1.size(0), 1, 1, device=x_1.device))  # Logit-normal t in (0, 1)
    x_t = (1.0 - t) * x_0 + t * x_1
    v_pred = model(x_t, x_1, mask, t)
    v_target = x_1 - x_0
    target_region = 1.0 - mask  # Evaluate MSE strictly on synthesized frames
    return ((v_pred - v_target).pow(2) * target_region).sum() / (target_region.sum() * x_1.size(-1))

@torch.no_grad()
def heun_infill_sample(
    model: nn.Module, x_seed: torch.Tensor, mask: torch.Tensor, steps: int = 8
) -> torch.Tensor:
    """2nd-order Heun ODE sampler with raised-cosine boundary crossfade on final 25% steps."""
    B, T, _ = x_seed.shape
    seed_end = int(mask[0, :, 0].sum().item())
    fade_len = min(4, max(0, T - seed_end))
    soft_mask = mask.clone()
    if fade_len > 0:
        w = 0.5 * (1.0 + torch.cos(torch.linspace(0.0, math.pi, fade_len + 2, device=x_seed.device)[1:-1]))
        soft_mask[:, seed_end : seed_end + fade_len, 0] = w.unsqueeze(0)

    x_t = torch.randn_like(x_seed)
    dt = 1.0 / steps
    for step in range(steps):
        t = torch.full((B, 1, 1), step * dt, device=x_seed.device)
        t_next = torch.full((B, 1, 1), min(1.0, (step + 1) * dt), device=x_seed.device)
        m_blend = soft_mask if step >= int(0.75 * steps) else mask
        x_in = m_blend * x_seed + (1.0 - m_blend) * x_t

        v1 = model(x_in, x_seed, mask, t)
        x_pred = x_in + dt * v1
        v2 = model(m_blend * x_seed + (1.0 - m_blend) * x_pred, x_seed, mask, t_next)
        x_t = x_in + 0.5 * dt * (v1 + v2)
    return mask * x_seed + (1.0 - mask) * x_t

if __name__ == "__main__":
    torch.manual_seed(42)
    B, T_frames, D_lat = 2, 64, 16
    x_master = torch.sin(torch.linspace(0, 4 * math.pi, T_frames)).view(1, T_frames, 1).repeat(B, 1, D_lat)
    seed_mask = torch.zeros(B, T_frames, 1)
    seed_mask[:, :16, :] = 1.0  # First 16 frames locked as master vocal seed

    net = AudioInfillCFM(d_latent=D_lat, d_model=64)
    loss = masked_cfm_loss(net, x_master, seed_mask)
    infill = heun_infill_sample(net, x_master, seed_mask, steps=8)
    seed_drift = (infill[:, :16] - x_master[:, :16]).abs().max().item()
    print(f"[PASS] Locked Seed Frames: {int(seed_mask[0].sum().item())}/{T_frames} | Masked Infill CFM Loss: {loss.item():.4f}")
    print(f"[PASS] Heun Infill Shape : {tuple(infill.shape)} | Seed Drift Max Error: {seed_drift:.2e}")
# Output:
# [PASS] Locked Seed Frames: 16/64 | Masked Infill CFM Loss: 1.7610
# [PASS] Heun Infill Shape : (2, 64, 16) | Seed Drift Max Error: 0.00e+00

2. Generative Audio Architectures: Head-to-Head

Dimension Autoregressive RVQ (MusicGen / VALL-E) Score Diffusion (AudioLDM 2) Masked Audio Flow Matching (Voicebox / Stable Audio)
Latent Representation Discrete EnCodec / SoundStream RVQ tokens Continuous mel-VAE latents (curved SDE) Continuous stereo VAE / mel frames (straight OT ODE)
Vocal Seed Extension Prefix-only; drifts over $\gt 30\,\text{s}$ rollouts Requires approximate repaint score guidance Exact bidirectional prefix + suffix infilling ($m \in \{0,1\}$)
Sampling Latency $O(T_{\text{tokens}})$ sequential decoding steps $50\text{–}100$ U-Net / DiT NFE $16\text{–}24$ DiT Heun ODE steps across full window