AI/ML Physics Neural Operators Navier–Stokes PyTorch

Fourier Neural Operators (FNO) vs. PINNs: Zero-Shot Super-Resolution on Navier–Stokes


In Weight-Space Programming on Physics Neural Networks, we inspected how coordinate MLPs encode single PDE solutions directly inside their weight matrices. In Solving Navier–Stokes at 60 FPS, we built a semi-Lagrangian Jacobi ping-pong solver in WebGL. Both approaches share a fundamental scaling bottleneck: a Physics-Informed Neural Network (PINN) approximates a single solution function $u(x, t)$ for one fixed boundary/initial condition, requiring full retraining whenever the initial vorticity field $a(x) = u(x, 0)$ changes.

Fourier Neural Operators (FNO) (Li et al., 2021, arXiv:2010.08895) reframe scientific machine learning from finite-dimensional function regression to infinite-dimensional Banach-space operator learning $\mathcal{G}^\dagger : \mathcal{A} \to \mathcal{U}$. By parameterizing integral kernel operators in Fourier frequency space, an FNO trained solely on coarse $64 \times 64$ Navier–Stokes grids can evaluate zero-shot super-resolution on $256 \times 256$ grids without retraining – running $1000\times$ faster than pseudo-spectral PDE integrators.


0. Function Approximation (PINNs) vs. Operator Learning (FNO)

Consider the 2D incompressible Navier–Stokes equations in vorticity-streamfunction form on the periodic torus $\mathbb{T}^2 = [0, 2\pi)^2$ with kinematic viscosity $\nu$:

2D Vorticity Transport PDE & Solution Operator Mapping
$$\frac{\partial \omega(x, t)}{\partial t} + \bigl(u(x, t) \cdot \nabla\bigr)\omega(x, t) = \nu \nabla^2 \omega(x, t) + f(x), \qquad \nabla \cdot u = 0, \qquad \omega = \nabla \times u$$ $$\mathcal{G}^\dagger : \omega(\cdot, 0) \in L^2(\mathbb{T}^2; \mathbb{R}) \;\longmapsto\; \omega(\cdot, T) \in L^2(\mathbb{T}^2; \mathbb{R})$$

Whereas a PINN parameterizes $\omega_\theta : (x, y, t) \mapsto \mathbb{R}$ and differentiates through coordinate space via higher-order autograd, an FNO learns the solution operator $\mathcal{G}_\theta(\omega_0)$ mapping the entire initial field function to the evolved field function.

Why Coordinate PINNs Choke on High-Reynolds Turbulence

When optimizing the PINN PDE residual $\mathcal{L}_{\text{PDE}} = \|\partial_t \omega_\theta + u_\theta \cdot \nabla \omega_\theta - \nu \nabla^2 \omega_\theta\|_2^2$, the Laplacian operator $\nabla^2$ amplifies high-frequency modes by $|k|^4$ in the squared loss gradient. Combined with MLP spectral bias, standard coordinate networks suffer from ill-conditioned Hessian spectra on multi-scale vortex cascades.


1. The Fourier Integral Layer & Convolution Theorem

An FNO layer updates a hidden function representation $v_\ell(x) \in \mathbb{R}^{d_v}$ by combining a pointwise linear bypass $W_\ell v_\ell(x)$ with a non-local Green's kernel integral operator $(\mathcal{K}_\phi v_\ell)(x) = \int_D \kappa_\phi(x - y)\, v_\ell(y)\, dy$. Applying the convolution theorem turns the spatial integral into elementwise matrix multiplication on the truncated low-frequency Fourier modes $|k| \le k_{\max}$:

Fourier Neural Operator Layer Update (Li et al., 2021)
$$v_{\ell+1}(x) = \sigma\!\left( W_\ell\, v_\ell(x) + \mathcal{F}^{-1}\!\Bigl( R_\phi \cdot (\mathcal{F} v_\ell) \Bigr)(x) \right), \qquad \bigl(R_\phi \cdot (\mathcal{F} v_\ell)\bigr)_{k, o} = \sum_{i=1}^{d_v} R_{\phi, k, i, o}\, (\mathcal{F} v_\ell)_{k, i}$$

Because $R_\phi \in \mathbb{C}^{k_{\max, x} \times k_{\max, y} \times d_v \times d_v}$ is indexed strictly by integer wavevectors $k = (k_x, k_y) \in \mathbb{Z}^2$ rather than spatial grid coordinates, the learned weights are invariant to spatial grid resolution $H \times W$.

Input Field v_ℓ(x) [B, C, H, W] 2D Real FFT (rfft2) Truncate |k| ≤ k_max Complex Einsum R_φ bixy, ioxy → boxy Inverse FFT (irfft2) Zero-pad to (H, W) Pointwise 1×1 Conv W_ℓ Captures non-periodic edges ⊕ → GELU v_{ℓ+1}(x)
Figure 1: Dual-branch Fourier Neural Operator block. The upper spectral pathway filters global low-frequency modes via $R_\phi$ while the lower $1 \times 1$ bypass preserves sharp local gradients and non-periodic boundary structure.
PyTorch Production Footguns (AMP & Contiguity in torch.fft.rfft2): (1) cuFFT does not support bfloat16 or non-power-of-two float16 complex transforms – always wrap spectral einsums in torch.amp.autocast("cuda", enabled=False) and cast inputs to float32 before rfft2. (2) In 2D real FFTs, only the second spatial dimension $W$ is halved ($W // 2 + 1$), while the first dimension $H$ contains both positive $[0 : k_{\max}]$ and negative $[-k_{\max} :]$ frequencies; omitting the negative $k_x$ corner slices breaks isotropic wave propagation!

2. Self-Contained PyTorch Implementation & Zero-Shot Super-Resolution Test

Below is a minimal, runnable 2D FNO spectral block in PyTorch verifying exact discretization invariance between a coarse $32 \times 32$ training grid and a $128 \times 128$ zero-shot evaluation grid:

💻 Python: 2D Spectral Convolution & Zero-Shot Super-Resolution VerificationPyTorch
import math
import torch
import torch.nn as nn

class SpectralConv2d(nn.Module):
    """Resolution-invariant 2D Fourier integral kernel operator (Li et al., arXiv:2010.08895)."""
    def __init__(self, in_channels: int, out_channels: int, modes_x: int, modes_y: int) -> None:
        super().__init__()
        self.modes_x = modes_x
        self.modes_y = modes_y
        scale = 1.0 / (in_channels * out_channels)
        # Positive and negative k_x corners in rfft2 spectrum: [C_in, C_out, M_x, M_y]
        self.weights_pos = nn.Parameter(
            scale * torch.randn(in_channels, out_channels, modes_x, modes_y, dtype=torch.cfloat)
        )
        self.weights_neg = nn.Parameter(
            scale * torch.randn(in_channels, out_channels, modes_x, modes_y, dtype=torch.cfloat)
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: [B, C_in, H, W] -> x_ft: [B, C_in, H, W // 2 + 1]
        B, _, H, W = x.shape
        with torch.amp.autocast(device_type=x.device.type, enabled=False):
            x_f32 = x.to(torch.float32).contiguous()
            # Normalize by 1/(H*W) ("forward") so Fourier coefficients are invariant to grid size (H, W)
            x_ft = torch.fft.rfft2(x_f32, norm="forward")  # [B, C_in, H, W // 2 + 1]

            out_ft = torch.zeros(
                B, self.weights_pos.size(1), H, W // 2 + 1,
                dtype=torch.cfloat, device=x.device
            )
            # Multiply low-frequency modes along positive and negative k_x branches
            out_ft[:, :, :self.modes_x, :self.modes_y] = torch.einsum(
                "bixy,ioxy->boxy", x_ft[:, :, :self.modes_x, :self.modes_y], self.weights_pos
            )
            out_ft[:, :, -self.modes_x:, :self.modes_y] = torch.einsum(
                "bixy,ioxy->boxy", x_ft[:, :, -self.modes_x:, :self.modes_y], self.weights_neg
            )
            # Inverse transform back to target spatial grid s=(H, W)
            return torch.fft.irfft2(out_ft, s=(H, W), norm="forward").to(x.dtype)  # [B, C_out, H, W]

def verify_fno_super_resolution() -> None:
    torch.manual_seed(42)
    spec_conv = SpectralConv2d(in_channels=4, out_channels=4, modes_x=6, modes_y=6)

    # Evaluate same continuous vorticity field on 32x32 vs 128x128 periodic grids
    g32 = torch.linspace(0, 2 * math.pi, 33)[:-1]
    y32, x32 = torch.meshgrid(g32, g32, indexing="ij")
    u32 = (torch.sin(2 * x32) * torch.cos(3 * y32)).view(1, 1, 32, 32).repeat(2, 4, 1, 1)

    g128 = torch.linspace(0, 2 * math.pi, 129)[:-1]
    y128, x128 = torch.meshgrid(g128, g128, indexing="ij")
    u128 = (torch.sin(2 * x128) * torch.cos(3 * y128)).view(1, 1, 128, 128).repeat(2, 4, 1, 1)

    out32 = spec_conv(u32)    # [2, 4, 32, 32]
    out128 = spec_conv(u128)  # [2, 4, 128, 128] (Zero-shot 4x super-resolution!)

    # Subsample 128x128 every 4th grid point to compare pointwise against 32x32
    max_diff = (out32 - out128[:, :, ::4, ::4]).abs().max().item()
    print(f"[PASS] Coarse 32x32 Shape: {tuple(out32.shape)} | Zero-Shot 128x128 Shape: {tuple(out128.shape)}")
    print(f"[PASS] Zero-Shot 4x Super-Resolution Pointwise Max Diff: {max_diff:.2e}")

if __name__ == "__main__":
    verify_fno_super_resolution()
# Output:
# [PASS] Coarse 32x32 Shape: (2, 4, 32, 32) | Zero-Shot 128x128 Shape: (2, 4, 128, 128)
# [PASS] Zero-Shot 4x Super-Resolution Pointwise Max Diff: 4.66e-08

3. PDE Solver Paradigms: Head-to-Head Comparison

Dimension Coordinate PINN (WSP Post) WebGL Jacobi Solver (NS Post) Fourier Neural Operator (FNO)
Mathematical Object Learned Single solution function $u_\theta(x, t)$ Direct time-stepping on fixed grid Banach-space operator $\mathcal{G}_\theta : a \mapsto u(\cdot, T)$
New Initial Condition $\omega_0(x)$ Requires full gradient retraining Instant (runs step-by-step in FBO) Single forward pass ($\sim 5\,\text{ms}$)
Grid Discretization Invariance Continuous $(x, t)$ coordinates Locked to CFL timestep & $\Delta x$ Zero-shot super-resolution ($64^2 \to 256^2$)
Primary Failure Mode Spectral bias & $\nabla^2$ Hessian stiffness Numerical viscosity from semi-Lagrangian lerp Gibbs ringing near sharp non-periodic shocks
Best Production Use Case Sparse sensor inverse problems Interactive 60 FPS browser physics Fast surrogate ensembles (FourCastNet weather)