Fourier Neural Operators (FNO) vs. PINNs: Zero-Shot Super-Resolution on Navier–Stokes
In Weight-Space Programming on Physics Neural Networks, we inspected how coordinate MLPs encode single PDE solutions directly inside their weight matrices. In Solving Navier–Stokes at 60 FPS, we built a semi-Lagrangian Jacobi ping-pong solver in WebGL. Both approaches share a fundamental scaling bottleneck: a Physics-Informed Neural Network (PINN) approximates a single solution function $u(x, t)$ for one fixed boundary/initial condition, requiring full retraining whenever the initial vorticity field $a(x) = u(x, 0)$ changes.
Fourier Neural Operators (FNO) (Li et al., 2021, arXiv:2010.08895) reframe scientific machine learning from finite-dimensional function regression to infinite-dimensional Banach-space operator learning $\mathcal{G}^\dagger : \mathcal{A} \to \mathcal{U}$. By parameterizing integral kernel operators in Fourier frequency space, an FNO trained solely on coarse $64 \times 64$ Navier–Stokes grids can evaluate zero-shot super-resolution on $256 \times 256$ grids without retraining – running $1000\times$ faster than pseudo-spectral PDE integrators.
0. Function Approximation (PINNs) vs. Operator Learning (FNO)
Consider the 2D incompressible Navier–Stokes equations in vorticity-streamfunction form on the periodic torus $\mathbb{T}^2 = [0, 2\pi)^2$ with kinematic viscosity $\nu$:
Whereas a PINN parameterizes $\omega_\theta : (x, y, t) \mapsto \mathbb{R}$ and differentiates through coordinate space via higher-order autograd, an FNO learns the solution operator $\mathcal{G}_\theta(\omega_0)$ mapping the entire initial field function to the evolved field function.
Why Coordinate PINNs Choke on High-Reynolds Turbulence
When optimizing the PINN PDE residual $\mathcal{L}_{\text{PDE}} = \|\partial_t \omega_\theta + u_\theta \cdot \nabla \omega_\theta - \nu \nabla^2 \omega_\theta\|_2^2$, the Laplacian operator $\nabla^2$ amplifies high-frequency modes by $|k|^4$ in the squared loss gradient. Combined with MLP spectral bias, standard coordinate networks suffer from ill-conditioned Hessian spectra on multi-scale vortex cascades.
1. The Fourier Integral Layer & Convolution Theorem
An FNO layer updates a hidden function representation $v_\ell(x) \in \mathbb{R}^{d_v}$ by combining a pointwise linear bypass $W_\ell v_\ell(x)$ with a non-local Green's kernel integral operator $(\mathcal{K}_\phi v_\ell)(x) = \int_D \kappa_\phi(x - y)\, v_\ell(y)\, dy$. Applying the convolution theorem turns the spatial integral into elementwise matrix multiplication on the truncated low-frequency Fourier modes $|k| \le k_{\max}$:
Because $R_\phi \in \mathbb{C}^{k_{\max, x} \times k_{\max, y} \times d_v \times d_v}$ is indexed strictly by integer wavevectors $k = (k_x, k_y) \in \mathbb{Z}^2$ rather than spatial grid coordinates, the learned weights are invariant to spatial grid resolution $H \times W$.
torch.fft.rfft2):
(1) cuFFT does not support bfloat16 or non-power-of-two float16 complex transforms – always wrap spectral einsums in torch.amp.autocast("cuda", enabled=False) and cast inputs to float32 before rfft2.
(2) In 2D real FFTs, only the second spatial dimension $W$ is halved ($W // 2 + 1$), while the first dimension $H$ contains both positive $[0 : k_{\max}]$ and negative $[-k_{\max} :]$ frequencies; omitting the negative $k_x$ corner slices breaks isotropic wave propagation!
2. Self-Contained PyTorch Implementation & Zero-Shot Super-Resolution Test
Below is a minimal, runnable 2D FNO spectral block in PyTorch verifying exact discretization invariance between a coarse $32 \times 32$ training grid and a $128 \times 128$ zero-shot evaluation grid:
💻 Python: 2D Spectral Convolution & Zero-Shot Super-Resolution VerificationPyTorch
import math
import torch
import torch.nn as nn
class SpectralConv2d(nn.Module):
"""Resolution-invariant 2D Fourier integral kernel operator (Li et al., arXiv:2010.08895)."""
def __init__(self, in_channels: int, out_channels: int, modes_x: int, modes_y: int) -> None:
super().__init__()
self.modes_x = modes_x
self.modes_y = modes_y
scale = 1.0 / (in_channels * out_channels)
# Positive and negative k_x corners in rfft2 spectrum: [C_in, C_out, M_x, M_y]
self.weights_pos = nn.Parameter(
scale * torch.randn(in_channels, out_channels, modes_x, modes_y, dtype=torch.cfloat)
)
self.weights_neg = nn.Parameter(
scale * torch.randn(in_channels, out_channels, modes_x, modes_y, dtype=torch.cfloat)
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
# x: [B, C_in, H, W] -> x_ft: [B, C_in, H, W // 2 + 1]
B, _, H, W = x.shape
with torch.amp.autocast(device_type=x.device.type, enabled=False):
x_f32 = x.to(torch.float32).contiguous()
# Normalize by 1/(H*W) ("forward") so Fourier coefficients are invariant to grid size (H, W)
x_ft = torch.fft.rfft2(x_f32, norm="forward") # [B, C_in, H, W // 2 + 1]
out_ft = torch.zeros(
B, self.weights_pos.size(1), H, W // 2 + 1,
dtype=torch.cfloat, device=x.device
)
# Multiply low-frequency modes along positive and negative k_x branches
out_ft[:, :, :self.modes_x, :self.modes_y] = torch.einsum(
"bixy,ioxy->boxy", x_ft[:, :, :self.modes_x, :self.modes_y], self.weights_pos
)
out_ft[:, :, -self.modes_x:, :self.modes_y] = torch.einsum(
"bixy,ioxy->boxy", x_ft[:, :, -self.modes_x:, :self.modes_y], self.weights_neg
)
# Inverse transform back to target spatial grid s=(H, W)
return torch.fft.irfft2(out_ft, s=(H, W), norm="forward").to(x.dtype) # [B, C_out, H, W]
def verify_fno_super_resolution() -> None:
torch.manual_seed(42)
spec_conv = SpectralConv2d(in_channels=4, out_channels=4, modes_x=6, modes_y=6)
# Evaluate same continuous vorticity field on 32x32 vs 128x128 periodic grids
g32 = torch.linspace(0, 2 * math.pi, 33)[:-1]
y32, x32 = torch.meshgrid(g32, g32, indexing="ij")
u32 = (torch.sin(2 * x32) * torch.cos(3 * y32)).view(1, 1, 32, 32).repeat(2, 4, 1, 1)
g128 = torch.linspace(0, 2 * math.pi, 129)[:-1]
y128, x128 = torch.meshgrid(g128, g128, indexing="ij")
u128 = (torch.sin(2 * x128) * torch.cos(3 * y128)).view(1, 1, 128, 128).repeat(2, 4, 1, 1)
out32 = spec_conv(u32) # [2, 4, 32, 32]
out128 = spec_conv(u128) # [2, 4, 128, 128] (Zero-shot 4x super-resolution!)
# Subsample 128x128 every 4th grid point to compare pointwise against 32x32
max_diff = (out32 - out128[:, :, ::4, ::4]).abs().max().item()
print(f"[PASS] Coarse 32x32 Shape: {tuple(out32.shape)} | Zero-Shot 128x128 Shape: {tuple(out128.shape)}")
print(f"[PASS] Zero-Shot 4x Super-Resolution Pointwise Max Diff: {max_diff:.2e}")
if __name__ == "__main__":
verify_fno_super_resolution()
# Output:
# [PASS] Coarse 32x32 Shape: (2, 4, 32, 32) | Zero-Shot 128x128 Shape: (2, 4, 128, 128)
# [PASS] Zero-Shot 4x Super-Resolution Pointwise Max Diff: 4.66e-08
3. PDE Solver Paradigms: Head-to-Head Comparison
| Dimension | Coordinate PINN (WSP Post) | WebGL Jacobi Solver (NS Post) | Fourier Neural Operator (FNO) |
|---|---|---|---|
| Mathematical Object Learned | Single solution function $u_\theta(x, t)$ | Direct time-stepping on fixed grid | Banach-space operator $\mathcal{G}_\theta : a \mapsto u(\cdot, T)$ |
| New Initial Condition $\omega_0(x)$ | Requires full gradient retraining | Instant (runs step-by-step in FBO) | Single forward pass ($\sim 5\,\text{ms}$) |
| Grid Discretization Invariance | Continuous $(x, t)$ coordinates | Locked to CFL timestep & $\Delta x$ | Zero-shot super-resolution ($64^2 \to 256^2$) |
| Primary Failure Mode | Spectral bias & $\nabla^2$ Hessian stiffness | Numerical viscosity from semi-Lagrangian lerp | Gibbs ringing near sharp non-periodic shocks |
| Best Production Use Case | Sparse sensor inverse problems | Interactive 60 FPS browser physics | Fast surrogate ensembles (FourCastNet weather) |