Weight-Space Programming on Physics Neural Networks: Can an LLM Read and Write PDE Weights?
Yesterday I read Sparks of Weight-Space Programming by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab, and immediately wanted to test the same idea on physics networks. Their headline result was wild: frontier LLMs reading raw neural weights without forward passes, writing MinAtar and Go agents from scratch without gradients, and patching 0.016% of Qwen3.5-0.8B to fix reasoning loops.
Physics-Informed Neural Networks (PINNs) are a much cleaner laboratory for testing whether an LLM genuinely understands weight-space geometry. Unlike a 270M-parameter transformer, a standard 1D PDE solver (2 -> 32 -> 32 -> 1 with tanh) has only 1,185 parameters, smooth differential structure, and exact analytical solutions to score against. So I wired up a three-part benchmark against Gemini 4 Carbon (with automatic fallback to Gemini 4 Argon on 180s timeouts) to see where native weight-space programming actually works β and where it breaks.
0. Experimental Setup & Baselines
Every task uses the exact same PhysicsMLP architecture: Linear(2, 32) -> Tanh -> Linear(32, 32) -> Tanh -> Linear(32, 1), taking $[x, t] \in [0, 1]^2$ and outputting scalar $u(x, t)$. All 1,185 weights easily fit inside context as raw JSON.
For the gradient-descent baseline, I trained each network on CPU for 1,500 Adam steps (512 interior collocation points, 128 IC points, and 128 BC points per step). Notice that 1,500 steps of Adam already gets wave down to 0.025 relative $L^2$ error, while advection still struggles at 0.231 because periodic travelling waves are notoriously finicky for small tanh MLPs early in training.
1. Writing Weights Zero-Shot: Direct JSON vs. Analytical Compiler
I tested two strict zero-shot writing protocols (zero training data, zero gradients, zero forward passes, and AST-verified to ban lstsq, polyfit, minimize, or autograd):
- Direct JSON (
zero_shot_json): Emit all 1,185 floating-point numbers directly in one JSON response. - Analytical Weight Compiler (
zero_shot_code): Write a puremath/numpyPython functionbuild_weights()that derives the weights analytically from the PDE's structure.
| PDE Task | Method | Model / Steps | Rel. $L^2$ Error | MAE | PDE Residual MSE |
|---|---|---|---|---|---|
| 1D Advection | Adam PINN | 1,500 steps (7.4s) | 0.231492 | 0.118659 | 0.057022 |
| 1D Advection | Direct JSON | Gemini 4 Argon (53.8s) | 1.393939 | 0.822975 | 0.000000 |
| 1D Advection | Analytical Compiler | Gemini 4 Argon (23.7s) | 0.001483 | 0.000856 | 0.000000 |
| 1D Wave | Adam PINN | 1,500 steps (16.0s) | 0.025226 | 0.010264 | 0.002480 |
| 1D Wave | Direct JSON | Gemini 4 Argon (54.5s) | 2.834325 | 1.271076 | 0.000000 |
| 1D Wave | Analytical Compiler | Gemini 4 Argon (24.8s) | 1.125316 | 0.442824 | 0.000000 |
| 1D Heat | Adam PINN | 1,500 steps (26.6s) | 0.045560 | 0.019577 | 0.069039 |
| 1D Heat | Direct JSON | Gemini 4 Argon (52.0s) | 1.992105 | 0.924512 | 1046.2275 |
| 1D Heat | Analytical Compiler | Gemini 4 Argon (28.7s) | 0.715262 | 0.325082 | 0.199918 |
Look at that advection compiler result: 0.001483 relative $L^2$ error and 0.000000 PDE residual β 156x more accurate than 1,500 steps of Adam, with zero data and zero gradients. What made it work is genuinely clever. Instead of hand-guessing numbers, Gemini 4 Argon noticed that $\frac{d}{dz}[\frac{1}{2}\tanh(\alpha(z - z_k))] = \frac{\alpha}{2}\operatorname{sech}^2(\alpha(z - z_k))$ acts as a smooth Dirac delta along the characteristic coordinate $z = x - 0.5t$. It then computed the exact Fourier transform of $\operatorname{sech}^2$ and divided its quadrature weights by $\hat K(2\pi)$ to deconvolve the smoothing kernel!
π» Python: Gemini 4 Argon's Deconvolved sechΒ² Quadrature Compiler for 1D AdvectionPython
import math
import numpy as np
def build_weights() -> dict[str, list]:
N = 32
z_min, z_max = -0.65, 1.15
dz = (z_max - z_min) / (N - 1)
z_nodes = [z_min + i * dz for i in range(N)]
alpha = 1.4 / dz
w0 = np.zeros((N, 2), dtype=np.float64)
b0 = np.zeros(N, dtype=np.float64)
for k in range(N):
w0[k, 0] = alpha
w0[k, 1] = -0.5 * alpha
b0[k] = -alpha * z_nodes[k]
# Deconvolve finite-width sech^2 kernel at frequency omega = 2*pi
omega = 2.0 * math.pi
xi = (math.pi * omega) / (2.0 * alpha)
deconv_factor = math.sinh(xi) / xi
c = np.zeros(N, dtype=np.float64)
for k in range(N):
weight_factor = 0.5 if (k == 0 or k == N - 1) else 1.0
c[k] = 0.5 * omega * math.cos(omega * z_nodes[k]) * dz * weight_factor * deconv_factor
val_at_quarter = sum(c[k] * math.tanh(alpha * (0.25 - z_nodes[k])) for k in range(N))
C_offset = 1.0 - val_at_quarter
# Distributed linear pass-through in Layer 2 via small-signal scaling eps
eps = 1e-3
w2 = np.zeros((N, N), dtype=np.float64)
b2 = np.zeros(N, dtype=np.float64)
for j in range(N):
w2[j, :] = eps * c
b2[j] = eps * C_offset
w4 = np.full((1, N), 1.0 / (N * eps), dtype=np.float64)
b4 = np.zeros(1, dtype=np.float64)
return {
"net.0.weight": w0.tolist(), "net.0.bias": b0.tolist(),
"net.2.weight": w2.tolist(), "net.2.bias": b2.tolist(),
"net.4.weight": w4.tolist(), "net.4.bias": b4.tolist(),
}
The failures on wave and heat are just as revealing. Notice how wave got pde_residual_mse = 0.0 in both Direct JSON and Compiler mode yet scored l2_rel_error = 1.125. Why? Because any sum $f(x - t) + g(x + t)$ satisfies $u_{tt} - u_{xx} = 0$ identically by d'Alembert's theorem β the model locked onto the exact characteristic rays $w_t = \pm w_x$, then flipped a sign when matching the boundary/initial amplitude. On heat, it even derived a 4-point second-difference bilinear multiplier around the inflection bias $b_{\text{opt}} = \operatorname{asinh}(1/\sqrt{2})$ (where $\tanh'''(b_{\text{opt}}) = 0$) to multiply $f(x) \cdot g(t)$ inside a tanh layer, though its hand-estimated temporal decay knots drifted (0.715 error). Direct JSON without code fared worst across the board: LLMs reason far better when compiling weights symbolically in Python than when emitting 1,185 raw floats as tokens.
2. Read-Only Weight Forensics (Zero Forward Passes)
Next, I flipped the problem around. I took the three Adam-trained PINNs, stripped all names and metadata, randomly permuted the hidden neurons in layers 0 and 2 (which preserves $f(x, t)$ while destroying neuron ordering), and handed model_alpha, model_beta, and model_gamma to Gemini 4 Carbon with strict instructions: zero forward passes, classify which weights belong to Heat, Wave, and Advection, and cite the mechanistic evidence.
Gemini 4 Carbon nailed 3/3 (100% accuracy, 0.99 confidence on all three) in 127.7 seconds. Even with randomly shuffled neurons, the PDE physics leaves an unmistakable geometric fingerprint in $(w_x, w_t)$:
| Mystery File | Predicted (Conf.) | Ground Truth | Key Weight-Space Signature Cited by Gemini 4 Carbon |
|---|---|---|---|
model_alpha |
Heat (0.99) | Heat | Stationary spatial boundary detectors ($w_t \approx 0$), asymmetric temporal damping units, and high Layer-2 spatial gain $(W_2 W_0)_x \approx +9.7 \approx 3\pi$ for $\sin(3\pi x)$. |
model_beta |
Wave (0.99) | Wave | Bidirectional d'Alembert characteristic families in Layer 0 and $W_2 W_0$: paired ratios $w_t / w_x \approx +1$ ($x + t$) and $w_t / w_x \approx -1$ ($x - t$). |
model_gamma |
Advection (0.99) | Advection | Coherent unidirectional ridge: active $W_2 W_0$ rows cluster tightly around slope $w_t / w_x \approx -0.5$ with spatial wavenumber $\approx 2\pi \approx 6.28$. |
You can inspect this directly in the interactive lab above: switch between 1D Advection and 1D Wave and watch how the Adam-trained Layer-0 weight vectors align along the theoretical characteristic lines $w_t / w_x = -0.5$ and $w_t / w_x = \pm 1$.
3. Surgical Weight-Space Patching (And Where It Struggled)
Finally, I tested whether an LLM could repair a broken PINN in one shot by editing strictly fewer than 5% of weights ($\lt 59$ of 1,185 parameters). I injected a localised defect into the clean Adam wave PINN (l2_rel_error = 0.0252): I shifted net.4.bias by +0.55 and inverted-and-doubled net.0.weight[idx, 1] ($w_t$) on the two largest-norm Layer-0 neurons, blowing the $L^2$ error up to 5.2504. Then I gave the corrupted weights plus a single $9 \times 3$ diagnostic snapshot to the model.
| Model State | Params Edited | Rel. $L^2$ Error | MAE | PDE Residual MSE |
|---|---|---|---|---|
Clean Adam PINN (wave) |
β | 0.025226 | 0.010264 | 0.002480 |
Corrupted (+0.55 bias, 2 flipped $w_t$) |
3 / 1,185 (0.253%) | 5.250414 | 2.462109 | 7.258197 |
| Patched (Gemini 4 Argon, 7.2s) | 8 / 1,185 (0.675%) | 3.668127 | 1.524339 | 5.431135 |
Gemini 4 Carbon timed out at 180s here, and the fast fallback (Gemini 4 Argon, 7.23s) immediately spotted and reversed the exact +0.55 shift in net.4.bias (-0.044), cutting the error from 5.25 down to 3.67. However, without running intermediate probes (like the 10β50 forward passes Takehi et al. allowed in their Qwen3.5-0.8B experiment), it misattributed the remaining temporal asymmetry to seven weights in net.4.weight instead of tracing the defect upstream to the two flipped net.0.weight[:, 1] rows.
Takeaways for Weight-Space Programming
Even on a tiny 1,185-parameter network, three clear rules stood out from running this end-to-end:
- Symbolic compilers beat raw weight tokens. Asking an LLM to emit 1,185 floats directly in JSON fails badly, whereas letting it write a 40-line
numpycompiler produced0.001483$L^2$ error on 1D advection β beating 1,500 Adam steps by two orders of magnitude. - Reading weights is already surprisingly reliable. Identifying Heat vs. Wave vs. Advection with 3/3 accuracy from shuffled raw matrices works because PDE symmetries and characteristic velocities ($w_t / w_x$) live directly in the geometry of $W_0$ and $W_2 W_0$.
- One-shot patching breaks when defects sit behind mixing layers. Detecting an output bias offset is trivial from one snapshot, but localising a flipped neuron two layers deep without any forward-pass budget is where zero-feedback debugging hits a wall.
References
- Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab, Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights (2026).
- Raissi, Perdikaris, and Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations (2019).