Executive summary
- The data resolve a one-fifth toroidal domain. The physical mode convention is n=5k, where k is the Fourier harmonic in the simulated wedge. Physical n=35 remains measurable and temporally coherent; the low-coherence tail begins near n≥80.
- f8 is an effective aggregate-error baseline with a known spectral limit. Its 11-cell toroidal latent can represent at most k=5, or n=25. z44 reconstructs the n=35 edge component. z22 has enough nominal Nyquist capacity for n=35 but has neither demonstrated that reconstructed peak nor completed a matched held-out rollout gate.
- Rollout-CRPS fine-tuning produced a modest checkpoint-level error improvement, not calibrated ensembles. Epoch 4 improves mean held-out free error by 6.8% and anchored error by 2.7% relative to the parent. Held-out free SSR increases from about 0.328 to 0.376, still far below the target value 1.
- ETKF conditioning can reduce rollout error. φ-bearing idealized proxy suites rank strongly for the baseline f8 prior, but φ is gauge-dependent and the current channels are not complete diagnostic forward models. The rollout-CRPS checkpoint has not undergone the 11-suite ranking.
- The emulator is not yet established as transport-faithful. Aggregate RMSE, spectral content, calibration, cross-field phase, radial flux and blob propagation are distinct requirements. The transport-sensitive battery remains mixed, so diagnostic rankings should be treated as model-conditional hypotheses.
01Dataset, cadence and toroidal modes
The study uses two simulations in one operating regime. Shot 85604 supplies training and same-shot validation windows; shot 85606 supplies a held-out temporal test. The stored frame interval is 3.1319 μs. This supports within-regime short-horizon tests, but it does not establish robustness across equilibria, forcing conditions or discharge regimes.
What the data can test
Short-horizon stochastic rollout skill; temporal transfer to shot 85606; codec fidelity; mode-by-mode predictability; and the response of an emulator ensemble to idealized sensor information.
What remains unidentified
Cross-regime generalization; real-instrument performance; uncertainty under distribution shift; and whether a sensor ranking persists across equilibria, noise levels and forward models.
Toroidal coordinates
The simulated interval is 2π/5 of the full torus and is repeated five times. Therefore only physical toroidal modes n=0,5,10,… occur:
For a field's wedge Fourier coefficient Fk(t,x,y), lag-1 complex coherence measures persistence of complex phase and amplitude direction between adjacent stored frames. The temporal mean is removed before the toroidal FFT:
The non-axisymmetric power share pk is the energy in one wedge harmonic divided by the energy in all positive harmonics:
Coherence and power answer different questions: a small mode can remain predictable, while an energetic mode can lose exact phase.
| field | n=35 coherence | n=35 power | n=20–35 coherence | n=20–35 power | n≥80 coherence | n≥80 power |
|---|---|---|---|---|---|---|
| Ne | 0.325 | 2.42% | 0.576 | 64.59% | 0.008 | 0.35% |
| Te | 0.316 | 1.53% | 0.539 | 54.33% | 0.011 | 0.18% |
| phi | 0.311 | 1.65% | 0.557 | 44.47% | 0.008 | 0.28% |
| Vort | 0.110 | 5.12% | 0.281 | 37.97% | 0.007 | 12.29% |
The practical target suggested by these truth statistics is phase-sensitive prediction through approximately k=7–8 (n≈35–40), together with statistically faithful treatment of the low-coherence tail beginning around k=16 (n≈80). That tail cannot simply be discarded: vorticity retains 12.3% of its non-axisymmetric power there.
02Emulator pipeline and representation choices
LOLA is a latent diffusion emulator. An autoencoder (AE) compresses a five-field 3D state into a smaller latent grid. A masked diffusion transformer samples future latent frames conditional on one or more known context frames. The decoder returns the sampled latent trajectory to field space. Independent Gaussian diffusion noise produces different ensemble members; in that precise sense, the model is still “rolling dice.”
The fields are electron density Ne, electron temperature Te, ion temperature Ti, electrostatic potential φ and parallel ion velocity Vi. f8, z22 and z44 are experiment labels for codecs with 11, 22 and 44 toroidal latent cells. Their direct real-FFT Nyquist ceilings are k=5, 11 and 22, or physical n=25, 55 and 110.
| variant | latent z cells | latent Nyquist ceiling | dominant reconstructed Ne edge mode | AE reconstruction VRMSE | held-out rollout status |
|---|---|---|---|---|---|
| f8 | 11 | k=5 / n=25 | n=10 | 0.0366 | held-out scored |
| f8-short | 11 | k=5 / n=25 | n=10 | 0.0366 | held-out scored |
| z22 | 22 | k=11 / n=55 | n=25 | 0.0322 | not rollout-scored |
| z44 | 44 | k=22 / n=110 | n=35 | 0.0392 | held-out scored |
AE reconstruction VRMSE is computed in physical field units for each field and frame, then averaged:
The “dominant reconstructed Ne edge mode” is the largest positive non-axisymmetric toroidal component in reconstructed edge density. It is an empirical content check, distinct from the Nyquist ceiling, which is only a mathematical capacity limit.
03Two LOLA training procedures
Procedure A: masked-denoising continuation
Jobs 6791660 (z22) and 6791711 (f8 equal-budget control) continue training from an existing best checkpoint. “Continuation” means the denoiser weights begin from a pretrained run rather than random initialization; the script creates a new optimizer and learning-rate schedule. Each example contains five consecutive frames. A contiguous prefix of one to four frames is marked as context, and the model learns to denoise the remaining latent cells at a random diffusion time.
With clean latent trajectory x, Gaussian noise z, diffusion time t uniformly sampled on [0,1], and schedule coefficients αt and σt, the corrupted input and denoising objective are
Here m is the context mask, μθ is the denoiser mean prediction, vt,i is the fixed schedule variance used as a weight, and stopgrad means that weight receives no gradient. Context cells are pinned to √(αt²+σt²)x. These runs use no CRPS or spectral term. Both use 40 continuation epochs, 1,024 training examples per epoch, batch 8, AdamW at 5×10−5 with cosine decay, and gradient clipping at 1.
| Figure 4 curve | formal meaning | appropriate interpretation |
|---|---|---|
| training mean DenoiserLoss | Arithmetic mean of 128 stochastic batch losses in one epoch. | Whether optimization is reducing the masked-denoising objective on sampled training windows. |
| validation mean DenoiserLoss | Arithmetic mean of 32 shuffled validation-batch losses with fresh masks, t and z. | A same-shot one-window generalization diagnostic; not a deterministic curve and not a rollout score. |
| red minimum | Epoch with the smallest logged validation DenoiserLoss. | The checkpoint saved as state_best under this selector. |
Procedure B: sampled rollout-CRPS fine-tuning
Job 6791994 initializes the denoiser from the pretrained f8 state_best, freezes the autoencoder and updates the entire denoiser at a lower learning rate of 10−5. This is a fine-tune: it asks whether changing the training objective can reshape an existing sampler while holding the representation fixed. A from-scratch retrain would also change initialization and representation learning, making that narrow causal comparison harder.
Each training example samples M=4 diffusion members. Every member uses two composed five-frame windows with one-frame overlap: four new frames per block and eight predicted frames total. Each block uses 16 reverse-diffusion steps. Gradients pass through every reverse step and through the carried state between blocks. The 40 epochs contain 128 examples each, with batch 2 and four-step gradient accumulation for an effective batch of 8; toroidal-roll augmentation is enabled.
For predictions x(m), truth y, ensemble size M and α=0.95, define ε=(1−α)/M=0.0125. The index i below runs over scored latent cells:
A rewards accuracy; D rewards ensemble separation; fair CRPS uses ε=0. bandCRPS applies a toroidal FFT, filters a band, divides by that truth band's detached RMS, evaluates afCRPS and averages bands. The requested edges [1,6,16] meet an f8 real-FFT grid containing only k=0,…,5, so the effective non-axisymmetric band is k∈[1,6). This objective cannot directly score k=7 (n=35) because that component is absent from the f8 latent.
| Figure 5 quantity | definition and evaluation set |
|---|---|
| training objective | Epoch mean of the optimized M=4, eight-frame ℒRCRPS on augmented training examples. |
| validation fair CRPS | afCRPS with ε=0 on a fixed unaugmented shot-85604 validation slice, M=8 and identical sampling seeds each epoch. |
| validation rollout RMSE | Square root of the mean squared difference between the M=8 latent ensemble mean and latent truth on the same fixed slice. |
| validation SSR | Mean validation-batch spread divided by mean validation-batch RMSE. This aggregate estimator is specific to Figure 5. |
04Held-out rollout accuracy and calibration
All rollout errors below are measured in standardized model space: Ne is log-transformed and then standardized; Te, Ti, φ and Vi are standardized directly. Each standardized field and grid cell receives equal weight. These values are dimensionless and should not be read as physical-unit errors.
For an M-member ensemble at forecast frame t, the evaluator uses ensemble-mean error Et, ensemble spread St, spread-to-error ratio SSR and anchoring gain ρ:
SSR=1 means spread and ensemble-mean error have the same average magnitude under this estimator; SSR<1 is underdispersion. ρ>1 means anchoring lowers mean error. Figure 5 uses a different aggregate validation SSR estimator, so its ≈0.7 values are not a before/after pair with the ≈0.3–0.4 held-out framewise estimator.
| checkpoint | validation selection rule | held-out free RMSE | held-out anchored RMSE | free change vs parent | anchored change vs parent |
|---|---|---|---|---|---|
| parent f8 | pretrained state_best | 0.3024 | 0.2693 | reference | reference |
| epoch 7 | minimum validation fair CRPS | 0.2880 | 0.2707 | +4.8% | -0.5% |
| epoch 18 | maximum validation gain ρ | 0.3003 | 0.2744 | +0.7% | -1.9% |
| epoch 4 | minimum validation anchored RMSE | 0.2818 | 0.2620 | +6.8% | +2.7% |
The epoch-4 result is modest but real under this gate: all three starts contribute to the means, and both free and anchored error improve relative to the parent. It is not the checkpoint chosen by minimum validation fair CRPS, and all evaluated checkpoints remain underdispersed. The training objective decreases from 0.2416 to 0.2246, while fixed-slice validation fair CRPS and validation SSR do not improve over the parent. The evidence therefore supports a useful early fine-tune checkpoint, but not a general calibration solution.
05ETKF conditioning and sensor experiments
The ensemble transform Kalman filter (ETKF) is a data-assimilation method. At each analysis time it compares synthetic observations from each forecast member with an observed synthetic value. Ensemble anomalies estimate the covariance between measured channels and every latent-state direction. The observation innovation is then mapped into a state update. Multiplicative inflation expands forecast anomalies before the analysis; M is ensemble size and H is rollout horizon in frames.
In compact form, the ensemble-mean update has the Kalman structure
Superscripts f and a denote forecast and analysis, h is the synthetic observation operator, y is the observation and K is estimated from the ensemble covariance and assumed observation error. Narrow or misoriented ensemble covariance can therefore produce an early update that raises global error even when later updates help. This behavior diagnoses covariance, localization, observation-error or forward-model mismatch; it does not by itself imply that early sensor samples should be removed.
Figure 9 is a baseline model-conditional information ranking. The potential φ is gauge-dependent: replacing φ(x) with φ(x)+C leaves the electric field and E×B velocity unchanged because their dynamics depend on spatial gradients. A physical follow-up should use a declared gauge, assimilate φ anomalies or gradients, or implement a floating-potential forward model with realistic transfer and noise. The localization settings kmax=2 and 5 correspond to physical n≤10 and n≤25; a test that includes n=35 requires kmax≥7 and a representation capable of k=7.
06Physics validity beyond aggregate error
Low standardized RMSE is useful but insufficient for the research goal. A trustworthy edge/SOL emulator should also preserve the relationships that drive transport: spectral power in the coherent band, density–potential phase, radial E×B particle flux, intermittency and the direction and speed of localized structures.
Toroidal mode versus blob
n=35 is one Fourier component, whereas a localized blob is broadband. Blob fidelity is tested through coherent radial propagation and the coupled Ne–φ structure that sets E×B motion, not by the presence of one toroidal mode alone.
Unpredictable phase versus valid statistics
For the n≥80 tail, a stochastic emulator may be allowed to lose exact future phase if it reproduces conditional power, intermittency, dependence and transport contribution. Vorticity makes those tail statistics consequential.
07Evidence table and proposed sequence
| topic | supported statement | evidence limit / next decision |
|---|---|---|
| Toroidal learnability | n=35 is present and coherent; the low-coherence tail begins near n≥80. | Repeat the truth-only calculation on shot 85606 and any additional operating points. |
| f8 representation | Efficient baseline with competitive aggregate RMSE and direct ceiling n=25. | Cannot directly represent n=35; use as baseline and assimilation sandbox. |
| z44 representation | Reconstructs the n=35 edge peak. | Its free/anchored aggregate gate is weaker than f8 and it allocates capacity into the low-coherence tail. |
| z22 representation | Nominal ceiling n=55; trained continuation completed. | No held-out rollout gate, and reconstructed edge Ne peaks at n=25. It remains an untested middle candidate. |
| rollout CRPS | Epoch 4 modestly improves held-out free and anchored RMSE. | Underdispersion persists; fair-CRPS validation does not select that checkpoint; the full sensor ranking was not repeated. |
| φ-bearing suites | Strongest baseline-f8 idealized interventions in the three-start factorial. | Needs a gauge-safe, noisy physical forward model and replication with a calibrated prior. |
| transport fidelity | Some coarse gates pass, but cross-phase/flux and blob-motion tests are mixed or failing. | These are acceptance gates before diagnostic-design conclusions. |
- Complete the missing comparison. Evaluate z22 job 6791660 on shot 85606 with the same three starts, M=64, H=48 and stated ETKF protocol.
- Adopt an explicit codec gate. Require edge/SOL reconstruction and phase fidelity through k=7 (n=35), alongside physical-unit reconstruction error.
- Build a middle, band-aware representation. Test 16–22 toroidal latent cells or an explicit spectral branch, with reconstruction weight through k=7–8; use z44 as an upper-capacity reference.
- Separate predictable and stochastic targets. Use phase-sensitive rollout losses through n≈35–40, and distributional scores for the n≥80 tail, including vorticity statistics.
- Align selection with deployment. Evaluate checkpoints on absolute held-out free and anchored errors, calibration and physics gates; do not select on gain alone.
- Upgrade sensor realism. Declare observation operators, gauge handling, transfer functions, spatial averaging, correlations and noise for GPI, reflectometry and target probes.
- Expand the data. Add shots or operating points before interpreting architecture or diagnostic rankings as robust rather than within-regime.
08Definitions and reading guide
| term | definition in this report |
|---|---|
| AE / codec | Autoencoder: encoder plus decoder used to move between five physical fields and a compressed latent grid. |
| LOLA | Latent diffusion emulator that samples future latent trajectories conditional on context frames. |
| epoch | One configured training unit: 1,024 examples for Figure 4 runs; 128 rollout examples for job 6791994. |
| fine-tune | Continue from pretrained weights at a lower learning rate while holding the f8 autoencoder fixed; not a from-scratch retrain. |
| free / anchored rollout | Free uses only the emulator after the initial context. Anchored applies ETKF analyses using synthetic observations every four generated frames. |
| ensemble mean / spread | Mean prediction across M sampled members / RMS of their unbiased cellwise variance. |
| RMSE | Root mean squared error between ensemble mean and truth. Unless explicitly labeled AE VRMSE, rollout RMSE is dimensionless standardized-model-space error. |
| SSR | Spread-to-error ratio. The precise aggregation differs between Figure 5 validation and Figures 6–7 held-out evaluation and is stated with each figure. |
| ρ, anchoring gain | Mean free error divided by mean anchored error; ρ>1 indicates improvement. |
| CRPS / afCRPS / bandCRPS | Continuous ranked probability score; its almost-fair finite-ensemble form; and the same score applied after toroidal band filtering and truth-RMS normalization. |
| ETKF / inflation | Ensemble transform Kalman filter / multiplicative expansion of forecast anomalies before assimilation. |
| M / H / start | Ensemble-member count / rollout horizon in frames / first truth-frame index of an evaluated window. |
| Ne, Te, Ti, φ, Vi, Vort | Electron density, electron temperature, ion temperature, electrostatic potential, parallel ion velocity and vorticity. |
| edge / SOL / separatrix | The outer radial evaluation region / scrape-off layer on open field lines / boundary between closed and open magnetic flux surfaces. Edge-only Figure 7 uses cropped-grid x≥16. |
| R–Z / top-down | Poloidal major-radius–vertical cross-section / view down the machine's vertical axis. |
| GPI / reflectometry / target proxy | Gas-puff-imaging-inspired support / radial microwave-diagnostic-inspired Ne chord / divertor-target point support. Current evaluator channels are idealized state samples, not complete instruments. |
| gauge dependence | φ and φ+C describe the same electric field for spatially constant C; gradients are gauge-invariant. |
| cross-phase α / coherence γ² | Argument of the cross-spectrum between two fields / normalized magnitude-squared cross-spectral association. |
| Ne-tilde / Γn | Density fluctuation about a local mean / radial particle-flux proxy based on density fluctuation and radial E×B velocity. |
| Qe,cond / λNe / λTe | Conductive electron heat-flux proxy / midplane SOL e-folding width of electron density / corresponding e-folding width of electron temperature. |
| realizability / tail concentration | Requirement that decoded physical temperatures remain positive / share of absolute flux carried by the largest 1% of events. |
| particle-count drift | Ratio of the final to initial domain-integrated particle count; a value near the truth ratio indicates comparable net drift over the rollout. |
| skewness / kurtosis | Normalized third moment measuring asymmetry / normalized fourth moment measuring tail weight. |
09Provenance, limitations and reproducibility
| artifact | location / job | supports |
|---|---|---|
| BOUT metadata | /mnt/home/sdelaurentiis/ceph/tcv-fresh-proj/85604/BOUT.dmp.0.nc | ZMAX=0.2, zperiod=5 and the one-fifth domain. |
| Truth HDF5 | /mnt/home/sdelaurentiis/ceph/tcv_well/TCV_85604/data/train/TCV_85604_train.hdf5 | Figure 1 coherence and power. |
| Representation manifest | docs/frontier_points.json | Figure 3 codec metrics and f8/z44 gates. |
| z22 continuation | job 6791660 | Figure 4; training complete, held-out rollout gate absent. |
| f8 equal-budget control | job 6791711, gate 6792464 | Figures 3–4. |
| rollout-CRPS fine-tune | job 6791994 | Figure 5 and validation checkpoint selectors. |
| held-out checkpoint gates | jobs 6791995, 6791996, 6792287 | Figures 6–7. |
| baseline sensor factorial | jobs 6771840, 6771851, 6771852 | Figure 9. |
| band-f8 sensor replication | jobs 6790070–6790072 | Independent φ/non-φ split at a smaller gain level. |
| physics battery | docs/edge_physics_battery/physbat_6777298/ | Figures 10–12. |
| machine-readable report inputs | docs/emulator_state_of_union_2026-08-10_data.json | Derived Figures 1–7. |
| report builder | docs/build/build_emulator_state_of_union_2026_08_10.py | Recreates the derived figures and this self-contained HTML. |
Primary method references
- BOUT++ input options: ZPERIOD defines the number of identical toroidal periods.
- LOLA: Lost in Latent Space (NeurIPS 2025): latent diffusion for probabilistic physical-system emulation.
- Probabilistic Retrofitting of Learned Simulators (2026): proper-scoring-rule retrofits for deterministic simulators; this motivates the separate comparison branch in Figure 2.
The report summarizes what is supported by the current two-shot simulation dataset and logged experiments. It does not validate a real TCV diagnostic, reactor diagnostic selection, cross-regime generalization or transport-faithful digital-twin performance.