ERF reference solutions

GUST is validated against ERF. ERF solves the same equations with the same turbulence model, but in time (unsteady RANS). Running ERF long enough and averaging over the final hour gives a reference steady solution for each test case (Test cases). The wall time and cost of each ERF run is also the baseline that GUST aims to beat.

Note

The reference runs need NLR’s Kestrel and have not been run yet. Everything that can be tested without Kestrel (the averaging, the archive format and its checks) is tested locally and in CI (Testing).

Pinned versions

external/VERSIONS pins the commits that GUST builds on or compares with (the file holds the full 40-character hashes):

Project

Commit

Role

ERF

92410d57

Reference solutions; source of the files in extern_erf/.

AMReX

347a3a2d

The external/amrex submodule. The CTest versions_check fails if the two differ.

FastWindTerrain

648f8f1e

Method of the mass-consistent initial field; no code copied.

Kynema-SGF

8500c568

Cross-check of the turbulence model; no code copied.

Producing the references

tools/kestrel/erf_references.sh builds ERF at the pinned commit, runs the cases and archives the results:

export GUST_ACCOUNT=<allocation>
tools/kestrel/build_kestrel.sh cpu                 # GUST and gust_refavg
tools/kestrel/erf_references.sh build cpu          # ERF at the pinned commit
salloc -A $GUST_ACCOUNT -N 1 -t 01:00:00
tools/kestrel/erf_references.sh smoke cpu          # ERF's own RANS tests, inside the allocation
tools/kestrel/erf_references.sh submit cpu all
tools/kestrel/erf_references.sh archive cpu Neutral_ABL_Flat   # after each job ends

Use gpu instead of cpu for the H100 nodes. archive always uses the CPU build, so run it from a CPU login node. A run that reaches its time limit is rerun from the start.

Run lengths

Case

Simulated time

Averaging window

CPU ranks / GPUs

Note

Neutral_ABL_Flat

48 h

1 h

8 / 1

ERF’s input file runs 12 h (see below).

Neutral_Hill_2D

48 h

1 h

32 / 1

ERF’s input file runs 6 h.

Neutral_Hill_3D

48 h

1 h

64 / 1

ERF’s input file runs 4 h.

Askervein (inputs_anel)

4800 s

2400 s

104 / 4

ERF’s input file stops at 600 s; the lee flow takes about 2000 s to cross the 7 km domain.

Stable_ABL_Flat

9 h

1 h

8 / 1

Transient by design; not a steady reference.

Convective_ABL_Flat

4 h

1 h

8 / 1

Transient by design; not a steady reference.

Why 48 hours? The flat case and both hills start from a uniform geostrophic wind with Coriolis forcing. This starts an inertial oscillation with a period of \(2\pi/f\) = 17.5 h (\(f = 10^{-4}\) s-1), which a one-hour average does not remove. A 1-D estimate puts the wind 1.3–1.4 m/s from steady after 4–6 h, 0.8 m/s after 12 h and 0.03 m/s after 48 h.

Time averaging

For snapshots at times \(t_0 < t_1 < \dots < t_n\) in the window, the reference is the trapezoidal time average

(4)\[\bar f = \sum_{i=0}^{n} w_i f_i , \qquad w_i = \frac{t_{i+1} - t_{i-1}}{2\,(t_n - t_0)} ,\]

with \(t_{-1} = t_0\) and \(t_{n+1} = t_n\). The weights allow unevenly spaced snapshots. The sum is formed in a fixed order, so the result is the same on every machine and rank count. At least 3 snapshots are needed, and a snapshot containing NaN is refused. ERF writes a snapshot every window/24 of simulated time.

Is the run stationary? The same average is also taken over each half of the window. For each variable the archive records the largest difference between the two half-window averages, and the largest \(|\bar f|\) for scale. A run that is still drifting, or that oscillates with a period close to the window, shows a large difference.

Archive

The tool gust_refavg writes one archive directory per case:

<case>_<cpu|gpu>/
  average/     AMReX plotfile of the time average
  inputs/      the ERF input file, its data files and the command-line overrides
  MANIFEST     case, ERF commit, machine, nodes, ranks, wall time, allocation units, the snapshots,
               per-variable sum, minimum, maximum and half-window change, and a SHA-256 of every file

The allocation units (AU) are nodes × wall hours × the machine’s rate: 10 AU per node-hour on Kestrel CPU nodes and 100 on GPU nodes. Partial nodes count as whole nodes, for ERF and GUST alike.

Before an archive is used, it is checked. The check fails, naming the file or variable, if a file’s checksum differs, a file is missing or not listed, the AU do not match, the manifest does not parse, or the fields do not have exactly the recorded sum, minimum and maximum.

Only cell-centred variables are averaged. On terrain cases the cell-centre height z_phys is kept when the ERF input file writes it (both hills and Askervein do).

The inputs of gust_refavg are listed in Input reference.