Solution algorithm
The steady equations \(R(q) = 0\) are nonlinear and strongly coupled, so they cannot be solved in one step. GUST approaches the solution by a pseudo-time iteration: it marches an artificial time until nothing changes any more. The march does not need to be time-accurate, so each cell may take its own, as-large-as-stable time step. Only the end point matters.
Pseudo-time iteration
Each iteration solves for a correction \(\delta q\):
\(R(q)\): the full second-order residual (Spatial discretization).
\(\Delta\tau\): the local pseudo-time step.
\(A\): an approximation of the Jacobian \(\partial R/\partial q\).
A stage chooses \(A\) and the way \(\delta q\) is computed. When \(\delta q = 0\), \(R(q) = 0\), whatever \(A\) is. A cheaper or rougher \(A\) changes how fast the iteration converges, not the answer.
The production stage is SIMPLEC (steady.stage = simplec), optionally with grid sequencing (below).
The Newton–Krylov stage (steady.stage = simplec_newton) finishes the run; multigrid is in development.
Local time step and CFL ramp
In every cell the pseudo-time step is set by a CFL number:
Vertical diffusion is treated implicitly, so it does not limit the step.
Step floor. The rate in the denominator is kept at or above a fraction of the level’s fastest convective rate
(steady.dtau_floor, \(\varphi\) = 0.1 by default):
so a cell at rest, in a recirculation pocket or under a dense forest canopy, steps at most \(1/\varphi\) times
as the fastest cell does, instead of \(\mathrm{CFL}/10^{-12}\) s. Without the floor such a cell’s momentum rows have
\(1/\Delta\tau \to 0\); with a canopy its drag derivative \(f(|U| + u_c^2/|U|)\) vanishes with the wind as
well, so the SIMPLEC \(d = a_P - \sum a_{nb}\) and with it the bound on the pressure correction’s velocity update
in the cell vanish, and the iteration cycles once the CFL cap passes about 3 (Forest canopy, steep terrain and the
boundaries: the forested escarpment and the same forest on flat ground, both steady at the default cap with the
floor). This is the observation, not a derivation: a 2-D SIMPLEC model with the same terms (quadratic drag, its
derivative diagonal, lagged pressure, local steps) converges at every CFL, so what closes the loop in the 3-D solver
is not identified. \(\varphi\) from 0.03 to 1 converges them (182 and 256 iterations at 0.1, 124 and 165 at 0.3, 109 and
101 at 1 on the flat forest and the escarpment). The default 0.1 is the largest value that leaves the empty cases’
pseudo-time paths close to what they were: the 2-D hill takes 4576 iterations at \(\varphi\) = 0, 0.1 and 0.3
and its Newton switch stays near iteration 1812 at 0.1 and 0.3 (4155 at 1); the steep ridge with the \(w\) cap
takes 2234 iterations at 0, 2012 at 0.05, 2558 at 0.1 and more than 3500 at 0.3 and 1 (its residual rises to 4 to 6
before it falls, and the floor lengthens that excursion); the thin-cell cap case is unchanged at 0.1 and 0.3 and
needs the cap less at 1 (287 iterations without it against 3713). \(U_{\max}\) is
one all-reduce per iteration, a maximum, so it is the same on any ranks; steady.dtau_floor = 0 is the plain
local step, the form before the floor. The floor changes the pseudo-time path, not the converged answer.
Cost of the global form. The floor binds wherever a cell’s own rate is below \(\varphi\) of the fastest
cell’s. On a neutral log law (12 m/s aloft, 20 m cells, an 8 m first layer) it binds nowhere at \(\varphi\) ≤
0.3; under a 25 m/s low-level jet over a slow surface (0.9 m/s at 4 m) it binds in most of the column at 0.1 and
shortens the near-ground step 2.8 times, so a stable high-shear run converges at a tenth of the jet’s rate there
(steady.dtau_floor = 0 or a smaller value restores the local step). Not tried: a rate from the cell’s own face
speeds (the cell-centred \(|u_c|\) of a reversal cell is 0 while its faces move, and the arithmetic face mean of
\(\Delta\tau\) hands half of a rest cell’s step to its neighbours’ faces) or a neighbourhood rate, which would
bound such cells locally with no all-reduce; \(\varphi\) = 0.05 was not swept on the forest cases.
The CFL number starts small and grows as the residual falls (switched evolution relaxation, SER; Mulder and van Leer 1985):
\(\mathrm{CFL}_0\), \(\mathrm{CFL}_\mathrm{max}\):
steady.cfl0andsteady.cfl_max.\(r_n\): the largest residual relative to its reference value (Convergence criteria).
\(p\):
steady.ser_exponent; \(g\):steady.ser_max_growth, the largest growth per iteration.
SIMPLEC stage
SIMPLEC (Van Doormaal and Raithby 1984) solves the momentum equations first, then corrects pressure and velocity so that mass is conserved. One iteration:
Fill the ghost cells and evaluate the wall model at the current state \(q\).
Evaluate the residual \(R(q)\) and the turbulence model.
Momentum predictor. Solve \((1/\Delta\tau + A_1)\,\delta\mathbf u = -R_\mathbf u\), giving \(\mathbf u^*\).
Pressure correction. Solve a Poisson equation for \(\phi\) and set \(\mathbf u = \mathbf u^* - \nabla\phi/d\) and \(p \leftarrow p + \alpha_p\phi\).
Scalars. Take one step of the same form for \(\theta\) (with its convection updated to the corrected velocity) and \(k\); bound \(k\) below by its floor.
\(A_1\) is the first-order operator of Spatial discretization.
Momentum predictor and line solvers
ABL meshes have cells much thinner than they are wide, so the vertical coupling is the strongest. The predictor
therefore solves each vertical column exactly (a tridiagonal solve with the Thomas algorithm). The coupling to the
horizontal neighbours is handled by sweeps; steady.sweep selects how:
zjacobi(default): z-line Jacobi. All columns are solved at once, with the horizontal neighbours taken from the previous sweep.steady.zline_sweeps(at least 2) sets the sweeps per iteration. More sweeps allow a larger stable CFL; the terrain cases use 16 sweeps with a CFL cap of 4.lusgs: line LU-SGS (lower–upper symmetric Gauss–Seidel). The columns of each box are visited in order: a forward pass uses the already-updated lower neighbours, and a backward pass the updated upper neighbours. One sweep is a forward and a backward pass. When the wind blows along the sweep direction, a pass resolves the upwind coupling exactly, which allows a much larger CFL. The hill cases use two sweeps with a CFL cap of 16. When \(u\) and \(v\) have opposite signs, one sweep is stable only to a CFL of about 2.5 (two sweeps to about 33), so use at least two sweeps. Box faces are coupled with lagged values, so the iteration path (not the answer) depends onmesh.max_grid_size, and boxes shorter than the CFL number slow the iteration.
Four options refine the LU-SGS sweeps. They change the path to the solution, not the solution.
Input |
Effect |
|---|---|
|
|
|
With |
|
|
|
|
On CPUs, lagged needs the fewest iterations. zero avoids communication inside the sweeps, which is expected
to matter on GPUs and on many ranks. Measurements are in Results.
Pressure correction
With the SIMPLEC coefficient \(d = a_P - \sum a_{nb}\), the pressure correction solves
SIMPLE neglects the neighbours’ velocity corrections and needs a pressure under-relaxation. SIMPLEC assumes they
equal the cell’s own, so \(\alpha_p = 1\) works (simplec.alpha_p).
The Poisson solve stops at the relative tolerance simplec.pressure_reltol (1e-3). A looser tolerance leaves
divergence that drives \(w\) through buoyancy, and the iteration diverges in stratified flow.
Point-implicit buoyancy and rotation
Two couplings oscillate in pseudo-time where \(\nu_t\) is small: \(w\) with \(\theta\) (buoyancy, frequency \(N\)) and \(u\) with \(v\) (Coriolis, frequency \(f\)). An explicit treatment amplifies each oscillation by \((1 + \omega^2\Delta\tau^2)^{1/2}\) per step. The predictor therefore treats both with a linearised backward Euler step:
\(\tau_\theta = 1/(1/\Delta\tau + a_{P,\theta})\): the step allowed by the \(\theta\) rows.
\(\langle R\rangle\): the horizontal mean, on each level, of the other component’s residual, each face counted once and summed in a fixed order, so the mean has the same bits on any number of ranks or GPU threads; \(\overline R\): its local value.
\(\tau_c = 1/\max(|f|/r, a_P - \sum a_{nb})\), with \(r\) =
simplec.rotation_relax(0.3).\(\tau_\mathrm{own}\): the row’s own step.
The inertial oscillation is horizontally uniform, so only the level mean uses the long inertial step. Every added term vanishes when \(R = 0\), so the steady solution does not change.
Prescribed potential temperature
With simplec.theta = frozen, \(\theta\) is data, not an unknown. This is the steady stratified form of
van der Laan et al. (2024): the stratification is prescribed, and buoyancy acts in the turbulence closure only.
\(\theta_\mathrm{in}(h)\): the inflow profile’s \(\theta\) at the height \(h\) above the ground, for every
bc.inflow_profileand whether or not a face is an inflow face: the file’s θ column withfile, the soundingbc.theta_profile(read by height above the ground) with the analytic kinds.\(h\): the cell centre’s height (the mean of its eight nodes) minus the mean of its four ground nodes, on each level’s own ground (the coarse levels’ ground is smoothed). It is not floored at
init.z_agl_min, as the inflow face’s ghost cells read the profile.\(R_w^{\,g=0}\): the \(w\) residual without the buoyancy force \(-g(\theta - \theta_\mathrm{ref})/\theta_0\). The predictor’s point-implicit buoyancy terms (above) are dropped with it.
\(N^2\), \(c_\mu\), \(c_\mu'\), \(\nu_t'\), \(P_b\): the closure’s stability terms (Turbulence model), unchanged;
forcing.gravityandforcing.theta0still set them.
What it does. Every sequenced level sets \(\theta\) from the inflow profile in place of the sounding or the
prolonged field, and logs the range and mean it set. The SIMPLEC stage then skips the \(\theta\) sweep and
reports a \(\theta\) residual of 0, so the convergence test (Convergence criteria) covers the momentum,
the pressure and \(k\). The ghost cells of
\(\theta\) are filled as usual (the inflow face takes the same profile), except at the ground: there the two
ghost layers continue the first two cells linearly, so the first cell’s centred \(\partial\theta/\partial z\)
(its \(N^2\)) is the profile’s between the first two cells. The wall law’s gradient would set it instead, and a
neutral wall would give \(N^2 = 0\) in the first cell. The surface layer still sees \(\theta\) through its
stability (neutral unless bc.surface_stability is set).
Why. A steady solve of \(\theta\) with a fixed surface heat flux has no steady state: the surface keeps adding or removing heat. With buoyancy in the momentum equations as well, stable and unstable cases on a forested escarpment diverged. With \(\theta\) frozen from a near-1-D column and buoyancy in the closure only, all four converged. Van der Laan et al. (2024) use the form for neutral and stable boundary layers; the unstable cases apply it beyond that, with \(N^2 < 0\) making the closure’s \(P_b\) a source of \(k\). The cost: the flow cannot change the stratification. A cold pool in a valley or air lifted over a ridge keeps the upstream profile by height above the ground, and there are no buoyancy forces, so no lee waves, gravity currents or katabatic and anabatic slope flows. Over terrain the omitted force is not small: with \(\theta\) by height above the ground, \(\partial\theta/\partial x = -\Gamma\tan\alpha\) at fixed \(z\) (\(\Gamma\) the profile’s gradient, \(\alpha\) the slope), and across an escarpment of height \(H\) the buoyancy \(g\Gamma H/\theta_0\) can be of the order of the advection \(U^2/L\). That field is not in hydrostatic balance, which is why it cannot be combined with the buoyancy force. Van der Laan et al. (2024) apply the form on flat terrain with a constant \(N\).
Limits. Refused with steady.stage = simplec_newton (the Newton stage solves \(\theta\)), with
io.restart (the checkpoint holds no record of the option), without grid sequencing (steady.sequencing = 1
and mesh.hierarchy_levels > 1: the levels set \(\theta\)), and with a stratified column first guess
(init.first_guess = column with a non-neutral init.column_stability, whose evolved \(\theta\) the frozen
field would replace). With \(\theta\) frozen, the inputs that act only on the \(\theta\) equation have no
effect: a Rayleigh layer or sponge on theta (bc.rayleigh_fields) and the \(\theta\) gradient through the
top (bc.zhi_theta_grad); bc.surface_heat_flux still sets the surface layer’s stability but adds no heat. A
checkpoint written with frozen can be restarted without it; the restart then solves \(\theta\) from the frozen
field, with the buoyancy back in the momentum equations. With the leaves’ heat exchange (canopy.heat = fixed),
canopy.heat_frozen = blend sets \(\theta\) in the canopy between the frozen field and the leaves’ \(\theta\)
each iteration, and skip (the default) leaves it out (Forest canopy, canopy heat).
Vertical-velocity cap
On meshes with thin first cells, \(w\) converges slowly. The vertical neighbour weights \(a_{ZM}, a_{ZP}\) of the \(w\) rows cancel in \(d\), so the pressure correction moves \(w\) too far by the vertical von Neumann number
and the next predictor undoes most of the move. simplec.w_vnn_cap = \(V \ge 16\) adds a term on the
\(w\) rows only (not on the ground and top rows):
The cap changes the path, not the steady answer: the predictor is in delta form, and \(d\) enters only the pressure correction (there is no Rhie–Chow interpolation on the staggered grid), so the fixed point is still \(R = 0\). It is off by default. Values below 16 are refused: on flat ground the iteration diverges for \(V \le 2\) at a CFL of 16 and for \(V = 4\) at a CFL of 64.
Sloping ground. On a \(z\)-face the pressure correction scales the whole normal gradient, slope term included, by \(\beta_z = 1/d_w\), while the \(x\)- and \(y\)-faces use \(\beta_h = 1/d_u, 1/d_v\). For one Fourier mode (wavenumber \(k\) along a slope \(h\), \(m\) in \(\zeta\)) the operator’s quadratic form is
which is definite only while \(r = \beta_z/\beta_h\) exceeds
Without the cap \(r \approx 1\). With it \(r = 1/(1 + \mathrm{VNN}/V)\), so thin cells on steep slopes cross
\(r_c\): on Askervein’s steepest slope (23°, \(r_c = 0.041\)) a cap of 16 fails once VNN exceeds 370, as with
a 0.4 m first cell. simplec.w_vnn_cap_slope_margin = \(s\) (default 4) limits the term on each \(w\) row
to
with \(h^2 = h_x^2 + h_y^2\) from the face’s node heights (the slope the operator’s cross term uses). The limit
assumes that the horizontal coefficient and the vertical one without the cap are both \(1/\Delta\tau\). Then
\(r \ge s\,r_c\) wherever \(s\,r_c < 1\) (below 53° at \(s = 4\); steeper rows get no cap). On the
solver’s own coefficients the margin is smaller where the vertical coefficient is stiffer (stable stratification,
the Rayleigh layer: at \(s = 4\) and 32.5°, \(r \ge 3.0\,r_c\) when it is twice \(1/\Delta\tau\) and
\(1.7\,r_c\) at five times), and larger in the first layer, where the surface drag stiffens the horizontal
coefficient. The definite band is also bounded above, \(r < 1/r_c\). On steep slopes the cap acts as a larger
local \(V\), and flat ground keeps \((a_{ZM} + a_{ZP})/V\) exactly. Giving the slope term \(\beta_h\) instead (FaceCoeffs::z_cross) would
keep the operator definite for any \(r\), but the default Poisson back end has no split coefficient. With
\(s = 0\) (the cap of #32) the run warns over terrain when the mesh aspect ratio is above the tested 50.78. See
Results for the tested cases.
Pressure Poisson solver
The pressure correction, the mass-consistent initial field and the Tucker wall distance all solve
where \(f\) are ERF’s face gradients, terrain cross terms included, and \(\beta\) are face coefficients
(\(1/d\) for SIMPLEC). solver.poisson_backend selects the solver:
Back end |
Method |
|---|---|
|
GMRES preconditioned by AMReX’s |
|
The same multigrid alone. |
|
ERF’s |
The default needs a nearly constant number of iterations per solve as the first layer becomes thinner, where the FFT preconditioner needs more (Results).
The solve stops at solver.poisson_reltol or solver.poisson_abstol; the pressure correction uses
simplec.pressure_reltol instead. gmres_mlmg measures the residual weighted by cell volume (\(J\)),
gmres_fft the plain residual and mlmg the maximum norm. The same tolerance therefore does not mean the same
divergence where \(J\) varies. With no Dirichlet face the mean of the right-hand side is removed first.
Grid sequencing
Most of the iterations of a run remove large-scale errors, which the fine mesh removes slowly. Grid sequencing removes them first on coarse meshes, where an iteration is cheap, and then starts each finer mesh from the coarser solution. It uses the grid hierarchy of Grid hierarchy.
steady.sequencing = 1 with mesh.hierarchy_levels = \(L \ge 2\) runs, on a fresh start:
Coarsest level \(L - 1\): the initial field (
init.*) on that mesh, then SIMPLEC to a loose stop.Levels \(L - 2, \dots, 1\): the solution of the level above is prolonged (interpolated, then made divergence-free), then SIMPLEC runs to a loose stop.
Level 0 (the run’s mesh): prolonged from level 1, then SIMPLEC to the normal convergence criteria.
Each level has its own boundary conditions, turbulence model, wall distance and Poisson solver. An iteration on
level \(l\) costs about \(4^{-l}\) of a level-0 iteration. steady.max_iterations applies to each level
separately. The log ends with a table of every level’s iterations and wall time.
Loose stop. A coarse level stops when, for every residual \(q\),
with \(d_\ell\) = steady.sequencing_drop_orders (2), or when the full convergence criteria hold.
steady.sequencing_monitor_tol adds an optional monitor test; it is off by default, because right after a
prolongation the monitors settle before the residuals have dropped.
Residual reference. A prolonged start is already close to the solution, so a level’s own first residuals are small. Measuring the drop from them would hold level 0 to a stricter standard than a run without sequencing. Every finer level therefore starts with the reference residuals \(S_q\) of the coarsest level, which, like a run without sequencing, starts from the initial field.
Reference hold. A driver normally runs its first steady.residual_ref_iterations iterations at
steady.cfl0 while it takes its reference. A seeded level already has its reference, so by default
(steady.sequencing_hold = 0) it skips this hold. From the first iteration it uses
in the CFL ramp, which then climbs at once (still limited by steady.ser_max_growth). steady.sequencing_hold =
1 keeps the hold on every level. steady.sequencing_cfl_start = previous starts each seeded level at the CFL the
previous level reached, instead of steady.cfl0.
What it does not change. The converged state is the same with and without sequencing, to the convergence tolerance. A restart continues level 0 only; the coarse levels run on a fresh start. While the coarse levels run, the level-0 solver stays allocated, so the peak memory is somewhat above that of level 0 alone.
On the 3-D hill, three levels reduce the level-0 iterations from 2091 to 706 and the wall time from 145 s to 54 s (Results).
Newton–Krylov stage
With steady.stage = simplec_newton a Newton–Krylov stage finishes the iteration once SIMPLEC is close to the
solution. Each Newton step solves
by GMRES, with the Jacobian–vector products of Spatial discretization.
\(J = \partial R/\partial q\), including the dependence of \(\nu_t\) on \(q\).
\(D\): \(1/\Delta\tau\) on the momentum, \(\theta\) and \(k\) rows; 0 on continuity and boundary rows. \(D = 0\) is plain Newton.
The fields are scaled so GMRES weighs them alike.
Its inputs (steady.newton_*) are listed in Input reference.
Switching and fallback
NewtonStage (Source/NewtonStage.H) is the driver’s second stage.
Switch. The driver moves from SIMPLEC to Newton after
steady.switch_window(20) consecutive iterations with the relative residual at or belowsteady.switch_residual(1e-4) and every monitor settled: changed by at moststeady.switch_monitor_tol(1 %) of max(|value|, scale) over the window.Entry. The smoothness handling below enters (the k floor frozen, the smooth limiter in the Newton residual), and the pseudo-transient continuation starts afresh at
steady.cfl_max. Its CFL doubles (steady.ser_max_growth) with every accepted step up tosteady.newton_cfl_max(128); withsteady.newton_dtau_inf_at_cap = 1\(\Delta\tau\) is then infinite. At every finite step the preconditioner uses the step’s own \(\Delta\tau\), and at infinite \(\Delta\tau\) the Newton cap’s. A residual-driven ramp (the SER law of the SIMPLEC stage) settles near CFL 13 on the hill, where \(|R|\) falls slowly; jumping to infinite \(\Delta\tau\) after one step leaves the slow modes unpreconditioned (GMRES(30) and GMRES(200) stall at the 2000-iteration limit on ERF’s 2-D hill). On the 8 × 2 × 8 hill, from the switch to steady:Newton CFL cap
After the cap
Newton steps
GMRES iterations
128 (default)
infinite
8
480
1000
infinite
11
718
1e4
infinite
13
882
1000
held
28
2058
128
held
102
5338
Each iteration is one continuation step. The residual norms and monitors the driver records are those of the run’s own equations at the new state (the run’s limiter, the closure’s live decisions), so the steady criterion and the relative residual mean what they mean for SIMPLEC. That costs one more residual evaluation per step. Once the continuation has converged on equations that are the run’s own (|R| at or below \(10^{-\text{drop}}\,|R_0|\), with
steady.residual_drop_orders), the stage takes no more steps; the driver’s criterion then fills its monitor window at one residual evaluation per iteration.Fallback. After
steady.newton_fallback_failures(2) consecutive failed steps (every retry used) the stage hands the run back to SIMPLEC at once, whatever the hysteresis. SIMPLEC may switch forward again after the window; after three failed visits the stage refuses (SIMPLEC finishes the run). The switch-back rule on residual growth (steady.switch_back_growth) applies too. A stage that has converged on the run’s own equations but whose steady criterion has not held forsteady.monitor_windowiterations hands back as well.Final pass. When the Newton stage’s equations differ from the run’s (the limiter was swapped, or a k-floor decision moved), it hands the run back once its own residual has converged, and refuses any later switch: the run ends on SIMPLEC with the run’s own equations. With
convection.limiter = van_albadathere is no swap.Restart. The checkpoint holds
<checkpoint>/NewtonStage, the continuation state and the smoothness state (written before the stage is left at the end of a run); a restart inside the stage continues it bit for bit, with either preconditioner (measured on the 2-D hill, a checkpoint three Newton steps in). The stage fills the ghost cells on entry (NewtonStage::init) where the run it continues does not; the fill is a fixed point, so this changes nothing. Asteady.stage = simpleccheckpoint continues withsimplec_newton(the Newton stage starts with an empty account); the reverse aborts, namingsteady.stage.Grid sequencing. The coarse levels run SIMPLEC only (their loose stop ends them first); the Newton stage is level 0’s.
Relaxed \(\nu_t\). With
turb.nut_relax< 1 the closure’s history is the last SIMPLEC iteration’s, which the Newton stage does not update, so the stage’s equations are not the steady ones: it hands back for the final pass.
Limits.
Before the Newton CFL cap, the stage went to infinite \(\Delta\tau\) after one step; GMRES(30) and GMRES(200) then stalled on ERF’s 2-D hill (2000 iterations at a Krylov residual of 0.9) and the run cycled between the stages.
On that hill at full size (128 × 1 × 64, no Coriolis) the ramp does not remove the stall: GMRES(30) needs 105, 180 and 323 iterations at CFL 16, 32 and 64, and reaches the 2000-iteration limit from CFL 128 on (Krylov residual 0.08 at 128, 0.75 at 256, 0.97–0.99 at infinite \(\Delta\tau\)); those steps are still accepted, and \(|R|\) falls from 1.8e-4 to 7.8e-5 over 11 steps. The SIMPLEC preconditioner holds GMRES(30) to about CFL 64 on that grid (the measurement used a cap of 1000; the default is now 128). The cap, the restart length and the switch residual are provisional: a Newton study on Askervein has not been run yet.
The stage’s continuation takes its momentum diffusivity from the closure at its own state, not from the SIMPLEC stage (whose is not restored by a restart inside the Newton stage).
steady.newton_gmres_max_iters bounds the GMRES iterations of one solve over all its restarts. A solve that
reaches it is unconverged; its update is used when the Krylov residual is finite and below \(|b|\), like any
other.
The Newton preconditioner
GMRES solves the Newton system with a right preconditioner \(M^{-1} \approx (D + J)^{-1}\), chosen by
steady.newton_preconditioner (SimplecPreconditioner, Source/NewtonPreconditioner.H). identity is
no preconditioner. simplec (the default) applies one SIMPLEC iteration to the linearized system
\((D + J)\,\delta q = r\), starting from \(\delta q = 0\):
The momentum predictor of the SIMPLEC stage, \((1/\Delta\tau + A_1)\,\delta u^* = r_u\), with the coefficients, rotation and buoyancy terms at \(q\).
The pressure correction with the continuity part \(r_p\) as a source,
\[\nabla\cdot\left(\frac{1}{a_P}\nabla p'\right) = \nabla\cdot\delta u^* - r_p, \qquad \delta u = \delta u^* - \frac{1}{a_P}\nabla p', \qquad \delta p = \alpha_p\,p',\]so that \(\nabla\cdot\delta u = r_p\). \(a_P\) is the predictor’s diagonal and \(\alpha_p\) is
simplec.alpha_p.The \(\theta\) and \(k\) line steps on their parts of \(r\). The change of the \(\theta\) convection by \(\delta u\) is a directional derivative, \([C(F(u + \sigma\delta u)) - C(F(u))]/\sigma\) with \(\sigma\) as in the Jacobian-vector product, and the line operators use the transport of \(u\), so
applyis linear in \(r\) (to the finite-difference and Poisson tolerances). With the stage’s denominator (below) the full \(\delta u\) and the transport of \(u + \delta u\) are used instead, as in the stage. The increment \(\delta k\) is not clipped.On the Dirichlet rows of \(u, v, w\), which are identity rows of the Newton system, \(\delta q = r\).
The sweeps are those of the SIMPLEC stage: steady.zline_sweeps of steady.sweep. setup rebuilds the
coefficients once per Newton step, at the current state and \(\Delta\tau\). Inside the continuation loop
that is every attempt. At infinite \(\Delta\tau\) the preconditioner uses the local time step at
steady.cfl_max.
The denominator. The SIMPLEC stage divides by \(d = a_P - \sum a_{nb}\). In the interior that is about
\(1/\Delta\tau\), so the pressure increment vanishes as \(\Delta\tau\) grows. The preconditioner
therefore uses the SIMPLE denominator \(a_P\). On the 3-D hill of
NewtonPreconditioner.IterationsBoundedOverCfl the GMRES iterations per Newton step then stay near 10 from CFL 10
to 1e4; with \(d\) they grow from 18 to more than 350 (Results). With \(d\) the preconditioner is
exactly one iteration of the stage (NewtonPreconditioner.ApplyIsOneSimplecIteration).
Shared faces. A face shared by two boxes, or a periodic seam face and its image, is one unknown. \(r\) arrives with equal copies, and \(\delta q\) is returned with every copy equal to the owner’s.
The pressure gauge. Without a Dirichlet pressure face, a constant pressure is the null space of
\(D + J\). The preconditioner removes the net source of \(r_p\) that the boundaries cannot carry, and
returns \(\delta p\) with zero \(J\)-weighted mean. The Newton update then keeps the mean of \(p\), and
a constant added to \(p\) changes neither the preconditioner nor the solve
(NewtonPreconditioner.ConstantPressureShiftLeavesTheSolveUnchanged).
Limits.
On the horizontally uniform Ekman column the pressure plays no part. The stiff part at large CFL is the balance of rotation and vertical diffusion, and
simplecneeds about as many GMRES iterations asidentity.The pressure solve’s reductions depend on the rank order, so with
simplecthe Newton iterates are not bit for bit the same on different rank counts. Withidentitythey are.Memory: about 25 state-size fields and the Poisson solver’s hierarchy, on top of the Krylov vectors.
Pseudo-transient continuation
A continuation loop (Continuation, Source/Continuation.H) wraps the Newton step. The application does not
run it yet. One continuation step from \(q_n\):
\(\Delta\tau\) from the local time step at \(\mathrm{CFL}_n\), or no pseudo-time term (\(D = 0\)) once the CFL has reached its cap and
steady.newton_dtau_inf_at_cap = 1.The direction \(\delta q\) from \((D + J)\,\delta q = -R(q_n)\) (one GMRES solve).
A backtracking line search on \(|R| = |T^{-1}R(\mathrm{fill}(q))|_2\), the norm of the Newton step:
\[|R(q_n + \lambda\,\delta q)| \le (1 - c\,\lambda)\,|R(q_n)|, \qquad \lambda = 1, \beta, \beta^2, \ldots \ge \lambda_\mathrm{min},\]with \(c\) =
steady.newton_armijo_c, \(\beta\) =steady.newton_backtrackand \(\lambda_\mathrm{min}\) =steady.newton_lambda_min. Each trial point keeps \(k\) at or aboveturb.tke_floorand goes through the ghost fill before \(R\) is evaluated. At a finite \(\Delta\tau\) the direction is a pseudo-time step and need not reduce \(|R|\). With \(\gamma\) =steady.newton_ptc_growth> 1, when no \(\lambda\) meets the condition there, the full step is accepted if \(|R(q_n + \delta q)| \le \gamma\,|R(q_n)|\); the CFL law then lowers the next CFL.Accepted: \(q_{n+1} = q_n + \lambda\,\delta q\), and the CFL follows the SER law of the SIMPLEC stage,
\[\mathrm{CFL}_{n+1} = \min\!\left(\mathrm{CFL}_\mathrm{max},\; g\,\mathrm{CFL}_n,\; \mathrm{CFL}_0 \left(\frac{|R_0|}{|R_{n+1}|}\right)^{p}\right),\]with
steady.cfl0,steady.cfl_max, \(g\) =steady.ser_max_growthand \(p\) =steady.ser_exponent. \(|R_0|\) is the first finite \(|R|\), or one the caller supplies (set_reference_residual). While \(\Delta\tau\) is infinite the law starts from \(\mathrm{CFL}_\mathrm{max}\).A skipped update or a line search that accepts no \(\lambda\) leaves \(q_n\) unchanged, multiplies the CFL by
steady.newton_cfl_cut(from infinite \(\Delta\tau\), \(\mathrm{CFL}_\mathrm{max}\) times it) and retries, up tosteady.newton_max_retriestimes. Then the step returns failed, and the Newton stage will fall back to SIMPLEC.
Every attempt prints one line: the step, \(|R|\) before and after, the CFL (inf for \(D = 0\)),
\(\lambda\), the backtracks, the GMRES iterations and status, the Krylov relative residual and the outcome
(accepted, skipped or rejected). The continuation state (step count, CFL, the switch to infinite
\(\Delta\tau\), \(|R_0|\) and the last \(|R|\)) is written to and read from a checkpoint directory
(Continuation, hex floats); a restart gives the uninterrupted run’s bits.
Limits.
With
steady.newton_ptc_growth = 1(the default) a pseudo-time direction that raises \(|R|\) for every \(\lambda\) is rejected, and a CFL cut need not help. On the Ekman column withsteady.ser_exponent = 0.5the retries run out at 3.5e-2 of \(|R_0|\); with 1.5 the column converges (Results).Once \(\Delta\tau\) is infinite it stays infinite through damped steps and unconverged solves; only a failed attempt returns it to a finite CFL. Returning to \(\mathrm{CFL}_\mathrm{max}\) after such a step was tried and measured worse.
With GMRES(30) and infinite \(\Delta\tau\), the 30 m hill stagnates near 6e-3 of \(|R_0|\) for twelve steps at the 2000-iteration limit, then fails at 7e-8. With
steady.newton_dtau_inf_at_cap = 0it reaches 1e-8 in 23 steps, and with GMRES(200) in 8. The restart length and the cap will be tuned with the Newton stage.
Smoothness handling
The Jacobian-vector product is a forward difference of the residual. Across a kink of the residual it measures the
jump of the slope, not the derivative, and Newton loses its convergence rate. NewtonSmoothness
(Source/Smoothness.H) handles the two kinks before the switch to Newton:
The limiter. The van Leer slope \(2ab/(a+b)\) (0 when \(ab \le 0\)) has a kink wherever \(a\) or \(b\) passes through 0: at every extremum, and in nearly uniform regions where the differences are round-off of either sign. The Newton stage’s residual uses the smooth van Albada slope instead (
van_leer→van_albada, withconvection.smooth_eps_*;upwindandvan_albadaare kept).The k floor. With
turb.frozen_switch = 1(the default) the floor decisions are frozen at the state where Newton starts. A cell at the floor there stays at \(k = k_\mathrm{floor}\) for the whole stage, so \(k_f = \max(k, k_\mathrm{floor})\) has no kink as its \(k\) rises; its branch and \(R_t\) smoothings are decided at that \(k\). A cell above the floor is floored again only if its \(k\) falls to the floor or below, so no \(k \le 0\) (from a line-search trial or a Jacobian perturbation) reaches the closure. The stable/unstable branch and the \(R_t\) smoothing are not frozen: they are continuously differentiable (Turbulence model), and freezing them would replace the state’s decisions by those of the switch point.
After the Newton stage the floor thaws and a final SIMPLEC pass with the run’s own inputs (van Leer, live decisions)
converges the run’s discrete equations from the Newton solution. leave reports how many floor decisions moved
during the stage. The module’s state (active, the frozen decisions) goes to the checkpoint
(<checkpoint>/Smoothness, Level_0/closure_switches) and is restored on any layout.
The stability branch stays live. It is not frozen with the other decisions. In a neutral layer \(N^2\) is round-off and its sign flips: on the 2-D hill 286–507 of 512 cells changed branch during Newton. With the branch frozen at the switch state Newton was at 6.0e-6 after 8 steps on the inversion (van Leer), against 9.4e-8 without it, and failed at 5.7e-9 on the neutral van Albada run, against 1.4e-13 (Results).
Cost. The two limiters give different discrete solutions (0.4–0.6 % of the largest \(u\) on the 2-D hill),
so the verification pass is not short: 3650–4316 SIMPLEC iterations, 44–74 % of what SIMPLEC needs from 1e-4 to
1e-9 on its own, on top of the Newton stage. On a van Leer run the swap therefore saves little; with convection.limiter = van_albada
for the whole run Newton solves the run’s own equations, reaches 4e-11 in 5 steps, and needs no verification
pass, at the cost of a 0.4–0.6 % different answer (Results).
Physical time term (dual time stepping)
Unsteady RANS by dual time stepping takes one backward-Euler (BDF1) step of physical length \(\Delta t\) by a pseudo-time iteration to convergence. Each transported equation gains a physical time term,
where \(R_q\) is the residual per unit volume (as in the rest of this page), \(q\) the current iterate and \(q^n\) the solution at the last physical step. Continuity has no time term. When the pseudo-time iteration has driven \(R_q\) to zero, \(q\) is \(q^{n+1}\) of the backward-Euler step.
The implicit rows. The term’s derivative with respect to \(q\) is \(1/\Delta t\) on the row’s own
diagonal. implicit_rows adds it to \(a_P\), to which the callers add the local pseudo-time term
\(1/\Delta\tau\), so the
line solves of \(u, v, w, \theta, k\), the momentum predictor and the SIMPLEC denominator
\(d = a_P - \sum a_{nb}\) all carry it. The Newton core’s \(Jv\), a difference of residuals, sees it without
a change. The rotation’s step and the buoyancy step of the predictor, which use the same \(a_P\), also see
\(1/\Delta t\): where \(\Delta t\) is shorter than the rotation’s own step, the rotation is advanced by
backward Euler over \(\Delta t\). The fixed point does not change.
Rows that replace their residual. The term is added before the rows that replace their residual: the wall-k rows
with bc.surface_wall_k = dirichlet, the Dirichlet rows of the boundary conditions and the fixed rows of buildings.
implicit_rows adds \(1/\Delta t\) before the same rows become identity rows. Such a row carries no time term
in either. Rows that add to the residual (damping, the source form of the wall-k rows) keep it. The k rows of cells
partly inside a building scale the residual and the row by \(1 - f\) (Buildings), and so keep
\(1 - f\) of the term in both.
Storage. \(q^n\) is a copy of \(u, v, w, \theta, k\) with their ghost cells, held by the TimeTerm
module (Source/TimeTerm.H). The residual reads only its valid points. Without a time term the residual and the
rows are unchanged, bit for bit. \(q^n\) is not written to the checkpoint: a checkpoint is written at the end
of a step, where \(q^n\) of the next step is the state itself. The module checks that \(\Delta t\) is finite and positive with a finite \(1/\Delta t\).
Time driver
With steady.stage = unsteady the run marches physical time from 0 to unsteady.stop_time in backward-Euler
steps (Source/TimeDriver.H). Each step does three things.
The step length. The ghost cells are filled and \(\Delta t\) is chosen:
unsteady.dt, or withunsteady.cfl> 0 one global step\[\Delta t = \frac{C}{\max_\text{cells}\left(|u_c|/\Delta x + |v_c|/\Delta y + |\Omega_c|/(J\,\Delta\zeta)\right)},\]where \(C\) is
unsteady.cfl, \(u_c, v_c\) the cell-centred horizontal velocities, \(\Omega_c\) the cell-centred contravariant vertical velocity and \(J\,\Delta\zeta\) the cell’s height: the convective rate of the local pseudo-time step without its diffusive part or floor. The last step ends the run atunsteady.stop_time: a step within 0.1 % of the time left takes all of it, so no sliver of a step is left. A \(\Delta t\) that is not finite, or a target CFL over a field at rest (no convective rate), stops the run with a message. The time is accumulated in double precision (\(n\,\Delta t\) for a fixed step).The old state. \(q^n \leftarrow q\), and the time term takes \(\Delta t\).
The inner iterations. Pseudo-time iterations of the SIMPLEC stage, with the time term, run until every scaled residual satisfies
\[R_q^m \le \max\left(10^{-d}\,R_q^1,\ \tau_R\right),\]where \(R_q^m\) is the norm of equation \(q\) at inner iteration \(m\), \(d\) is
unsteady.inner_drop_ordersand \(\tau_R\) issteady.residual_tol; or untilunsteady.max_inneriterations have run. \(R_q^1\) is the residual at \(q = q^n\), where the time term is zero: the steady residual of the previous step’s answer. The floor \(\tau_R\) ends a step whose start is already steady at once, instead of asking for round-off.
How far to converge a step. \(R_q^1\) is about \(|\partial q/\partial t|\), so the inner stop leaves an algebraic error of order \(10^{-d}\,\Delta t\,|\partial q/\partial t|\), against the backward-Euler error of order \(\Delta t^2 |\partial^2 q/\partial t^2|/2\) per step. In a 1-D model of Stokes’ first problem with a slowly converging inner solver (point Jacobi), the algebraic error at the end was 3 to 5 times the time error with \(d = 1\), about half of it with \(d = 2\) and below a tenth with \(d = 3\); with the line sweeps it stayed below 0.06 of it for every \(d\). The default is \(d = 3\); \(d = 2\) is cheaper where the time history need not be accurate (a run to a periodic or settled state).
Why the steady stop is not used. The steady verdict also asks the monitors to settle over a window and the
residual to drop from a reference held over the first iterations. Inside a physical step neither applies: the step’s
answer is not steady, and the reference changes every step. The driver’s CFL ramp (SER) runs whatever the verdict says,
and the ramp and the histories carry on across steps. The hold at steady.cfl0 lasts the first
steady.residual_ref_iterations inner iterations of the run, which may span several steps (the 2-D hill deck holds
for 100 iterations, about two and a half steps).
Restart. A run may start from a checkpoint of a steady run (io.restart). Its physical time starts at 0, the
state is the checkpoint’s, and the driver’s history continues: there is no new hold, and the CFL and the residual
reference are the checkpoint’s. A steady.stage = simplec_newton checkpoint cannot be continued (it has two stages).
From a checkpoint of an unsteady run the run continues where it was (Unsteady outputs).
What a step reports. One line per step: the time, \(\Delta t\), the inner iterations and the largest
\(R_q^m / \max(R_q^1, \tau_R)\) at the last one. A step that reaches unsteady.max_inner is marked and the run
goes on; the summary counts such steps, and the run still exits normally (check the count). A residual that becomes
NaN or infinite ends the run as diverged. The plotfile carries the physical time; its number is the total of inner
iterations. With io.turb_plot_file the residual fields include the time term of the last step, the residual that
step converged.
Order. Backward Euler is first order in \(\Delta t\): halving the step halves the time error
(TimeDriver.StokesFirstProblemFirstOrder, Stokes’ first problem against its erfc profile).
Not supported yet. Grid sequencing, the Newton stage and the hybrid upwind blend (convection.hybrid_sigma_max > 0:
it follows the inner residual, so the operator would change within and between steps). A deck that asks for
sequencing (steady.sequencing = 1) or the blend stops with a message.
Unsteady outputs
An unsteady run (Source/UnsteadyOutput.H) keeps running averages, writes probe rows, and writes plotfiles and
checkpoints during the run, from which a later run continues.
Running averages. After each step from \(t^n\) to \(t^{n+1}\) the mean and the variance of \(x \in \{u, v, w, \theta, k\}\) (the cell-centred values, velocities averaged from the faces) are updated by the weighted form of West (D. H. D. West, Updating mean and variance estimates: an improved method, Communications of the ACM 22(9), 532–535, 1979):
where \(t_a\) is unsteady.average_start, \(w\) the step’s weight (no update if \(w \le 0\)),
\(W\) the time averaged so far, \(\bar x\) the mean and \(M_2/W\) the variance (the population
variance in time, with no \(W/(W - w)\) correction). Each step’s state stands for the whole step, as backward
Euler does: the averages are the time integrals of the piecewise-constant answer, a right-end rectangle rule. For a
smoothly varying answer that rule differs from the time integral of the continuous signal by
\(O(\Delta t)\), the order of the scheme itself. The update gives the two-pass weighted mean and variance of
the same series up to round-off (UnsteadyOutput.AveragesAreTheWeightedSeries: relative \(10^{-14}\) for the
mean and \(10^{-11}\) for the variance in double precision; \(10^{-6}\) and \(10^{-3}\) in single, where
the fields and the update are float). Why this form: one pass, no sum of squares (whose difference would lose
the variance of \(\theta\), about \(10^{-11}\) K² on 300 K, to round-off), and the same answer for any step
lengths.
Probes. unsteady.probes_file lists points x y h. A probe’s column is the cell containing
\((x, y)\) (a point on the domain’s high face takes the last cell); its height above the ground of a cell is the
mean of its 8 node heights less the mean of the 4 bottom nodes of the column. The probe takes the cell whose height is
nearest \(h\), the lower on a tie; values are the cell’s, not interpolated. The CSV’s header lists each probe’s
point, cell and cell height; then one row t, p1_u, …, pN_tke at the start and one per step, printed to 17 digits.
Plotfiles and checkpoints. Every unsteady.plot_interval and unsteady.checkpoint_interval physical steps,
and at the end. A checkpoint adds a text record Unsteady (the time, the step count, the fixed \(\Delta t\),
\(t_a\), \(W\) and the time anchor below, as hexadecimal floats) and \(\bar x\) and \(M_2\) as MultiFabs. A run that
diverges writes no final checkpoint: its state is not the state at the record’s time.
Restart. From an unsteady checkpoint the run takes its state, history, averages and record. The time goes on from the record’s. With a fixed \(\Delta t\), total step \(N\) ends at
with the anchor \((t_\star, n_\star) = (0, 0)\) at the start of a run. The anchor moves to
\((t^N, N)\) after a clipped last step (one that ends at unsteady.stop_time more than 0.1 % of
\(\Delta t\) from the formula), and to \((t_0, n_0)\) at a restart with another \(\Delta t\) than the
checkpoint’s, where \(t_0\) and \(n_0\) are the checkpoint’s time and step count. The record keeps the anchor.
Why: the time is a product, not a sum of rounded steps, so a restart with the same \(\Delta t\) gives the times
of the run without the restart bit for bit; after a clipped last step the next step is one \(\Delta t\) later; a
chain of restarts gives the times of one restart (TimeDriver.ResumeContinuesTheTime). With unsteady.cfl
the time is the sum of the steps. Steps are numbered on from \(n_0\). The probe file (<io.plot_file>_probes.csv,
so the restart must run where that path finds the earlier file) keeps its rows up to \(t_0\), and its header must
match the probes given; the log gives the time of the last row kept. A restart that finds no probe file starts a new
one at \(t_0\) and says so in the log. A checkpoint at a step boundary continues the run bit for bit
(rt_unsteady_hill2d). A last step that ended within 0.1 % of \(\Delta t\) of the formula keeps the anchor:
the first step after a restart then spans \(t^{N+1} - t^N\), which differs from the \(\Delta t\) of its
time term by at most that 0.1 %; the averages weight it by the span. unsteady.average_start must equal the record’s, and unsteady.stop_time must be later
than \(t_0\). A steady run may also start from an unsteady checkpoint; it takes the state and history and ignores
the averages.
Convergence criteria
A run is declared steady when the residuals and the monitored quantities have both converged.
Residuals. With \(R_q^n\) the scaled norm of equation \(q\) at iteration \(n\), every equation must satisfy
\(\tau_R\):
steady.residual_tol(5e-5).\(N\):
steady.residual_drop_orders(4), the orders of magnitude the residual must drop.\(K\):
steady.residual_ref_iterations(5). The reference is the peak over the first \(K\) iterations, so the drop does not depend on how low the first iteration happened to start.
The scaled norm is the root mean square of the equation’s residual over the cells (faces for the velocities), divided by a fixed scale:
\(r_{q,i}\): the residual of equation \(q\) at cell or face \(i\); \(N_q\): their number. Each point counts once: a face shared by two boxes, or a periodic face and its image, is one point, so the norm does not depend on
mesh.max_grid_size. (Before audit A4 such a face counted twice. A checkpoint written before then holds its reference and residuals in that norm; a restart compares them with the new norm, which differs by about the share of seam faces, so the restarted run is not bit for bit the run it was. The driver history records the norm from format 4 on; a restart from format 1 to 3 says in the log that it cannot tell.)\(U\) = 10 m/s, \(\theta_s\) = 10 K, \(k_s\) = 0.1 m²/s²: fixed scales, not inputs.
\(H\): the depth of the layers, from the ground level to the top level. On a uniform mesh this is
mesh.prob_hiz; withmesh.stretch_ratioormesh.zlevels_fileit is where the layers end, which can be well abovemesh.prob_hiz (749 m against 440 m on Askervein). Before this was fixed, \(H\) wasmesh.prob_hiz on every mesh, so on a stretched mesh the norms were smaller by that ratio.\(H\) is a property of the mesh, not of the flow. The same flow on a taller mesh has larger norms, so \(\tau_R\) is a tighter test there. The drop \(N\) does not depend on \(H\).
Monitors. Quantities of interest, such as the hilltop speed-up or \(u_*\), must also settle. Each monitor \(M_j\) must stay within \(T_j = \tau_M\max(|M_j|, s_j)\) over the last \(W\) iterations, and its extrapolated limit (Aitken’s method) must lie within the same distance:
where \(d_1, d_2\) are the changes over the two halves of the window, \(\tau_M\) is
steady.monitor_rel_tol (1e-3) and \(W\) is steady.monitor_window (50). The extrapolation matters: for
a slowly converging monitor, a small change over the window alone can stop 10 % away from the limit.
The outcome is Steady, NotSteady (listing what has not converged) or Diverged (a NaN or infinity).
Oscillations with a period up to \(4W\) are never declared steady.
Confidence stop. A run can sit at a fixed state where most of the field has converged
and a few cells (often in a separated lee) have not. With steady.confidence \(= c > 0\) the run is also
steady when
the share of cells that meet the residual criterion cell by cell is at least \(c\). A cell meets it when \(|r_{q,i}|/s_q \le \min\bigl(\tau_R,\ 10^{-N} \max(R_q^\mathrm{ref}, \tau_R)\bigr)\) for \(p, \theta, k\) at the cell and \(u, v, w\) on both of its faces: the threshold of the root-mean-square criterion, applied to each cell (if every cell meets it, so does the norm);
every monitor criterion above is met; and
with
steady.confidence_height\(> 0\), every cell that does not meet it lies within that height above the ground.
The stop is not tried during the reference hold. Its limits: the residual of the cells that do not meet the
threshold is not bounded (1 % of cells at 100 times the threshold give a norm 10 times the threshold), and the
monitors are global, so a limit cycle confined to a few cells that does not move them can be declared steady. The
iteration log prints the share and the height of the highest unconverged cell (after r in the table), and a run
stopped this way names the stop in its verdict. The SIMPLEC stage computes it; the stop does not act during the Newton stage. A stop by
confidence is not recorded in the checkpoint (a restart from it iterates again).
Stage switching, restart and reproducibility
Stage switching. The driver moves to the next stage after
steady.switch_windowiterations belowsteady.switch_residualwith the monitors settled, back when the residual grows bysteady.switch_back_growthover a window, and back at once when the Newton stage asks (Switching and fallback).Restart. The driver records every iteration (stage, CFL, residuals, monitors, wall time, allocation units) in the checkpoint. A restarted run reproduces the uninterrupted run bit for bit on the same ranks and box layout, over flat ground and over terrain. The restart restores the valid points; the stage fills the ghost cells before the first step (the fill is a fixed point, Boundary conditions, so the extra fill changes nothing). The checkpoint also records \(H\). A checkpoint written before it did (driver history format 1 or 2) had its norms scaled with
mesh.prob_hiz; on a stretched mesh the restart multiplies the stored reference and residuals by \(H\) /prob_hiz and says so in the log. That run then continues in the new scale, and so is not bit for bit the run it was.Reproducibility. Norms and monitors are reduced over all ranks, so every rank reaches the same verdict at the same iteration. They are sums in a fixed order, the same bits on any number of ranks for a given box layout, as are the Coriolis level mean and the mean the Poisson solve removes. The pressure solver’s own reductions (any
solver.poisson_backend) are not, so a run on a different number of ranks agrees with the original to round-off, not bit for bit.steady.line_order = diagonalreproducessequentialbit for bit.