Solution algorithm ================== The steady equations :math:`R(q) = 0` are nonlinear and strongly coupled, so they cannot be solved in one step. GUST approaches the solution by a *pseudo-time* iteration: it marches an artificial time until nothing changes any more. The march does not need to be time-accurate, so each cell may take its own, as-large-as-stable time step. Only the end point matters. Pseudo-time iteration --------------------- Each iteration solves for a correction :math:`\delta q`: .. math:: :label: pseudo-time \left(\frac{1}{\Delta\tau} + A\right)\delta q = -R(q), \qquad q \leftarrow q + \delta q . - :math:`R(q)`: the full second-order residual (:doc:`discretization`). - :math:`\Delta\tau`: the local pseudo-time step. - :math:`A`: an approximation of the Jacobian :math:`\partial R/\partial q`. A *stage* chooses :math:`A` and the way :math:`\delta q` is computed. When :math:`\delta q = 0`, :math:`R(q) = 0`, whatever :math:`A` is. A cheaper or rougher :math:`A` changes how fast the iteration converges, not the answer. The production stage is SIMPLEC (``steady.stage = simplec``), optionally with grid sequencing (below). The Newton–Krylov stage (``steady.stage = simplec_newton``) finishes the run; multigrid is in development. Local time step and CFL ramp ---------------------------- In every cell the pseudo-time step is set by a CFL number: .. math:: \Delta\tau = \frac{\mathrm{CFL}}{\dfrac{|u_c|}{\Delta x} + \dfrac{|v_c|}{\Delta y} + \dfrac{|\Omega_c|}{J\,\Delta\zeta} + 2\nu\left(\dfrac{1}{\Delta x^2} + \dfrac{1}{\Delta y^2}\right)} . Vertical diffusion is treated implicitly, so it does not limit the step. **Step floor.** The rate in the denominator is kept at or above a fraction of the level's fastest convective rate (``steady.dtau_floor``, :math:`\varphi` = 0.1 by default): .. math:: \Delta\tau \le \frac{\mathrm{CFL}\,\min(\Delta x, \Delta y)}{\varphi\, U_{\max}}, \qquad U_{\max} = \max_{\text{faces}}\,(|u|, |v|), so a cell at rest, in a recirculation pocket or under a dense forest canopy, steps at most :math:`1/\varphi` times as the fastest cell does, instead of :math:`\mathrm{CFL}/10^{-12}` s. Without the floor such a cell's momentum rows have :math:`1/\Delta\tau \to 0`; with a canopy its drag derivative :math:`f(|U| + u_c^2/|U|)` vanishes with the wind as well, so the SIMPLEC :math:`d = a_P - \sum a_{nb}` and with it the bound on the pressure correction's velocity update in the cell vanish, and the iteration cycles once the CFL cap passes about 3 (:doc:`canopy`, steep terrain and the boundaries: the forested escarpment and the same forest on flat ground, both steady at the default cap with the floor). This is the observation, not a derivation: a 2-D SIMPLEC model with the same terms (quadratic drag, its derivative diagonal, lagged pressure, local steps) converges at every CFL, so what closes the loop in the 3-D solver is not identified. :math:`\varphi` from 0.03 to 1 converges them (182 and 256 iterations at 0.1, 124 and 165 at 0.3, 109 and 101 at 1 on the flat forest and the escarpment). The default 0.1 is the largest value that leaves the empty cases' pseudo-time paths close to what they were: the 2-D hill takes 4576 iterations at :math:`\varphi` = 0, 0.1 and 0.3 and its Newton switch stays near iteration 1812 at 0.1 and 0.3 (4155 at 1); the steep ridge with the :math:`w` cap takes 2234 iterations at 0, 2012 at 0.05, 2558 at 0.1 and more than 3500 at 0.3 and 1 (its residual rises to 4 to 6 before it falls, and the floor lengthens that excursion); the thin-cell cap case is unchanged at 0.1 and 0.3 and needs the cap less at 1 (287 iterations without it against 3713). :math:`U_{\max}` is one all-reduce per iteration, a maximum, so it is the same on any ranks; ``steady.dtau_floor = 0`` is the plain local step, the form before the floor. The floor changes the pseudo-time path, not the converged answer. **Cost of the global form.** The floor binds wherever a cell's own rate is below :math:`\varphi` of the fastest cell's. On a neutral log law (12 m/s aloft, 20 m cells, an 8 m first layer) it binds nowhere at :math:`\varphi` ≤ 0.3; under a 25 m/s low-level jet over a slow surface (0.9 m/s at 4 m) it binds in most of the column at 0.1 and shortens the near-ground step 2.8 times, so a stable high-shear run converges at a tenth of the jet's rate there (``steady.dtau_floor = 0`` or a smaller value restores the local step). Not tried: a rate from the cell's own face speeds (the cell-centred :math:`|u_c|` of a reversal cell is 0 while its faces move, and the arithmetic face mean of :math:`\Delta\tau` hands half of a rest cell's step to its neighbours' faces) or a neighbourhood rate, which would bound such cells locally with no all-reduce; :math:`\varphi` = 0.05 was not swept on the forest cases. The CFL number starts small and grows as the residual falls (switched evolution relaxation, SER; Mulder and van Leer 1985): .. math:: \mathrm{CFL}_{n+1} = \min\bigl(\mathrm{CFL}_\mathrm{max},\ g\,\mathrm{CFL}_n,\ \mathrm{CFL}_0\,r_n^{-p}\bigr). - :math:`\mathrm{CFL}_0`, :math:`\mathrm{CFL}_\mathrm{max}`: ``steady.cfl0`` and ``steady.cfl_max``. - :math:`r_n`: the largest residual relative to its reference value (:ref:`convergence-criteria`). - :math:`p`: ``steady.ser_exponent``; :math:`g`: ``steady.ser_max_growth``, the largest growth per iteration. SIMPLEC stage ------------- SIMPLEC (Van Doormaal and Raithby 1984) solves the momentum equations first, then corrects pressure and velocity so that mass is conserved. One iteration: #. Fill the ghost cells and evaluate the wall model at the current state :math:`q`. #. Evaluate the residual :math:`R(q)` and the turbulence model. #. **Momentum predictor.** Solve :math:`(1/\Delta\tau + A_1)\,\delta\mathbf u = -R_\mathbf u`, giving :math:`\mathbf u^*`. #. **Pressure correction.** Solve a Poisson equation for :math:`\phi` and set :math:`\mathbf u = \mathbf u^* - \nabla\phi/d` and :math:`p \leftarrow p + \alpha_p\phi`. #. **Scalars.** Take one step of the same form for :math:`\theta` (with its convection updated to the corrected velocity) and :math:`k`; bound :math:`k` below by its floor. :math:`A_1` is the first-order operator of :doc:`discretization`. Momentum predictor and line solvers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ABL meshes have cells much thinner than they are wide, so the vertical coupling is the strongest. The predictor therefore solves each vertical column exactly (a tridiagonal solve with the Thomas algorithm). The coupling to the horizontal neighbours is handled by sweeps; ``steady.sweep`` selects how: - ``zjacobi`` (default): **z-line Jacobi.** All columns are solved at once, with the horizontal neighbours taken from the previous sweep. ``steady.zline_sweeps`` (at least 2) sets the sweeps per iteration. More sweeps allow a larger stable CFL; the terrain cases use 16 sweeps with a CFL cap of 4. - ``lusgs``: **line LU-SGS** (lower–upper symmetric Gauss–Seidel). The columns of each box are visited in order: a forward pass uses the already-updated lower neighbours, and a backward pass the updated upper neighbours. One sweep is a forward and a backward pass. When the wind blows along the sweep direction, a pass resolves the upwind coupling exactly, which allows a much larger CFL. The hill cases use two sweeps with a CFL cap of 16. When :math:`u` and :math:`v` have opposite signs, one sweep is stable only to a CFL of about 2.5 (two sweeps to about 33), so use at least two sweeps. Box faces are coupled with lagged values, so the iteration path (not the answer) depends on ``mesh.max_grid_size``, and boxes shorter than the CFL number slow the iteration. Four options refine the LU-SGS sweeps. They change the path to the solution, not the solution. .. list-table:: :header-rows: 1 :widths: 28 72 * - Input - Effect * - ``steady.line_ghosts`` - ``lagged`` (default): neighbours across a box face take the previous pass's values, exchanged before every pass. ``zero``: no communication inside the sweeps; each box sweeps on its own, with :math:`\delta q = 0` outside it (a restricted additive Schwarz method). * - ``steady.line_overlap`` - With ``zero``: each box sweeps a region grown by this many columns, then keeps only its own points. * - ``steady.line_order`` - ``sequential`` or ``diagonal``. ``diagonal`` visits the diagonal planes :math:`i + j = \mathrm{const}`, one column per GPU thread. It gives the same result as ``sequential``, bit for bit. * - ``steady.line_alternate`` - ``1``: every second sweep reverses the y direction. When :math:`u` and :math:`v` have opposite signs, one of the two sweep pairs then follows the wind. On CPUs, ``lagged`` needs the fewest iterations. ``zero`` avoids communication inside the sweeps, which is expected to matter on GPUs and on many ranks. Measurements are in :doc:`results`. Pressure correction ^^^^^^^^^^^^^^^^^^^ With the SIMPLEC coefficient :math:`d = a_P - \sum a_{nb}`, the pressure correction solves .. math:: \nabla\cdot\Bigl(\frac{1}{d}\,\nabla\phi\Bigr) = \nabla\cdot\mathbf u^*, \qquad \mathbf u = \mathbf u^* - \frac{1}{d}\,\nabla\phi, \qquad p \leftarrow p + \alpha_p\,\phi . SIMPLE neglects the neighbours' velocity corrections and needs a pressure under-relaxation. SIMPLEC assumes they equal the cell's own, so :math:`\alpha_p = 1` works (``simplec.alpha_p``). The Poisson solve stops at the relative tolerance ``simplec.pressure_reltol`` (1e-3). A looser tolerance leaves divergence that drives :math:`w` through buoyancy, and the iteration diverges in stratified flow. Point-implicit buoyancy and rotation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Two couplings oscillate in pseudo-time where :math:`\nu_t` is small: :math:`w` with :math:`\theta` (buoyancy, frequency :math:`N`) and :math:`u` with :math:`v` (Coriolis, frequency :math:`f`). An explicit treatment amplifies each oscillation by :math:`(1 + \omega^2\Delta\tau^2)^{1/2}` per step. The predictor therefore treats both with a linearised backward Euler step: .. math:: w:\ \ a_P \mathrel{+}= \frac{g}{\theta_0}\max(\partial_z\theta, 0)\,\tau_\theta,\quad \mathrm{rhs} \mathrel{-}= \frac{g}{\theta_0}\,\tau_\theta R_\theta ; \qquad u, v:\ \ a_P \mathrel{+}= f^2\max(\tau_c, \tau_\mathrm{own}),\quad \mathrm{rhs} \mathrel{\mp}= f\bigl(\tau_c\langle R\rangle + \tau_\mathrm{own}(\overline R - \langle R\rangle)\bigr). - :math:`\tau_\theta = 1/(1/\Delta\tau + a_{P,\theta})`: the step allowed by the :math:`\theta` rows. - :math:`\langle R\rangle`: the horizontal mean, on each level, of the other component's residual, each face counted once and summed in a fixed order, so the mean has the same bits on any number of ranks or GPU threads; :math:`\overline R`: its local value. - :math:`\tau_c = 1/\max(|f|/r, a_P - \sum a_{nb})`, with :math:`r` = ``simplec.rotation_relax`` (0.3). - :math:`\tau_\mathrm{own}`: the row's own step. The inertial oscillation is horizontally uniform, so only the level mean uses the long inertial step. Every added term vanishes when :math:`R = 0`, so the steady solution does not change. Prescribed potential temperature ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ With ``simplec.theta = frozen``, :math:`\theta` is data, not an unknown. This is the steady stratified form of van der Laan et al. (2024): the stratification is prescribed, and buoyancy acts in the turbulence closure only. .. math:: \theta(\mathbf x) = \theta_\mathrm{in}\bigl(h(\mathbf x)\bigr),\qquad R_w = R_w^{\,g=0},\qquad N^2 = \frac{g}{\theta_0}\frac{\partial\theta}{\partial z}\ \text{in } c_\mu,\ c_\mu',\ P_b = -\nu_t' N^2. - :math:`\theta_\mathrm{in}(h)`: the inflow profile's :math:`\theta` at the height :math:`h` above the ground, for every ``bc.inflow_profile`` and whether or not a face is an inflow face: the file's θ column with ``file``, the sounding ``bc.theta_profile`` (read by height above the ground) with the analytic kinds. - :math:`h`: the cell centre's height (the mean of its eight nodes) minus the mean of its four ground nodes, on each level's own ground (the coarse levels' ground is smoothed). It is not floored at ``init.z_agl_min``, as the inflow face's ghost cells read the profile. - :math:`R_w^{\,g=0}`: the :math:`w` residual without the buoyancy force :math:`-g(\theta - \theta_\mathrm{ref})/\theta_0`. The predictor's point-implicit buoyancy terms (above) are dropped with it. - :math:`N^2`, :math:`c_\mu`, :math:`c_\mu'`, :math:`\nu_t'`, :math:`P_b`: the closure's stability terms (:doc:`turbulence_model`), unchanged; ``forcing.gravity`` and ``forcing.theta0`` still set them. **What it does.** Every sequenced level sets :math:`\theta` from the inflow profile in place of the sounding or the prolonged field, and logs the range and mean it set. The SIMPLEC stage then skips the :math:`\theta` sweep and reports a :math:`\theta` residual of 0, so the convergence test (:ref:`convergence-criteria`) covers the momentum, the pressure and :math:`k`. The ghost cells of :math:`\theta` are filled as usual (the inflow face takes the same profile), except at the ground: there the two ghost layers continue the first two cells linearly, so the first cell's centred :math:`\partial\theta/\partial z` (its :math:`N^2`) is the profile's between the first two cells. The wall law's gradient would set it instead, and a neutral wall would give :math:`N^2 = 0` in the first cell. The surface layer still sees :math:`\theta` through its stability (neutral unless ``bc.surface_stability`` is set). **Why.** A steady solve of :math:`\theta` with a fixed surface heat flux has no steady state: the surface keeps adding or removing heat. With buoyancy in the momentum equations as well, stable and unstable cases on a forested escarpment diverged. With :math:`\theta` frozen from a near-1-D column and buoyancy in the closure only, all four converged. Van der Laan et al. (2024) use the form for neutral and stable boundary layers; the unstable cases apply it beyond that, with :math:`N^2 < 0` making the closure's :math:`P_b` a source of :math:`k`. The cost: the flow cannot change the stratification. A cold pool in a valley or air lifted over a ridge keeps the upstream profile by height above the ground, and there are no buoyancy forces, so no lee waves, gravity currents or katabatic and anabatic slope flows. Over terrain the omitted force is not small: with :math:`\theta` by height above the ground, :math:`\partial\theta/\partial x = -\Gamma\tan\alpha` at fixed :math:`z` (:math:`\Gamma` the profile's gradient, :math:`\alpha` the slope), and across an escarpment of height :math:`H` the buoyancy :math:`g\Gamma H/\theta_0` can be of the order of the advection :math:`U^2/L`. That field is not in hydrostatic balance, which is why it cannot be combined with the buoyancy force. Van der Laan et al. (2024) apply the form on flat terrain with a constant :math:`N`. **Limits.** Refused with ``steady.stage = simplec_newton`` (the Newton stage solves :math:`\theta`), with ``io.restart`` (the checkpoint holds no record of the option), without grid sequencing (``steady.sequencing = 1`` and ``mesh.hierarchy_levels`` > 1: the levels set :math:`\theta`), and with a stratified column first guess (``init.first_guess = column`` with a non-neutral ``init.column_stability``, whose evolved :math:`\theta` the frozen field would replace). With :math:`\theta` frozen, the inputs that act only on the :math:`\theta` equation have no effect: a Rayleigh layer or sponge on ``theta`` (``bc.rayleigh_fields``) and the :math:`\theta` gradient through the top (``bc.zhi_theta_grad``); ``bc.surface_heat_flux`` still sets the surface layer's stability but adds no heat. A checkpoint written with ``frozen`` can be restarted without it; the restart then solves :math:`\theta` from the frozen field, with the buoyancy back in the momentum equations. With the leaves' heat exchange (``canopy.heat = fixed``), ``canopy.heat_frozen = blend`` sets :math:`\theta` in the canopy between the frozen field and the leaves' :math:`\theta` each iteration, and ``skip`` (the default) leaves it out (:doc:`canopy`, canopy heat). Vertical-velocity cap ^^^^^^^^^^^^^^^^^^^^^ On meshes with thin first cells, :math:`w` converges slowly. The vertical neighbour weights :math:`a_{ZM}, a_{ZP}` of the :math:`w` rows cancel in :math:`d`, so the pressure correction moves :math:`w` too far by the vertical von Neumann number .. math:: \mathrm{VNN} = \Delta\tau\,(a_{ZM} + a_{ZP}), and the next predictor undoes most of the move. ``simplec.w_vnn_cap`` = :math:`V \ge 16` adds a term on the :math:`w` rows only (not on the ground and top rows): .. math:: \frac{1}{\Delta\tau} \;\rightarrow\; \frac{1}{\Delta\tau} + \frac{a_{ZM} + a_{ZP}}{V} \quad\text{in } a_P \text{ and in } d, \qquad \mathrm{VNN}_\mathrm{eff} = \frac{\mathrm{VNN}}{1 + \mathrm{VNN}/V} < V . The cap changes the path, not the steady answer: the predictor is in delta form, and :math:`d` enters only the pressure correction (there is no Rhie–Chow interpolation on the staggered grid), so the fixed point is still :math:`R = 0`. It is off by default. Values below 16 are refused: on flat ground the iteration diverges for :math:`V \le 2` at a CFL of 16 and for :math:`V = 4` at a CFL of 64. *Sloping ground.* On a :math:`z`-face the pressure correction scales the whole normal gradient, slope term included, by :math:`\beta_z = 1/d_w`, while the :math:`x`- and :math:`y`-faces use :math:`\beta_h = 1/d_u, 1/d_v`. For one Fourier mode (wavenumber :math:`k` along a slope :math:`h`, :math:`m` in :math:`\zeta`) the operator's quadratic form is .. math:: \beta_h k^2 + \beta_z (1 + h^2)\, m^2 - h (\beta_h + \beta_z)\, k\, m , which is definite only while :math:`r = \beta_z/\beta_h` exceeds .. math:: r_c = \frac{h^2}{\left(\sqrt{1 + h^2} + 1\right)^2} . Without the cap :math:`r \approx 1`. With it :math:`r = 1/(1 + \mathrm{VNN}/V)`, so thin cells on steep slopes cross :math:`r_c`: on Askervein's steepest slope (23°, :math:`r_c = 0.041`) a cap of 16 fails once VNN exceeds 370, as with a 0.4 m first cell. ``simplec.w_vnn_cap_slope_margin`` = :math:`s` (default 4) limits the term on each :math:`w` row to .. math:: \min\left(\frac{a_{ZM} + a_{ZP}}{V},\; \frac{\max\bigl(0,\, 1/(s\,r_c) - 1\bigr)}{\Delta\tau}\right), with :math:`h^2 = h_x^2 + h_y^2` from the face's node heights (the slope the operator's cross term uses). The limit assumes that the horizontal coefficient and the vertical one without the cap are both :math:`1/\Delta\tau`. Then :math:`r \ge s\,r_c` wherever :math:`s\,r_c < 1` (below 53° at :math:`s = 4`; steeper rows get no cap). On the solver's own coefficients the margin is smaller where the vertical coefficient is stiffer (stable stratification, the Rayleigh layer: at :math:`s = 4` and 32.5°, :math:`r \ge 3.0\,r_c` when it is twice :math:`1/\Delta\tau` and :math:`1.7\,r_c` at five times), and larger in the first layer, where the surface drag stiffens the horizontal coefficient. The definite band is also bounded above, :math:`r < 1/r_c`. On steep slopes the cap acts as a larger local :math:`V`, and flat ground keeps :math:`(a_{ZM} + a_{ZP})/V` exactly. Giving the slope term :math:`\beta_h` instead (``FaceCoeffs::z_cross``) would keep the operator definite for any :math:`r`, but the default Poisson back end has no split coefficient. With :math:`s = 0` (the cap of #32) the run warns over terrain when the mesh aspect ratio is above the tested 50.78. See :doc:`results` for the tested cases. Pressure Poisson solver ----------------------- The pressure correction, the mass-consistent initial field and the Tucker wall distance all solve .. math:: \frac{1}{J}\Bigl[\delta_\xi\bigl(a_x\beta_x f_x\bigr) + \delta_\eta\bigl(a_y\beta_y f_y\bigr) + \delta_\zeta\bigl(a_z\beta_z f_z\bigr)\Bigr] = \mathrm{rhs}, where :math:`f` are ERF's face gradients, terrain cross terms included, and :math:`\beta` are face coefficients (:math:`1/d` for SIMPLEC). ``solver.poisson_backend`` selects the solver: .. list-table:: :header-rows: 1 :widths: 20 80 * - Back end - Method * - ``gmres_mlmg`` (default) - GMRES preconditioned by AMReX's ``MLTerrainPoisson`` geometric multigrid. The multigrid coarsens in x and y only and solves each column exactly, which suits thin cells. * - ``mlmg`` - The same multigrid alone. * - ``gmres_fft`` - ERF's ``TerrainPoisson`` operator with GMRES and ERF's FFT preconditioner (the flat stretched Laplacian: FFT in x and y, tridiagonal in z). The default needs a nearly constant number of iterations per solve as the first layer becomes thinner, where the FFT preconditioner needs more (:doc:`results`). The solve stops at ``solver.poisson_reltol`` or ``solver.poisson_abstol``; the pressure correction uses ``simplec.pressure_reltol`` instead. ``gmres_mlmg`` measures the residual weighted by cell volume (:math:`J`), ``gmres_fft`` the plain residual and ``mlmg`` the maximum norm. The same tolerance therefore does not mean the same divergence where :math:`J` varies. With no Dirichlet face the mean of the right-hand side is removed first. Grid sequencing --------------- Most of the iterations of a run remove large-scale errors, which the fine mesh removes slowly. Grid sequencing removes them first on coarse meshes, where an iteration is cheap, and then starts each finer mesh from the coarser solution. It uses the grid hierarchy of :doc:`grid_hierarchy`. ``steady.sequencing = 1`` with ``mesh.hierarchy_levels`` = :math:`L \ge 2` runs, on a fresh start: #. **Coarsest level** :math:`L - 1`: the initial field (``init.*``) on that mesh, then SIMPLEC to a *loose stop*. #. **Levels** :math:`L - 2, \dots, 1`: the solution of the level above is prolonged (interpolated, then made divergence-free), then SIMPLEC runs to a loose stop. #. **Level 0** (the run's mesh): prolonged from level 1, then SIMPLEC to the normal convergence criteria. Each level has its own boundary conditions, turbulence model, wall distance and Poisson solver. An iteration on level :math:`l` costs about :math:`4^{-l}` of a level-0 iteration. ``steady.max_iterations`` applies to each level separately. The log ends with a table of every level's iterations and wall time. **Loose stop.** A coarse level stops when, for every residual :math:`q`, .. math:: R_q^n \le \max\!\left(R_q^{\mathrm{ref}}, \tau_R\right) 10^{-d_\ell} \quad\text{or}\quad R_q^n < \tau_R , with :math:`d_\ell` = ``steady.sequencing_drop_orders`` (2), or when the full convergence criteria hold. ``steady.sequencing_monitor_tol`` adds an optional monitor test; it is off by default, because right after a prolongation the monitors settle before the residuals have dropped. **Residual reference.** A prolonged start is already close to the solution, so a level's own first residuals are small. Measuring the drop from them would hold level 0 to a stricter standard than a run without sequencing. Every finer level therefore starts with the reference residuals :math:`S_q` of the coarsest level, which, like a run without sequencing, starts from the initial field. **Reference hold.** A driver normally runs its first ``steady.residual_ref_iterations`` iterations at ``steady.cfl0`` while it takes its reference. A seeded level already has its reference, so by default (``steady.sequencing_hold = 0``) it skips this hold. From the first iteration it uses .. math:: r_n = \max_q \frac{R_q^n}{\max\bigl(R_q^{\mathrm{ref},n}, \tau_R\bigr)}, \qquad R_q^{\mathrm{ref},n} = \max\bigl(S_q, R_q^0, \dots, R_q^{\min(n, K - 1)}\bigr), in the CFL ramp, which then climbs at once (still limited by ``steady.ser_max_growth``). ``steady.sequencing_hold = 1`` keeps the hold on every level. ``steady.sequencing_cfl_start = previous`` starts each seeded level at the CFL the previous level reached, instead of ``steady.cfl0``. **What it does not change.** The converged state is the same with and without sequencing, to the convergence tolerance. A restart continues level 0 only; the coarse levels run on a fresh start. While the coarse levels run, the level-0 solver stays allocated, so the peak memory is somewhat above that of level 0 alone. On the 3-D hill, three levels reduce the level-0 iterations from 2091 to 706 and the wall time from 145 s to 54 s (:doc:`results`). Newton–Krylov stage ------------------- With ``steady.stage = simplec_newton`` a Newton–Krylov stage finishes the iteration once SIMPLEC is close to the solution. Each Newton step solves .. math:: (D + J)\,\delta q = -R(q) by GMRES, with the Jacobian–vector products of :doc:`discretization`. - :math:`J = \partial R/\partial q`, including the dependence of :math:`\nu_t` on :math:`q`. - :math:`D`: :math:`1/\Delta\tau` on the momentum, :math:`\theta` and :math:`k` rows; 0 on continuity and boundary rows. :math:`D = 0` is plain Newton. - The fields are scaled so GMRES weighs them alike. Its inputs (``steady.newton_*``) are listed in :doc:`inputs`. Switching and fallback ^^^^^^^^^^^^^^^^^^^^^^ ``NewtonStage`` (``Source/NewtonStage.H``) is the driver's second stage. - **Switch.** The driver moves from SIMPLEC to Newton after ``steady.switch_window`` (20) consecutive iterations with the relative residual at or below ``steady.switch_residual`` (1e-4) and every monitor settled: changed by at most ``steady.switch_monitor_tol`` (1 %) of max(\|value\|, scale) over the window. - **Entry.** The smoothness handling below enters (the k floor frozen, the smooth limiter in the Newton residual), and the pseudo-transient continuation starts afresh at ``steady.cfl_max``. Its CFL doubles (``steady.ser_max_growth``) with every accepted step up to ``steady.newton_cfl_max`` (128); with ``steady.newton_dtau_inf_at_cap = 1`` :math:`\Delta\tau` is then infinite. At every finite step the preconditioner uses the step's own :math:`\Delta\tau`, and at infinite :math:`\Delta\tau` the Newton cap's. A residual-driven ramp (the SER law of the SIMPLEC stage) settles near CFL 13 on the hill, where :math:`|R|` falls slowly; jumping to infinite :math:`\Delta\tau` after one step leaves the slow modes unpreconditioned (GMRES(30) and GMRES(200) stall at the 2000-iteration limit on ERF's 2-D hill). On the 8 × 2 × 8 hill, from the switch to steady: .. list-table:: :header-rows: 1 * - Newton CFL cap - After the cap - Newton steps - GMRES iterations * - **128 (default)** - **infinite** - **8** - **480** * - 1000 - infinite - 11 - 718 * - 1e4 - infinite - 13 - 882 * - 1000 - held - 28 - 2058 * - 128 - held - 102 - 5338 - **Each iteration** is one continuation step. The residual norms and monitors the driver records are those of the run's own equations at the new state (the run's limiter, the closure's live decisions), so the steady criterion and the relative residual mean what they mean for SIMPLEC. That costs one more residual evaluation per step. Once the continuation has converged on equations that are the run's own (\|R\| at or below :math:`10^{-\text{drop}}\,|R_0|`, with ``steady.residual_drop_orders``), the stage takes no more steps; the driver's criterion then fills its monitor window at one residual evaluation per iteration. - **Fallback.** After ``steady.newton_fallback_failures`` (2) consecutive failed steps (every retry used) the stage hands the run back to SIMPLEC at once, whatever the hysteresis. SIMPLEC may switch forward again after the window; after three failed visits the stage refuses (SIMPLEC finishes the run). The switch-back rule on residual growth (``steady.switch_back_growth``) applies too. A stage that has converged on the run's own equations but whose steady criterion has not held for ``steady.monitor_window`` iterations hands back as well. - **Final pass.** When the Newton stage's equations differ from the run's (the limiter was swapped, or a k-floor decision moved), it hands the run back once its own residual has converged, and refuses any later switch: the run ends on SIMPLEC with the run's own equations. With ``convection.limiter = van_albada`` there is no swap. - **Restart.** The checkpoint holds ``/NewtonStage``, the continuation state and the smoothness state (written before the stage is left at the end of a run); a restart inside the stage continues it bit for bit, with either preconditioner (measured on the 2-D hill, a checkpoint three Newton steps in). The stage fills the ghost cells on entry (``NewtonStage::init``) where the run it continues does not; the fill is a fixed point, so this changes nothing. A ``steady.stage = simplec`` checkpoint continues with ``simplec_newton`` (the Newton stage starts with an empty account); the reverse aborts, naming ``steady.stage``. - **Grid sequencing.** The coarse levels run SIMPLEC only (their loose stop ends them first); the Newton stage is level 0's. - **Relaxed** :math:`\nu_t`. With ``turb.nut_relax`` < 1 the closure's history is the last SIMPLEC iteration's, which the Newton stage does not update, so the stage's equations are not the steady ones: it hands back for the final pass. **Limits.** - Before the Newton CFL cap, the stage went to infinite :math:`\Delta\tau` after one step; GMRES(30) and GMRES(200) then stalled on ERF's 2-D hill (2000 iterations at a Krylov residual of 0.9) and the run cycled between the stages. - On that hill at full size (128 × 1 × 64, no Coriolis) the ramp does not remove the stall: GMRES(30) needs 105, 180 and 323 iterations at CFL 16, 32 and 64, and reaches the 2000-iteration limit from CFL 128 on (Krylov residual 0.08 at 128, 0.75 at 256, 0.97–0.99 at infinite :math:`\Delta\tau`); those steps are still accepted, and :math:`|R|` falls from 1.8e-4 to 7.8e-5 over 11 steps. The SIMPLEC preconditioner holds GMRES(30) to about CFL 64 on that grid (the measurement used a cap of 1000; the default is now 128). The cap, the restart length and the switch residual are provisional: a Newton study on Askervein has not been run yet. - The stage's continuation takes its momentum diffusivity from the closure at its own state, not from the SIMPLEC stage (whose is not restored by a restart inside the Newton stage). ``steady.newton_gmres_max_iters`` bounds the GMRES iterations of one solve over all its restarts. A solve that reaches it is unconverged; its update is used when the Krylov residual is finite and below :math:`|b|`, like any other. The Newton preconditioner ^^^^^^^^^^^^^^^^^^^^^^^^^ GMRES solves the Newton system with a right preconditioner :math:`M^{-1} \approx (D + J)^{-1}`, chosen by ``steady.newton_preconditioner`` (``SimplecPreconditioner``, ``Source/NewtonPreconditioner.H``). ``identity`` is no preconditioner. ``simplec`` (the default) applies one SIMPLEC iteration to the linearized system :math:`(D + J)\,\delta q = r`, starting from :math:`\delta q = 0`: 1. The momentum predictor of the SIMPLEC stage, :math:`(1/\Delta\tau + A_1)\,\delta u^* = r_u`, with the coefficients, rotation and buoyancy terms at :math:`q`. 2. The pressure correction with the continuity part :math:`r_p` as a source, .. math:: \nabla\cdot\left(\frac{1}{a_P}\nabla p'\right) = \nabla\cdot\delta u^* - r_p, \qquad \delta u = \delta u^* - \frac{1}{a_P}\nabla p', \qquad \delta p = \alpha_p\,p', so that :math:`\nabla\cdot\delta u = r_p`. :math:`a_P` is the predictor's diagonal and :math:`\alpha_p` is ``simplec.alpha_p``. 3. The :math:`\theta` and :math:`k` line steps on their parts of :math:`r`. The change of the :math:`\theta` convection by :math:`\delta u` is a directional derivative, :math:`[C(F(u + \sigma\delta u)) - C(F(u))]/\sigma` with :math:`\sigma` as in the Jacobian-vector product, and the line operators use the transport of :math:`u`, so ``apply`` is linear in :math:`r` (to the finite-difference and Poisson tolerances). With the stage's denominator (below) the full :math:`\delta u` and the transport of :math:`u + \delta u` are used instead, as in the stage. The increment :math:`\delta k` is not clipped. 4. On the Dirichlet rows of :math:`u, v, w`, which are identity rows of the Newton system, :math:`\delta q = r`. The sweeps are those of the SIMPLEC stage: ``steady.zline_sweeps`` of ``steady.sweep``. ``setup`` rebuilds the coefficients once per Newton step, at the current state and :math:`\Delta\tau`. Inside the continuation loop that is every attempt. At infinite :math:`\Delta\tau` the preconditioner uses the local time step at ``steady.cfl_max``. **The denominator.** The SIMPLEC stage divides by :math:`d = a_P - \sum a_{nb}`. In the interior that is about :math:`1/\Delta\tau`, so the pressure increment vanishes as :math:`\Delta\tau` grows. The preconditioner therefore uses the SIMPLE denominator :math:`a_P`. On the 3-D hill of ``NewtonPreconditioner.IterationsBoundedOverCfl`` the GMRES iterations per Newton step then stay near 10 from CFL 10 to 1e4; with :math:`d` they grow from 18 to more than 350 (:doc:`results`). With :math:`d` the preconditioner is exactly one iteration of the stage (``NewtonPreconditioner.ApplyIsOneSimplecIteration``). **Shared faces.** A face shared by two boxes, or a periodic seam face and its image, is one unknown. :math:`r` arrives with equal copies, and :math:`\delta q` is returned with every copy equal to the owner's. **The pressure gauge.** Without a Dirichlet pressure face, a constant pressure is the null space of :math:`D + J`. The preconditioner removes the net source of :math:`r_p` that the boundaries cannot carry, and returns :math:`\delta p` with zero :math:`J`-weighted mean. The Newton update then keeps the mean of :math:`p`, and a constant added to :math:`p` changes neither the preconditioner nor the solve (``NewtonPreconditioner.ConstantPressureShiftLeavesTheSolveUnchanged``). **Limits.** - On the horizontally uniform Ekman column the pressure plays no part. The stiff part at large CFL is the balance of rotation and vertical diffusion, and ``simplec`` needs about as many GMRES iterations as ``identity``. - The pressure solve's reductions depend on the rank order, so with ``simplec`` the Newton iterates are not bit for bit the same on different rank counts. With ``identity`` they are. - Memory: about 25 state-size fields and the Poisson solver's hierarchy, on top of the Krylov vectors. Pseudo-transient continuation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A continuation loop (``Continuation``, ``Source/Continuation.H``) wraps the Newton step. The application does not run it yet. One continuation step from :math:`q_n`: 1. :math:`\Delta\tau` from the local time step at :math:`\mathrm{CFL}_n`, or no pseudo-time term (:math:`D = 0`) once the CFL has reached its cap and ``steady.newton_dtau_inf_at_cap = 1``. 2. The direction :math:`\delta q` from :math:`(D + J)\,\delta q = -R(q_n)` (one GMRES solve). 3. A backtracking line search on :math:`|R| = |T^{-1}R(\mathrm{fill}(q))|_2`, the norm of the Newton step: .. math:: |R(q_n + \lambda\,\delta q)| \le (1 - c\,\lambda)\,|R(q_n)|, \qquad \lambda = 1, \beta, \beta^2, \ldots \ge \lambda_\mathrm{min}, with :math:`c` = ``steady.newton_armijo_c``, :math:`\beta` = ``steady.newton_backtrack`` and :math:`\lambda_\mathrm{min}` = ``steady.newton_lambda_min``. Each trial point keeps :math:`k` at or above ``turb.tke_floor`` and goes through the ghost fill before :math:`R` is evaluated. At a finite :math:`\Delta\tau` the direction is a pseudo-time step and need not reduce :math:`|R|`. With :math:`\gamma` = ``steady.newton_ptc_growth`` > 1, when no :math:`\lambda` meets the condition there, the full step is accepted if :math:`|R(q_n + \delta q)| \le \gamma\,|R(q_n)|`; the CFL law then lowers the next CFL. 4. Accepted: :math:`q_{n+1} = q_n + \lambda\,\delta q`, and the CFL follows the SER law of the SIMPLEC stage, .. math:: \mathrm{CFL}_{n+1} = \min\!\left(\mathrm{CFL}_\mathrm{max},\; g\,\mathrm{CFL}_n,\; \mathrm{CFL}_0 \left(\frac{|R_0|}{|R_{n+1}|}\right)^{p}\right), with ``steady.cfl0``, ``steady.cfl_max``, :math:`g` = ``steady.ser_max_growth`` and :math:`p` = ``steady.ser_exponent``. :math:`|R_0|` is the first finite :math:`|R|`, or one the caller supplies (``set_reference_residual``). While :math:`\Delta\tau` is infinite the law starts from :math:`\mathrm{CFL}_\mathrm{max}`. 5. A skipped update or a line search that accepts no :math:`\lambda` leaves :math:`q_n` unchanged, multiplies the CFL by ``steady.newton_cfl_cut`` (from infinite :math:`\Delta\tau`, :math:`\mathrm{CFL}_\mathrm{max}` times it) and retries, up to ``steady.newton_max_retries`` times. Then the step returns *failed*, and the Newton stage will fall back to SIMPLEC. Every attempt prints one line: the step, :math:`|R|` before and after, the CFL (``inf`` for :math:`D = 0`), :math:`\lambda`, the backtracks, the GMRES iterations and status, the Krylov relative residual and the outcome (accepted, skipped or rejected). The continuation state (step count, CFL, the switch to infinite :math:`\Delta\tau`, :math:`|R_0|` and the last :math:`|R|`) is written to and read from a checkpoint directory (``Continuation``, hex floats); a restart gives the uninterrupted run's bits. **Limits.** - With ``steady.newton_ptc_growth = 1`` (the default) a pseudo-time direction that raises :math:`|R|` for every :math:`\lambda` is rejected, and a CFL cut need not help. On the Ekman column with ``steady.ser_exponent = 0.5`` the retries run out at 3.5e-2 of :math:`|R_0|`; with 1.5 the column converges (:doc:`results`). - Once :math:`\Delta\tau` is infinite it stays infinite through damped steps and unconverged solves; only a failed attempt returns it to a finite CFL. Returning to :math:`\mathrm{CFL}_\mathrm{max}` after such a step was tried and measured worse. - With GMRES(30) and infinite :math:`\Delta\tau`, the 30 m hill stagnates near 6e-3 of :math:`|R_0|` for twelve steps at the 2000-iteration limit, then fails at 7e-8. With ``steady.newton_dtau_inf_at_cap = 0`` it reaches 1e-8 in 23 steps, and with GMRES(200) in 8. The restart length and the cap will be tuned with the Newton stage. Smoothness handling ^^^^^^^^^^^^^^^^^^^ The Jacobian-vector product is a forward difference of the residual. Across a kink of the residual it measures the jump of the slope, not the derivative, and Newton loses its convergence rate. ``NewtonSmoothness`` (``Source/Smoothness.H``) handles the two kinks before the switch to Newton: 1. **The limiter.** The van Leer slope :math:`2ab/(a+b)` (0 when :math:`ab \le 0`) has a kink wherever :math:`a` or :math:`b` passes through 0: at every extremum, and in nearly uniform regions where the differences are round-off of either sign. The Newton stage's residual uses the smooth van Albada slope instead (``van_leer`` → ``van_albada``, with ``convection.smooth_eps_*``; ``upwind`` and ``van_albada`` are kept). 2. **The k floor.** With ``turb.frozen_switch = 1`` (the default) the floor decisions are frozen at the state where Newton starts. A cell at the floor there stays at :math:`k = k_\mathrm{floor}` for the whole stage, so :math:`k_f = \max(k, k_\mathrm{floor})` has no kink as its :math:`k` rises; its branch and :math:`R_t` smoothings are decided at that :math:`k`. A cell above the floor is floored again only if its :math:`k` falls to the floor or below, so no :math:`k \le 0` (from a line-search trial or a Jacobian perturbation) reaches the closure. The stable/unstable branch and the :math:`R_t` smoothing are not frozen: they are continuously differentiable (:doc:`turbulence_model`), and freezing them would replace the state's decisions by those of the switch point. After the Newton stage the floor thaws and a final SIMPLEC pass with the run's own inputs (van Leer, live decisions) converges the run's discrete equations from the Newton solution. ``leave`` reports how many floor decisions moved during the stage. The module's state (active, the frozen decisions) goes to the checkpoint (``/Smoothness``, ``Level_0/closure_switches``) and is restored on any layout. **The stability branch stays live.** It is not frozen with the other decisions. In a neutral layer :math:`N^2` is round-off and its sign flips: on the 2-D hill 286–507 of 512 cells changed branch during Newton. With the branch frozen at the switch state Newton was at 6.0e-6 after 8 steps on the inversion (van Leer), against 9.4e-8 without it, and failed at 5.7e-9 on the neutral van Albada run, against 1.4e-13 (:doc:`results`). **Cost.** The two limiters give different discrete solutions (0.4–0.6 % of the largest :math:`u` on the 2-D hill), so the verification pass is not short: 3650–4316 SIMPLEC iterations, 44–74 % of what SIMPLEC needs from 1e-4 to 1e-9 on its own, on top of the Newton stage. On a van Leer run the swap therefore saves little; with ``convection.limiter = van_albada`` for the whole run Newton solves the run's own equations, reaches 4e-11 in 5 steps, and needs no verification pass, at the cost of a 0.4–0.6 % different answer (:doc:`results`). .. _physical-time-term: Physical time term (dual time stepping) --------------------------------------- Unsteady RANS by dual time stepping takes one backward-Euler (BDF1) step of physical length :math:`\Delta t` by a pseudo-time iteration to convergence. Each transported equation gains a physical time term, .. math:: R_q \leftarrow R_q + \frac{q - q^n}{\Delta t}, \qquad q = u, v, w, \theta, k, where :math:`R_q` is the residual per unit volume (as in the rest of this page), :math:`q` the current iterate and :math:`q^n` the solution at the last physical step. Continuity has no time term. When the pseudo-time iteration has driven :math:`R_q` to zero, :math:`q` is :math:`q^{n+1}` of the backward-Euler step. **The implicit rows.** The term's derivative with respect to :math:`q` is :math:`1/\Delta t` on the row's own diagonal. ``implicit_rows`` adds it to :math:`a_P`, to which the callers add the local pseudo-time term :math:`1/\Delta\tau`, so the line solves of :math:`u, v, w, \theta, k`, the momentum predictor and the SIMPLEC denominator :math:`d = a_P - \sum a_{nb}` all carry it. The Newton core's :math:`Jv`, a difference of residuals, sees it without a change. The rotation's step and the buoyancy step of the predictor, which use the same :math:`a_P`, also see :math:`1/\Delta t`: where :math:`\Delta t` is shorter than the rotation's own step, the rotation is advanced by backward Euler over :math:`\Delta t`. The fixed point does not change. **Rows that replace their residual.** The term is added before the rows that replace their residual: the wall-k rows with ``bc.surface_wall_k = dirichlet``, the Dirichlet rows of the boundary conditions and the fixed rows of buildings. ``implicit_rows`` adds :math:`1/\Delta t` before the same rows become identity rows. Such a row carries no time term in either. Rows that add to the residual (damping, the source form of the wall-k rows) keep it. The k rows of cells partly inside a building scale the residual and the row by :math:`1 - f` (:doc:`buildings`), and so keep :math:`1 - f` of the term in both. **Storage.** :math:`q^n` is a copy of :math:`u, v, w, \theta, k` with their ghost cells, held by the ``TimeTerm`` module (``Source/TimeTerm.H``). The residual reads only its valid points. Without a time term the residual and the rows are unchanged, bit for bit. :math:`q^n` is not written to the checkpoint: a checkpoint is written at the end of a step, where :math:`q^n` of the next step is the state itself. The module checks that :math:`\Delta t` is finite and positive with a finite :math:`1/\Delta t`. .. _time-driver: Time driver ----------- With ``steady.stage = unsteady`` the run marches physical time from 0 to ``unsteady.stop_time`` in backward-Euler steps (``Source/TimeDriver.H``). Each step does three things. 1. **The step length.** The ghost cells are filled and :math:`\Delta t` is chosen: ``unsteady.dt``, or with ``unsteady.cfl`` > 0 one global step .. math:: \Delta t = \frac{C}{\max_\text{cells}\left(|u_c|/\Delta x + |v_c|/\Delta y + |\Omega_c|/(J\,\Delta\zeta)\right)}, where :math:`C` is ``unsteady.cfl``, :math:`u_c, v_c` the cell-centred horizontal velocities, :math:`\Omega_c` the cell-centred contravariant vertical velocity and :math:`J\,\Delta\zeta` the cell's height: the convective rate of the local pseudo-time step without its diffusive part or floor. The last step ends the run at ``unsteady.stop_time``: a step within 0.1 % of the time left takes all of it, so no sliver of a step is left. A :math:`\Delta t` that is not finite, or a target CFL over a field at rest (no convective rate), stops the run with a message. The time is accumulated in double precision (:math:`n\,\Delta t` for a fixed step). 2. **The old state.** :math:`q^n \leftarrow q`, and the time term takes :math:`\Delta t`. 3. **The inner iterations.** Pseudo-time iterations of the SIMPLEC stage, with the time term, run until every scaled residual satisfies .. math:: R_q^m \le \max\left(10^{-d}\,R_q^1,\ \tau_R\right), where :math:`R_q^m` is the norm of equation :math:`q` at inner iteration :math:`m`, :math:`d` is ``unsteady.inner_drop_orders`` and :math:`\tau_R` is ``steady.residual_tol``; or until ``unsteady.max_inner`` iterations have run. :math:`R_q^1` is the residual at :math:`q = q^n`, where the time term is zero: the steady residual of the previous step's answer. The floor :math:`\tau_R` ends a step whose start is already steady at once, instead of asking for round-off. **How far to converge a step.** :math:`R_q^1` is about :math:`|\partial q/\partial t|`, so the inner stop leaves an algebraic error of order :math:`10^{-d}\,\Delta t\,|\partial q/\partial t|`, against the backward-Euler error of order :math:`\Delta t^2 |\partial^2 q/\partial t^2|/2` per step. In a 1-D model of Stokes' first problem with a slowly converging inner solver (point Jacobi), the algebraic error at the end was 3 to 5 times the time error with :math:`d = 1`, about half of it with :math:`d = 2` and below a tenth with :math:`d = 3`; with the line sweeps it stayed below 0.06 of it for every :math:`d`. The default is :math:`d = 3`; :math:`d = 2` is cheaper where the time history need not be accurate (a run to a periodic or settled state). **Why the steady stop is not used.** The steady verdict also asks the monitors to settle over a window and the residual to drop from a reference held over the first iterations. Inside a physical step neither applies: the step's answer is not steady, and the reference changes every step. The driver's CFL ramp (SER) runs whatever the verdict says, and the ramp and the histories carry on across steps. The hold at ``steady.cfl0`` lasts the first ``steady.residual_ref_iterations`` inner iterations of the run, which may span several steps (the 2-D hill deck holds for 100 iterations, about two and a half steps). **Restart.** A run may start from a checkpoint of a steady run (``io.restart``). Its physical time starts at 0, the state is the checkpoint's, and the driver's history continues: there is no new hold, and the CFL and the residual reference are the checkpoint's. A ``steady.stage = simplec_newton`` checkpoint cannot be continued (it has two stages). From a checkpoint of an unsteady run the run continues where it was (:ref:`unsteady-outputs`). **What a step reports.** One line per step: the time, :math:`\Delta t`, the inner iterations and the largest :math:`R_q^m / \max(R_q^1, \tau_R)` at the last one. A step that reaches ``unsteady.max_inner`` is marked and the run goes on; the summary counts such steps, and the run still exits normally (check the count). A residual that becomes NaN or infinite ends the run as diverged. The plotfile carries the physical time; its number is the total of inner iterations. With ``io.turb_plot_file`` the residual fields include the time term of the last step, the residual that step converged. **Order.** Backward Euler is first order in :math:`\Delta t`: halving the step halves the time error (``TimeDriver.StokesFirstProblemFirstOrder``, Stokes' first problem against its erfc profile). **Not supported yet.** Grid sequencing, the Newton stage and the hybrid upwind blend (``convection.hybrid_sigma_max`` > 0: it follows the inner residual, so the operator would change within and between steps). A deck that asks for sequencing (``steady.sequencing = 1``) or the blend stops with a message. .. _unsteady-outputs: Unsteady outputs ---------------- An unsteady run (``Source/UnsteadyOutput.H``) keeps running averages, writes probe rows, and writes plotfiles and checkpoints during the run, from which a later run continues. **Running averages.** After each step from :math:`t^n` to :math:`t^{n+1}` the mean and the variance of :math:`x \in \{u, v, w, \theta, k\}` (the cell-centred values, velocities averaged from the faces) are updated by the weighted form of West (D. H. D. West, *Updating mean and variance estimates: an improved method*, Communications of the ACM 22(9), 532–535, 1979): .. math:: w = t^{n+1} - \max(t^n, t_a), \qquad W \leftarrow W + w, \qquad d = x - \bar x, \qquad \bar x \leftarrow \bar x + \frac{w}{W}\,d, \qquad M_2 \leftarrow M_2 + w\,d\,(x - \bar x), where :math:`t_a` is ``unsteady.average_start``, :math:`w` the step's weight (no update if :math:`w \le 0`), :math:`W` the time averaged so far, :math:`\bar x` the mean and :math:`M_2/W` the variance (the population variance in time, with no :math:`W/(W - w)` correction). Each step's state stands for the whole step, as backward Euler does: the averages are the time integrals of the piecewise-constant answer, a right-end rectangle rule. For a smoothly varying answer that rule differs from the time integral of the continuous signal by :math:`O(\Delta t)`, the order of the scheme itself. The update gives the two-pass weighted mean and variance of the same series up to round-off (``UnsteadyOutput.AveragesAreTheWeightedSeries``: relative :math:`10^{-14}` for the mean and :math:`10^{-11}` for the variance in double precision; :math:`10^{-6}` and :math:`10^{-3}` in single, where the fields and the update are ``float``). **Why this form:** one pass, no sum of squares (whose difference would lose the variance of :math:`\theta`, about :math:`10^{-11}` K² on 300 K, to round-off), and the same answer for any step lengths. **Probes.** ``unsteady.probes_file`` lists points ``x y h``. A probe's column is the cell containing :math:`(x, y)` (a point on the domain's high face takes the last cell); its height above the ground of a cell is the mean of its 8 node heights less the mean of the 4 bottom nodes of the column. The probe takes the cell whose height is nearest :math:`h`, the lower on a tie; values are the cell's, not interpolated. The CSV's header lists each probe's point, cell and cell height; then one row ``t, p1_u, …, pN_tke`` at the start and one per step, printed to 17 digits. **Plotfiles and checkpoints.** Every ``unsteady.plot_interval`` and ``unsteady.checkpoint_interval`` physical steps, and at the end. A checkpoint adds a text record ``Unsteady`` (the time, the step count, the fixed :math:`\Delta t`, :math:`t_a`, :math:`W` and the time anchor below, as hexadecimal floats) and :math:`\bar x` and :math:`M_2` as MultiFabs. A run that diverges writes no final checkpoint: its state is not the state at the record's time. **Restart.** From an unsteady checkpoint the run takes its state, history, averages and record. The time goes on from the record's. With a fixed :math:`\Delta t`, total step :math:`N` ends at .. math:: t^N = t_\star + (N - n_\star)\,\Delta t, with the anchor :math:`(t_\star, n_\star) = (0, 0)` at the start of a run. The anchor moves to :math:`(t^N, N)` after a clipped last step (one that ends at ``unsteady.stop_time`` more than 0.1 % of :math:`\Delta t` from the formula), and to :math:`(t_0, n_0)` at a restart with another :math:`\Delta t` than the checkpoint's, where :math:`t_0` and :math:`n_0` are the checkpoint's time and step count. The record keeps the anchor. **Why:** the time is a product, not a sum of rounded steps, so a restart with the same :math:`\Delta t` gives the times of the run without the restart bit for bit; after a clipped last step the next step is one :math:`\Delta t` later; a chain of restarts gives the times of one restart (``TimeDriver.ResumeContinuesTheTime``). With ``unsteady.cfl`` the time is the sum of the steps. Steps are numbered on from :math:`n_0`. The probe file (``_probes.csv``, so the restart must run where that path finds the earlier file) keeps its rows up to :math:`t_0`, and its header must match the probes given; the log gives the time of the last row kept. A restart that finds no probe file starts a new one at :math:`t_0` and says so in the log. A checkpoint at a step boundary continues the run bit for bit (``rt_unsteady_hill2d``). A last step that ended within 0.1 % of :math:`\Delta t` of the formula keeps the anchor: the first step after a restart then spans :math:`t^{N+1} - t^N`, which differs from the :math:`\Delta t` of its time term by at most that 0.1 %; the averages weight it by the span. ``unsteady.average_start`` must equal the record's, and ``unsteady.stop_time`` must be later than :math:`t_0`. A steady run may also start from an unsteady checkpoint; it takes the state and history and ignores the averages. .. _convergence-criteria: Convergence criteria -------------------- A run is declared steady when the residuals and the monitored quantities have both converged. **Residuals.** With :math:`R_q^n` the scaled norm of equation :math:`q` at iteration :math:`n`, every equation must satisfy .. math:: R_q^n < \tau_R \quad\text{and}\quad R_q^n \le 10^{-N} R_q^\mathrm{ref}, \qquad R_q^\mathrm{ref} = \max\Bigl(\max_{m < K} R_q^m,\ \tau_R\Bigr) . - :math:`\tau_R`: ``steady.residual_tol`` (5e-5). - :math:`N`: ``steady.residual_drop_orders`` (4), the orders of magnitude the residual must drop. - :math:`K`: ``steady.residual_ref_iterations`` (5). The reference is the peak over the first :math:`K` iterations, so the drop does not depend on how low the first iteration happened to start. The scaled norm is the root mean square of the equation's residual over the cells (faces for the velocities), divided by a fixed scale: .. math:: R_q = \frac{1}{s_q}\Bigl(\frac{1}{N_q}\sum_i r_{q,i}^2\Bigr)^{1/2}, \qquad s_{u,v,w} = \frac{U^2}{H}, \quad s_p = \frac{U}{H}, \quad s_\theta = \frac{U\,\theta_s}{H}, \quad s_k = \frac{U\,k_s}{H} . - :math:`r_{q,i}`: the residual of equation :math:`q` at cell or face :math:`i`; :math:`N_q`: their number. Each point counts once: a face shared by two boxes, or a periodic face and its image, is one point, so the norm does not depend on ``mesh.max_grid_size``. (Before audit A4 such a face counted twice. A checkpoint written before then holds its reference and residuals in that norm; a restart compares them with the new norm, which differs by about the share of seam faces, so the restarted run is not bit for bit the run it was. The driver history records the norm from format 4 on; a restart from format 1 to 3 says in the log that it cannot tell.) - :math:`U` = 10 m/s, :math:`\theta_s` = 10 K, :math:`k_s` = 0.1 m²/s²: fixed scales, not inputs. - :math:`H`: the depth of the layers, from the ground level to the top level. On a uniform mesh this is ``mesh.prob_hi`` z; with ``mesh.stretch_ratio`` or ``mesh.zlevels_file`` it is where the layers end, which can be well above ``mesh.prob_hi`` z (749 m against 440 m on Askervein). Before this was fixed, :math:`H` was ``mesh.prob_hi`` z on every mesh, so on a stretched mesh the norms were smaller by that ratio. - :math:`H` is a property of the mesh, not of the flow. The same flow on a taller mesh has larger norms, so :math:`\tau_R` is a tighter test there. The drop :math:`N` does not depend on :math:`H`. **Monitors.** Quantities of interest, such as the hilltop speed-up or :math:`u_*`, must also settle. Each monitor :math:`M_j` must stay within :math:`T_j = \tau_M\max(|M_j|, s_j)` over the last :math:`W` iterations, and its extrapolated limit (Aitken's method) must lie within the same distance: .. math:: \max_{n-W \le m \le n}|M_j^m - M_j^n| \le T_j, \qquad \left|\frac{\rho\,d_2}{1 - \rho}\right| \le T_j, \qquad \rho = \frac{d_2}{d_1},\ -1 < \rho < 1, where :math:`d_1, d_2` are the changes over the two halves of the window, :math:`\tau_M` is ``steady.monitor_rel_tol`` (1e-3) and :math:`W` is ``steady.monitor_window`` (50). The extrapolation matters: for a slowly converging monitor, a small change over the window alone can stop 10 % away from the limit. The outcome is ``Steady``, ``NotSteady`` (listing what has not converged) or ``Diverged`` (a NaN or infinity). Oscillations with a period up to :math:`4W` are never declared steady. .. _confidence-stop: **Confidence stop.** A run can sit at a fixed state where most of the field has converged and a few cells (often in a separated lee) have not. With ``steady.confidence`` :math:`= c > 0` the run is also steady when - the share of cells that meet the residual criterion cell by cell is at least :math:`c`. A cell meets it when :math:`|r_{q,i}|/s_q \le \min\bigl(\tau_R,\ 10^{-N} \max(R_q^\mathrm{ref}, \tau_R)\bigr)` for :math:`p, \theta, k` at the cell and :math:`u, v, w` on both of its faces: the threshold of the root-mean-square criterion, applied to each cell (if every cell meets it, so does the norm); - every monitor criterion above is met; and - with ``steady.confidence_height`` :math:`> 0`, every cell that does not meet it lies within that height above the ground. The stop is not tried during the reference hold. Its limits: the residual of the cells that do not meet the threshold is not bounded (1 % of cells at 100 times the threshold give a norm 10 times the threshold), and the monitors are global, so a limit cycle confined to a few cells that does not move them can be declared steady. The iteration log prints the share and the height of the highest unconverged cell (after ``r`` in the table), and a run stopped this way names the stop in its verdict. The SIMPLEC stage computes it; the stop does not act during the Newton stage. A stop by confidence is not recorded in the checkpoint (a restart from it iterates again). Stage switching, restart and reproducibility -------------------------------------------- - **Stage switching.** The driver moves to the next stage after ``steady.switch_window`` iterations below ``steady.switch_residual`` with the monitors settled, back when the residual grows by ``steady.switch_back_growth`` over a window, and back at once when the Newton stage asks (`Switching and fallback`_). - **Restart.** The driver records every iteration (stage, CFL, residuals, monitors, wall time, allocation units) in the checkpoint. A restarted run reproduces the uninterrupted run bit for bit on the same ranks and box layout, over flat ground and over terrain. The restart restores the valid points; the stage fills the ghost cells before the first step (the fill is a fixed point, :doc:`boundary_conditions`, so the extra fill changes nothing). The checkpoint also records :math:`H`. A checkpoint written before it did (driver history format 1 or 2) had its norms scaled with ``mesh.prob_hi`` z; on a stretched mesh the restart multiplies the stored reference and residuals by :math:`H` / ``prob_hi`` z and says so in the log. That run then continues in the new scale, and so is not bit for bit the run it was. - **Reproducibility.** Norms and monitors are reduced over all ranks, so every rank reaches the same verdict at the same iteration. They are sums in a fixed order, the same bits on any number of ranks for a given box layout, as are the Coriolis level mean and the mean the Poisson solve removes. The pressure solver's own reductions (any ``solver.poisson_backend``) are not, so a run on a different number of ranks agrees with the original to round-off, not bit for bit. ``steady.line_order = diagonal`` reproduces ``sequential`` bit for bit.