Installation

Requirements

  • CMake 3.25 or newer and a C++20 compiler (GCC, Clang, AppleClang, MSVC; nvcc for CUDA, ROCm clang++ for HIP, icpx for SYCL).

  • MPI (Open MPI, MPICH or Cray MPICH), unless GUST_MPI=OFF.

  • An FFT library for AMReX’s FFT module: FFTW on the CPU, cuFFT (CUDA), rocFFT (HIP) or oneMKL (SYCL).

  • Python 3, for the test scripts.

AMReX and GoogleTest are git submodules. AMReX is pinned to a fixed commit.

Quick start

git clone --recursive https://github.com/erf-model/gust-abl.git
cd gust-abl
cmake -S . -B build-rel -DCMAKE_BUILD_TYPE=Release
cmake --build build-rel -j 8
ctest --test-dir build-rel --output-on-failure

The executable is build-rel/gust. On a case-insensitive file system, name the build directory something other than build.

CMake options

An option outside its allowed values stops the configuration.

Option

Default

Effect

GUST_MPI

ON

Build with MPI.

GUST_PRECISION

DOUBLE

DOUBLE or SINGLE. The steady stop needs double precision; single precision is kept compiling only.

GUST_GPU_BACKEND

NONE

NONE, CUDA, HIP or SYCL. For an H100 add -DCMAKE_CUDA_ARCHITECTURES=90; HIP needs -DAMReX_AMD_ARCH (for example gfx90a) and ROCm’s clang++ as the C++ compiler; SYCL needs icpx.

GUST_FFT

ON

AMReX FFT. Required: the Poisson solver uses its FFT preconditioner. Configuring with OFF stops with an error, and an external AMReX must have been built with FFT (find_package asks for it).

GUST_HYPRE

OFF

AMReX with hypre, for the optional Newton–Krylov preconditioner (in development).

GUST_TESTING

OFF

AMReX_TESTING and assertions on, as in GPU CI. With assertions (or in a Debug build) the inner operators also re-check inputs that the solver has already checked once per iteration or per setup (the pseudo-time step at the faces and in the scalar sweeps, the wall distance in every closure evaluation). A Release build skips these repeated checks and their all-reduces.

GUST_ENABLE_TESTS

ON

Build the unit tests and register all tests with CTest.

GUST_MPIEXEC_PREFLAGS

empty

Extra mpiexec flags for the tests. CI passes --oversubscribe to Open MPI.

GUST_USE_INTERNAL_AMREX

ON

Use the AMReX submodule. OFF uses an installed AMReX found through AMReX_ROOT.

The options were called ABLSTEADY_* before the rename to GUST (October 2026). The old names still work: each sets its GUST_* option and prints a warning.

GUST runs MPI (and GPUs), not OpenMP. The GUST_OMP option was removed (October 2026): -DGUST_OMP=ON (or ABLSTEADY_OMP=ON) stops the configuration and drops the option from the build directory, so the next configure without it goes through; OFF is ignored. An external AMReX built with OpenMP is refused.

Windows

On Windows the solver builds with MSVC and no MPI, and FFTW comes from vcpkg. It needs Visual Studio 2022 with the “Desktop development with C++” workload (including “C++ CMake tools for Windows”, which provides CMake and Ninja), Git and Python 3. The vcpkg that ships with Visual Studio supports only manifest mode, so use a standalone clone. From a “Developer PowerShell for VS”:

git clone https://github.com/microsoft/vcpkg $HOME/vcpkg
& $HOME/vcpkg/bootstrap-vcpkg.bat
$env:VCPKG_ROOT = "$HOME/vcpkg"
& $env:VCPKG_ROOT/vcpkg install fftw3:x64-windows

git clone --recursive https://github.com/erf-model/gust-abl.git
cd gust-abl
cmake -S . -B build-rel -G Ninja -DCMAKE_BUILD_TYPE=Release `
  -DCMAKE_TOOLCHAIN_FILE="$env:VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" `
  -DVCPKG_TARGET_TRIPLET=x64-windows -DGUST_MPI=OFF
cmake --build build-rel
$env:PATH = "$env:VCPKG_ROOT/installed/x64-windows/bin;$env:PATH"
ctest --test-dir build-rel -L "unit|smoke" --output-on-failure

The FFTW DLLs are found through PATH. Without MPI every test runs on one rank, and MSVC does not trap floating-point exceptions (Testing). The Kestrel shell-script tests are not registered on Windows.

A CUDA build on Windows uses cl with nvcc. With CUDA 12.6 and C++20 it needs Windows SDK 10.0.22621.0: install that SDK, open the prompt with -winsdk=10.0.22621.0 (for example VsDevCmd.bat -arch=x64 -winsdk=10.0.22621.0) and configure with -DCMAKE_SYSTEM_VERSION=10.0.22621.0, as the CI job does.

Kestrel

tools/kestrel/build_kestrel.sh cpu (on a CPU login node) or tools/kestrel/build_kestrel.sh gpu (on a GPU login node) builds into build-kestrel-<cpu|gpu>. The modules come from tools/kestrel/kestrel_env.sh, which follows ERF’s Kestrel profiles at the pinned commit (after Kestrel’s RHEL9 upgrade, erf-model/ERF#4292). Kestrel’s module names change with system updates, so check them with module avail first. A CUDA build needs memory: on a shared node, build inside an allocation with --mem=240G or --exclusive.

The scaling runs on Performance and scaling were made on Kestrel by hand. ERF reference solutions describes the ERF reference runs, Continuous integration the nightly GPU job (tools/kestrel/nightly_gpu.sh).

Documentation

python3 -m pip install -r docs/requirements.txt
make -C docs html

Warnings are errors (-W in docs/Makefile), so the local build and the CI build are the same. The pages are written to docs/_build/html.