Continuous integration
Every pull request runs ci.yml and docs.yml.
.github/workflows/ci.yml
Job |
What it does |
|---|---|
CPU (GCC Release, Clang Release) |
Builds with Open MPI, runs |
Single precision |
Compiles only. The steady stop needs double precision. |
CUDA sm_90 |
Compiles only, for the H100. Hosted runners have no GPU. |
Windows MSVC Release |
Builds with MSVC, no MPI and FFTW from vcpkg, and runs |
Windows CUDA sm_90 |
Compiles only, with |
HIP gfx90a |
Compiles only, with ROCm 6.2.4 and rocFFT, for AMD GPUs (MI250X). |
SYCL |
Compiles only, with Intel |
|
Runs the drift check’s self-tests and the check itself against a clone of ERF, and lists upstream ERF changes to copied files (Code from other projects). |
Spelling and whitespace |
|
.github/workflows/debug-nightly.yml
GCC Debug with AMReX_TESTING runs the same tests. It takes about five hours, so it runs nightly on main or
by hand (workflow_dispatch), not on pull requests.
.github/workflows/docs.yml
Job |
What it does |
|---|---|
Build documentation |
|
Link check |
|
.github/workflows/static.yml
On every push to main (or by hand), the deploy job builds the documentation as above, with warnings as
errors, and publishes docs/_build/html to GitHub Pages. Only the built HTML is uploaded, not the repository.
Kestrel
The hosted runners have no GPU, so the CUDA, HIP and SYCL jobs above only compile. tools/kestrel/nightly_gpu.sh
runs the smoke tests on Kestrel H100s:
tools/kestrel/nightly_gpu.sh run # inside a GPU allocation (salloc --gpus=4 --ntasks=4 --mem=240G)
tools/kestrel/nightly_gpu.sh submit # the same as a batch job (GUST_ACCOUNT=<allocation>)
tools/kestrel/nightly_gpu.sh status # the result of the latest run
The job clones the repository at a branch or commit into its work directory, builds it with build_kestrel.sh
gpu and runs ctest -L smoke with one rank per GPU (GUST_MPIEXEC_PREFLAGS=--gpus-per-task=1;--gpu-bind=none:
every rank sees every GPU and AMReX picks one per rank; with each rank confined to its own GPU, Cray MPICH’s GPU IPC
fails between ranks). Before the tests it checks that every smoke run of gust carries
amrex.the_arena_init_size=0 amrex.init_snan=1 amrex.fpe_trap_invalid=1. kestrel_dryrun is left out: it
runs gust_refavg without srun, which Cray MPICH refuses inside an allocation, and it tests a CPU script
(CI runs it). The smoke tests include the 2-D hill
restart chain over terrain (rt_steady_hill2d_restart_*) and the 2-D hill with Coriolis run twice, compared at
zero tolerance (rt_coriolis_repeat_*). Each run writes logs/<date>_<sha>/ (info, build.log,
ctest.log, junit.xml, status: pass or fail: <step>); a clone or fetch that fails writes
logs/<date>_nofetch/ with fail: fetch. One run at a time holds the work directory (flock where it exists, as on
Kestrel, and --dependency=singleton for batch jobs).
Variable |
Default |
Effect |
|---|---|---|
|
none |
Kestrel allocation; required for |
|
|
Work directory (clone, build, logs, job scripts) |
|
|
Repository to test (also set on an existing clone) |
|
|
Branch or commit |
|
4 |
GPUs requested, 3 or 4 (the smoke tests use up to 3 ranks) |
|
none |
Mail address for a failed batch job |
|
30 |
Run logs kept, failed runs included |
|
|
Parallel build jobs |
The job is started by cron on a login node (scrontab is disabled on Kestrel), not from GitHub Actions: a
self-hosted runner on Kestrel would need the HPC center’s approval. The cron line starts a login shell, so that
sbatch and module are defined, and the job script is a login shell too:
0 1 * * * bash -lc 'GUST_ACCOUNT=<account> /path/to/gust-abl/tools/kestrel/nightly_gpu.sh submit'
Results are read on Kestrel (status) and are not reported to pull requests.
The regression tests labelled physics (ctest -L physics) and the performance tests will run nightly on
Kestrel CPU nodes and H100s once they exist.