Simplification of CI system

[jhale]

Describe new/missing feature

The CI has become a bit difficult to maintain and this is likely to increase as we move towards GPU support. I think it needs a cleanup/simplification.

Spack CI:

• The Spack-based CI is harmonised into a single runner file, parameterised over base platform and dependency sets specified in multiple static Spack .yaml files, rather than being written as spack add commands.
• Each Spack environment file would have a specific written reason for its existence in terms of covering a good set of build/testing combinations (this would be the first step, infact).
• FEniCS components would still be built and tested using regular commands, not Spack.

For native CI:

• Native macOS (Homebrew plus cached PETSc), Ubuntu (system packages, no PETSc) and Windows (vcpkg, no PETSc) would remain.

For Docker:

• Docker-based CI is removed.
• Hence, the gigantic images with all combinations of PETSc are removed.
• Dockerfiles and images are simplified with the main goal to provide the best end-user experience for dolfinx:nightly and dolfinx:stable. This means using as many pre-built Ubuntu and pypi packages as possible, possibly even PETSc (we will see).
• I would propose keeping the split between dev-env and dolfinx:nightly/stable - it's been useful for debugging with inexperienced users and reduces download times.

Some disadvantages:

• Nearly full reliance on Spack. We do now have the ability to hot fix Spack recipes at: https://github.com/FEniCS/spack-fenics without having to maintain a fork of upstream Spack packages.
• Downstream users of our testing images will be annoyed - however, these were never promoted as a public facing resource.
• In terms of replacing the Docker images, there is the potential to educate users on using Spack to generate Dockerfiles. https://spack.readthedocs.io/en/latest/containers.html#generating-recipes-for-docker-and-singularity

Suggested user interface

[jhale]

We could also try Spack's beta Windows support: https://spack.readthedocs.io/en/latest/windows.html

That would remove some of my hacky vcpkg work, where no one is really working in the HPC space to maintain vcpkg recipes.

[jorgensd]

Nearly full reliance on Spack. We do now have the ability to hot fix Spack recipes at: https://github.com/FEniCS/spack-fenics without having to maintain a fork of upstream Spack packages.

Spack is quite annoying in the sense that everyone that really wants to work with it has to have a package repo somewhere are the turnaround for them merging new releases of packages is incredibly slow (which means a cascading set of repos has to be added to everyone's test systems if one rely on multiple extensions of the DOLFINx code).

• Downstream users of our testing images will be annoyed - however, these were never promoted as a public facing resource.

Another downside is that everyone has to set up their own CI with spack caching, as using spack without cache is very bad for the environment.

I am worried that removing these images will decrease the amount of proper extensions to DOLFINx. If we complain that our docker images is hard to maintain, how will this make it easier for anyone else?
Additionally, it's unclear to me what is hard to maintain with the docker CI at the moment. The Dockerfile rarely change, as seen by the history:
https://github.com/FEniCS/dolfinx/commits/main/docker/Dockerfile.end-user
https://github.com/FEniCS/dolfinx/commits/main/docker/Dockerfile.test-env
were most of the changes here are version bumps.

We could of course remove at least two of the PETSc builds in the current docker images, as they are almost never used by any user, i.e.:

• PETSC_ARCH=linux-gnu-real32-32
• PETSC_ARCH=linux-gnu-complex64-32

as they are not exposed through source dolfinx-real-mode or source dolfinx-complex-mode.

I'm not against installing less packages by hand in the docker images, relying on apt rather than hand-installing the latest of whatever dependency.

[jhale]

There is no plan to require downstream users or developers to run their CI or dev environment with Spack.

DOLFINx main can be built on a raw Ubuntu runner, with PETSc, in the the same amount of time as it takes to pull a test image and build DOLFINx. This should simply be packaged up into an action like every other project.