1. Runtime flexibility and native MPI performance
HPC systems often feature dedicated hardware capable of superior performance, which can only be achieved through specific native software. If the container was not built with such native software, as it often happens with off-the-shelf Docker-based images, then it cannot achieve maximum performance on HPC systems. Sarus solves this issue by substituting software in the container with its host counterpart that is optimised for the relevant underlying hardware.
One of the most common examples is the use of a message-passing interface (MPI) as a programming interface for high-speed interconnects. In this case, Sarus transparently propagates the PMI-2 interface from the HPC host into containers, so that containers built with MPICH-like MPI implementations can be immediately used on HPC systems. Furthermore, if ABI-compatibility is satisfied, host MPI libraries can replace those in the container, unlocking native message-passing performance. Such substitution is activated by a simple --mpi command line option. OpenMPI-based containers can also be used if they support PMI-2 or, alternatively, through the SSH Hook (see an example here).
From a technical standpoint, MPI library substitution is performed by the Sarus MPI Hook, an external plugin interfacing with the container runtime through OCI industry standard interfaces supported by the Sarus design. Hooks are independent programs and can be easily developed to inject or replace other software inside containers.
The Sarus approach, with a Hook performing library replacement, enables:
- The complex task of describing the software injection to be delegated from the user to the system administrator.
- A more robust and controlled mechanism for performing file replacement (or injection).
- A more informative output of Hook execution towards the user.
- Enhancement with self-sufficient and portable container images.
Example of the OSU All-to-all benchmark for the native MPI and MPICH 3.1.4 container: These results were obtained on “Piz Daint”, CSCS’ Cray flagship, using an Ubuntu-based container (i.e. no Cray PE or Cray MPI in the image). The MPI in the container is replaced at runtime by the MPICH MPI Hook used by Sarus.
Practical information about MPI substitution and benchmark results can be found here.
More information about the Hook interface and its configuration can be found here.
2. Simplified user experience
Using a container engine to initiate containers on an HPC machine could be a complex task, especially for HPC ecosystems that have many specific components such as interconnect or parallel file systems. Sarus provides a set of features to improve the the user experience deploying containers on HPC:
- Sarus commands offer a command-line syntax similar to that of Docker, which is the most common container solution outside HPC. Sarus thus provides a familiar user experience and a smoother learning curve for Docker users. Some examples of commands include:
sarus images,sarus rmi,sarus pullorsarus run. - Users’ images are managed by Sarus on the system at-a-glance, and each user has his or her own individual repository for a personalised experience. Images can be pulled from different sources including Docker Hub or from an archived file.
- Environment variables defined in the host process environment will be automatically transferred into the container, unless an environment variable is defined inside the container image.
- In the context of parallel computing, Secure Shell Protocol (SSH) is often used as an underlying vehicle to connect processes running on different compute nodes. To enable connection among processes running inside containers on different hosts, Sarus provides an SSH capability inside the container with a simple command line:
--ssh.
3. Hardware vendor support for containers
Sarus Hooks enable specific and outside-of-Sarus programs to be executed during the container instantiation. These Hooks are used for several actions, such as software substitution or integration with HPC tools like SSH.
Another important use case for Hooks is to introduce specific hardware capabilities inside the container. In this case, Hooks can be supplied directly by vendors to support their proprietary technologies. One such example is the NVIDIA Container Runtime Hook, which provides access to NVIDIA GPUs.
Using vendor Hooks has the many advantages. For example, support for hardware devices and drivers is implemented and maintained directly by the hardware vendor, granting more features and robustness for the user. Furthermore, there is no need to wait for a new Sarus release in order to receive updates and fixes regarding dedicated devices.