The container engine for combining container portability with native HPC performance
Sarus is a container engine for high-performance computing (HPC) systems that provides a user-friendly way to instantiate feature-rich containers from Docker images. It is designed to blend the portability of containers with the unique requirements of HPC installations, such as: native performance from dedicated hardware, improved security due to the multi-tenant nature of the systems, support for network parallel filesystems and diskless computing nodes, and compatibility with a workload manager or job scheduler.
With flexibility, extensibility and community efforts at its core, Sarus relies on industry standards and open-source software. The adoption of the OCI specifications allows Sarus to implement a modular architecture and to extend the capabilities of the container runtime through external plugin programs, called “OCI hooks”. Hooks can customize containers to enable specific high-performance features, and they can be developed independently by third parties to introduce new technologies or improve support for existing ones.
- HPC native performance for common Docker containers
By injecting system-specific resources (e.g. device files, libraries) during container creation, Sarus provides the missing link between non-HPC specific software stack of a portable container and HPC specific hardware available at runtime. The container images stay portable but the container achieve native performance.
- Integrates with HPC infrastructures and software
Sarus has been developed by a team including experts of large and complex HPC systems. Sarus provides a simple integration into HPC ecosystem by storing images on a shared parallel filesystem, enabling MPI (PMI2 interface) and exposes GPU accelerators in a container and by securing container usage in a multi-user context.
- Customizes containers at runtime with standard plugins
Sarus adopts the OCI open industry standards, allowing for modularity and extensibility of its installation and configuration. Sarus is designed to use and configure third-party plugins, called “OCI Hooks”, which customize the container at runtime. The core runtime of Sarus is a replaceable upstream low-level runtime (runc): better maintainability, flexibility, and choice.
- Provides a Docker-like command-line interface
Sarus provides a familiar, approachable user experience with a command-line closely resembles Docker for a smoother learning curve. Individual user image repositories are being managed a la Docker/Podman with the capability for the user to list or remove his own images on the system.
Use cases
- Runtime flexibility and native MPI performance
HPC systems often feature dedicated hardware capable of superior performance, which can only be achieved through specific native software. If the container was not built with such native software, as it often happens with off-the-shelf Docker-based images, then it cannot achieve maximum performance on HPC systems. Sarus solves this issue by substituting software in the container with its host counterpart that is optimised for the relevant underlying hardware.
One of the most common examples is the use of a message-passing interface (MPI) as a programming interface for high-speed interconnects. In this case, Sarus transparently propagates the PMI-2 interface from the HPC host into containers, so that containers built with MPICH-like MPI implementations can be immediately used on HPC systems. Furthermore, if ABI-compatibility is satisfied, host MPI libraries can replace those in the container, unlocking native message-passing performance. Such substitution is activated by a simple --mpi command line option. OpenMPI-based containers can also be used if they support PMI-2 or, alternatively, through the SSH Hook (see an example here).
From a technical standpoint, MPI library substitution is performed by the Sarus MPI Hook, an external plugin interfacing with the container runtime through OCI industry standard interfaces supported by the Sarus design. Hooks are independent programs and can be easily developed to inject or replace other software inside containers.
The Sarus approach, with a Hook performing library replacement, enables:
- The complex task of describing the software injection to be delegated from the user to the system administrator.
- A more robust and controlled mechanism for performing file replacement (or injection).
- A more informative output of Hook execution towards the user.
- Enhancement with self-sufficient and portable container images.
Practical information about MPI substitution and benchmark results can be found here.
More information about the Hook interface and its configuration can be found here.
- Simplified user experience
Using a container engine to initiate containers on an HPC machine could be a complex task, especially for HPC ecosystems that have many specific components such as interconnect or parallel file systems. Sarus provides a set of features to improve the the user experience deploying containers on HPC:
- Sarus commands offer a command-line syntax similar to that of Docker, which is the most common container solution outside HPC. Sarus thus provides a familiar user experience and a smoother learning curve for Docker users. Some examples of commands include: sarus images, sarus rmi, sarus pull or sarus run.
- Users’ images are managed by Sarus on the system at-a-glance, and each user has his or her own individual repository for a personalised experience. Images can be pulled from different sources including Docker Hub or from an archived file.
- Environment variables defined in the host process environment will be automatically transferred into the container, unless an environment variable is defined inside the container image.
- In the context of parallel computing, Secure Shell Protocol (SSH) is often used as an underlying vehicle to connect processes running on different compute nodes. To enable connection among processes running inside containers on different hosts, Sarus provides an SSH capability inside the container with a simple command line: --ssh.
- Hardware vendor support for containers
Sarus Hooks enable specific and outside-of-Sarus programs to be executed during the container instantiation. These Hooks are used for several actions, such as software substitution or integration with HPC tools like SSH.
Another important use case for Hooks is to introduce specific hardware capabilities inside the container. In this case, Hooks can be supplied directly by vendors to support their proprietary technologies. One such example is the NVIDIA Container Runtime Hook, which provides access to NVIDIA GPUs.
Using vendor Hooks has the many advantages. For example, support for hardware devices and drivers is implemented and maintained directly by the hardware vendor, granting more features and robustness for the user. Furthermore, there is no need to wait for a new Sarus release in order to receive updates and fixes regarding dedicated devices.
Documentation
Please see our documentation at Read The Docs for details on installing and using Sarus.
Webinars
Publications
- Sarus: Highly Scalable Docker Containers for HPC Systems. Read more here.
Interviews and news
- SELVEDAS: Innovative approach to promote efficient data workflow between institutions. Read more here.
- Meet Sarus and FirecREST: CSCS launches new software product website in support of EuroCC Project. Read more here.
- Webinar - Sarus: Towards HPC Performance for Portable Containers. Read more here.
- Sarus achieves container portability with near-native performance: “This is unique”. Read more here.
- Connected and strong: the European HPC ecosystem. Read more here.