The REST interface for programming access to HPC resources
FirecREST is a RESTful services gateway and interface for managing HPC resources. Designed and developed as a high-performance and reusable framework of micro-services, it provides a coupling layer between the identity and access management of the cloud and web world and the essential POSIX interfaces used in HPC. FirecREST therefore serves as a secure, web-accessing interface for accessing HPC services such as HPC compute capability or high-speed storage.
Together with the use of standard technology, its modular design enables simple integration into existing HPC infrastructure. FirecREST allows communities that require HPC to build value-added workflows or services to maximise the productivity of their members. Developing such services is made easier with FirecREST because it uses standard technologies such as REST API. With the addition of pyFirecREST, any HPC user can develop a simple Python program that connects through the web to his/her HPC providers and enable the submission of jobs or transfer of data in and out of the HPC centres.
- Establishes a web interface for HPC systems
FirecREST is platform-agnostic and interoperable with different web technologies through HTTP protocols. This eliminates the need to use complex SSH keys setup, a VPN connection, or to establish a direct connection to the HPC centre. Thanks to FirecREST, your program just needs a simple network connection to be able to manage your HPC resources.
- Built modularly to integrate HPC ecosystems
FirecREST is a collection of loosely coupled services. Each service focuses on different capabilities, such as an identity translation from standard web identity protocol to POSIX user names, or manipulation of HPC resources to match the HPC infrastructure. With such flexible design, FirecREST can adapt to many technologies, and its interface can be extended to specific services provided by an HPC centre.
- Secures and monitors HPC resource access
FirecREST uses “OIDC connect” as an authentication protocol to ensure a valid authentication of the program accessing the infrastructure. For authorisation, FirecREST uses Open Policy Agent (OPA), which allows fine-grained rules for users, clusters, and micro-services when accessing resources. FirecREST provides an entry point for every API requests that enables an extra layer of monitoring and validation of requests.
- Facilitates programmatic HPC resource access
FirecREST simplifies the technical complexity of programming access to HPC resources. From a programmatic point of view, users can build or consolidate existing HPC workflows or HPC services by using a common REST API technology or by using the Python bindings of FirecREST.
Use cases
Enabling complex workflows and coupling research infrastructures
Scientific institutions often operate large-scale equipment that enables both scientists and industry researchers to perform complex and relevant experiments. During the lifespan of these experiments, an enormous amount of data is generated, so complex pipelines or workflows running in HPC facilities are needed in order to produce relevant output.
Therefore, general-purpose supercomputing should offer a flexible interface for a broad spectrum of scientific domain requirements. FirecREST fills this gap by providing an interface in which institutions can relay and build their workflows, or “FirecREST clients”, which interact with an HPC resource provider.
One example of such a coupling of research infrastructure using FirecREST has been co-developed by CSCS and the Paul Scherrer Institute (PSI) in support of the SELVEDAS project. Thanks to FirecREST, PSI could develop and integrate an online workflow using CSCS HPC resources to post-process the output of their scientific device in real time. The goal was to build a FirecREST client to support PSI online workflows in real-time.
Benefits of FirecREST:
- For PSI: by building their own client, they manage and maintain the entire pipeline of the experiments; and by using FirecREST standard technology, they easily integrate HPC resource access to it. PSI no longer needs to export the identities of their users to CSCS, and they can keep complete control of their business logic.
- For CSCS: by using standard technology together with a Python binding library, CSCS offers a simple service to its customers for creating and deploying specific clients. Moreover, CSCS does not have to maintain custom solutions for each of its customers and can instead provide one single interface. FirecREST brings more control and security for customer requests.
Documentation
Please see our documentation at Read The Docs for details on installing and using FirecREST.
Webinars
Publications
- FirecREST: a RESTful API to HPC systems. Read more here.
Interviews and news
- Meet Sarus and FirecREST: CSCS launches new software product website in support of EuroCC Project. Read more here.
- “FirecREST makes it easy to integrate HPC into web-based applications”. Read more here.
- FirecREST Hackathon: API for HPC. Read more here.