Enabling complex workflows and coupling research infrastructures
Scientific institutions often operate large-scale equipment that enables both scientists and industry researchers to perform complex and relevant experiments. During the lifespan of these experiments, an enormous amount of data is generated, so complex pipelines or workflows running in HPC facilities are needed in order to produce relevant output.
Therefore, general-purpose supercomputing should offer a flexible interface for a broad spectrum of scientific domain requirements. FirecREST fills this gap by providing an interface in which institutions can relay and build their workflows, or “FirecREST clients”, which interact with an HPC resource provider.
One example of such a coupling of research infrastructure using FirecREST has been co-developed by CSCS and the Paul Scherrer Institute (PSI) in support of the SELVEDAS project. Thanks to FirecREST, PSI could develop and integrate an online workflow using CSCS HPC resources to post-process the output of their scientific device in real time. The goal was to build a FirecREST client to support PSI online workflows in real-time.
Benefits of FirecREST:
For PSI: by building their own client, they manage and maintain the entire pipeline of the experiments; and by using FirecREST standard technology, they easily integrate HPC resource access to it. PSI no longer needs to export the identities of their users to CSCS, and they can keep complete control of their business logic.
For CSCS: by using standard technology together with a Python binding library, CSCS offers a simple service to its customers for creating and deploying specific clients. Moreover, CSCS does not have to maintain custom solutions for each of its customers and can instead provide one single interface. FirecREST brings more control and security for customer requests.