Created by: riga
What is this Python project?
Added link
- law - Large-scale luigi workflows on distributed resources with remote storage and container support.
Description
The Luigi Analysis Workflow (law) package is a tool to build complex and large-scale task workflows as found is many scientific applications. It is build on top of luigi and adds abstractions for run locations, storage locations and software environments. Law strictly disentangles these building blocks and ensures they remain interchangeable and resource-opportunistic.
More info can be found at J. Phys.: Conf. Ser. 898 (2017) 072047. Law was and is used for several large-scale analysis in the context of high-energy physics research (CERN) and has the potential to serve many applications outside this community.
What's the difference between this Python project and similar ones?
Compared to other tools such as luigi itself, this key features of this project are:
- Rich CLI with auto-completion and interactive status and dependency inspection.
- Remote storage of results with automatic retries and local caching. Supports WebDAV, HTTP, Dropbox, SFTP, S3 and all protocols within the Worldwide LHC Computing Grid (srm, xrootd, rfio, dcap, gsiftp, ...)
- Automatic submission of jobs to batch systems from within tasks with error detection and resubmission. Supports HTCondor, LSF, gLite, ARC, Slurm.
- Environment sandboxing configurable on task level. Supports Docker, Singularity, Sub-Shells.
Anyone who agrees with this pull request could submit an Approve review to it or vote for it by adding a