Important note:
Since change in my job I didn't had the chance to continue on this project. My main new project is here https://github.com/thomas-lehmann-private/hyperion-task-processor which is planned to be a improved task processing tool; step by step those one will be capable to have more functionality across all platforms (powershell, batch, bash, JShell, Groovy, Kotlin, Docker, ... features depend on platform). It's fully written in Java. Stay tuned.
Table Of Content:
Motivation
Quickstart
Example
Matrix
The Model
The Pipeline
Pipeline Stages
Tasks
The Shell
Environment Variables
Docker Container
Hooks
Include
Event Logging
Tool: spline-loc
How to contact?
Links
Features:
- Python support for 2.7.x, 3.4.x, 3.5.x, 3.6.x, PyPy and PyPy3
- automatic schema validation for yaml file
- matrix based pipeline with tags
- ordered and parallel pipelines (matrix)
- ordered and parallel tasks execution
- pipeline stages (named groups)
- shell script execution: inline and file
- environment variables merged across each level: matrix, pipeline, stage, and tasks
- support for model data (a dictionary of anything you need)
- cleanup hook
- filtered execution via tags
- supporting Jinja templating in Bash scripts (also nested inside model)
- support for Docker containers and Docker images
- support for the Packer tool
- execution time on each level: pipeline, stage, tasks and shell (event logging)
- documentation here and also at read the docs
- usable by Jenkinsfile as well as by a
.travis.yml(or other pipelines). - dry run and debug support
- support for Python scripts
- support for task variables
- support for conditional tasks
- enabled for code reuse: !include statement
- https://groups.google.com/forum/#!forum/spline-the-pipeline-tool
- https://github.com/Nachtfeuer/pipeline/issues
- https://github.com/keleshev/schema
- https://github.com/bats-core/bats-core
- https://pypi.python.org/pypi/PyHamcrest
- https://pypi.python.org/pypi/bandit
- https://www.contributor-covenant.org/
- http://ddt.readthedocs.io/en/latest/index.html
- http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html
- https://pymotw.com/2/multiprocessing/communication.html
- https://regex101.com/
- https://www.packer.io/intro/index.html