Cognite Transformations CLI

Transformations CLI

Use the Transformations command-line interface (Transformations CLI) to manage the lifecycle of your transformation jobs using the command line. With the Transformations CLI, you can process data from the CDF staging area (RAW) into the CDF data model. To learn more about how the Cognite Transformations CLI package works, see the documentation here

The Transformations CLI is based on Python and replaces the Jetfire CLI.

GitHub Action

The Transformations CLI provides a GitHub Action to deploy transformations. You'll find the documentation here.

We've also created a CI/CD template that uses GitHub Workflows. You'll find the documentation here.

Using Transformations CLI in Azure Pipelines

We publish transformations-cli docker images in dockerhub for every version released. The images tagged as cognite/transformations-cli:<version>-azure can be used in Azure Pipelines (See an example tag here). We suggest you check the example Azure Pipeline workflow configuration and the corresponding transformation manifest.

Migrating from Jetfire CLI

Transformations CLI replaces the Jetfire CLI. If you've already used the Jetfire CLI in a GitHub Action, we recommend migrating to the Transformations CLI GitHub Action. You'll find the migration guide here.

Contributing

We use poetry to manage dependencies and to administrate virtual environments. To develop Transformations CLI, follow these steps to set up your local environment:

1. Install poetry: (add --user if desirable)

   $ pip install poetry
   

2. Clone repository:

   $ git clone git@github.com:cognitedata/transformations-cli.git
   

3. Move into the newly created local repository:

   $ cd transformations-cli
   

4. Create a virtual environment and install dependencies:

   $ poetry install
   

5. All the code must pass black and isort style checks before it can be merged. We recommend installing pre-commit hooks to ensure this locally before you commit your code:

$ poetry run pre-commit install

6. To run tests:

$ poetry run pytest
$ poetry run pytest <test file path>
$ poetry run pytest <test file path>::<test function name>

7. To publish a new version, change the version in cognite/transformations_cli/__init__.py and pyproject.toml. Make sure to update the CHANGELOG.

This project adheres to the Contributor Covenant v2.0 as a code of conduct.