cldr-runner - Ansible Execution Environments for Cloudera Data Platform (CDP)

cldr-runner is set of Ansible Execution Environments for running Cloudera playbooks, examples, and general automation for Cloudera Data Platform (CDP) Public Cloud, Private Cloud, and Data Services. These images are appropriate for use with ansible-navigator and AWX/Red Hat Ansible Automation Platform (AAP).

Specifically, the project consists of execution-environment.yml configuration files and other supporting assets that power ansible-builder. The configurations encapsulate the necessary Ansible collections and roles, Python libraries, and system applications to work with Cloudera's products and cloud providers. Moreover, the resulting images have the needed tooling for managing infrastructure if so requested.

cldr-runner builds several profiles:

Tag	Description
base	Core Ansible, collections, and dependencies including Terraform
aws	base plus AWS-specific collections and dependencies, including the aws CLI
azure	base plus Azure-specific collections and dependencies, including the az CLI
gcp	base plus GCP-specific collections and dependencies, including the gcloud CLI
full	All of the above, plus additional CLI tools for in-container usage, e.g. git, vim, nano, tree, kubectl

Each image is tagged cloudera-labs/cldr-runner-<profile>:<version>.

Quickstart

cldr-runner is designed to run with ansible-navigator and other Execution Environment-based tools. You might want to install ansible-navigator before delving deeper.

1. Installing and using images
2. Building local images
3. Customizing or extending images
4. Making a local development environment

Roadmap

If you want to see what we are working on or have pending, check out:

• the Milestones and active issues to see our current activity,
• the issue backlog to see what work is pending or under consideration, and
• the Ideas discussion to see what we are considering..

Are we missing something? Let us know by creating a new issue or posting a new idea!

Contributing

For more information on how to get involved with the cldr-runner project, head over to CONTRIBUTING.md.

Installing and Using

You can run Ansible within cldr-runner Execution Environments a couple of different ways. Here are the most common:

ansible-navigator

Using a cldr-runner image in the ansible-navigator application as the designated Execution Environment is straightforward. Update your ansible-navigator.yml configuration file to enable the image:

ansible-navigator:
  execution-environment:
    container-engine: docker
    enabled: True
    image: ghcr.io/cloudera-labs/cldr-runner-aws:latest
    pull:
      policy: missing

Once defined, you can run your Ansible activities within the resulting cldr-runner container, e.g. ansible-navigator run your_playbook.yml. (You can specify the image via the ansible-navigator CLI; set --eei or --execution-environment-image.)

Note

If you want to "drop into" the container directly, i.e. run a shell within the container, run ansible-navigator exec -- /bin/bash and all the mounts, environment variables, etc. are handled for you!! Now from the shell, you can still run ansible-playbook and all other Ansible applications.

AWX/AAP

You can specify a cldr-runner image as an Execution Environment.

Once defined, the EE can be used by Job Templates, Container Groups, etc.

docker run

Warning

This mode of operation is not suggested. If you need direct container access, use ansible-navigator exec -- /bin/bash as suggested in the section above.

You can run the container directly in docker (or podman):

docker run -it ghcr.io/cloudera-labs/cldr-runner-aws:latest /bin/bash

Take care to assemble and mount the needed directories other supporting assets; the image is based on ansible-runner (as are all Execution Environments) and runs as such.

Building

If you need to construct a local image, first set up a Python virtual environment with the latest ansible-core and ansible-builder:

python -m venv ~/location/of/venv; source ~/location/of/venv/bin/activate; pip install ansible-builder

Note

If you have already set up ansible-navigator, then you have ansible-builder!

Then change into the directory of the cldr-runner variation you need to build and run:

ansible-builder build --prune-images --squash all --build-arg BUILD_VER=<your version> --build-arg BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ") --tag <your tag>

You may want to update the variation's execution-environment.yml configuration file to use a different base image, say a local image, or build the base image before constructing CSP or full image. You can make this change in the following section of the configuration file:

images:
  base_image:
    name: <your upstream base image>

The resulting image will now be loaded into your local image cache.

Customizing

A common approach to using cldr-runner is to use it as a base and add additional resources -- like other Ansible collections -- for use with your playbooks, and ansible-builder handles for this activity.

First, define your custom Execution Environment definition. In the example below, we are using the latest AWS image and adding two private Ansible collections, a public Ansible role, and a public Python library that we need to run our project's playbooks.

version: 3

images:
  base_image:
    name: ghcr.io/cloudera-labs/cldr-runner-aws:latest

dependencies:
  galaxy:
    collections:
      - name: https://internal.example.com/my-team/example.stuff
        type: git
        version: devel
      - name: https://internal.example.com/another-team/example.things
        type: git
        version: feature/cool-things
    roles:
      - ansible.scm

additional_build_steps:
  append_final:
    RUN pip install jsonschema

Construct your custom image with ansible-builder (in the command below, we are using ansible-navigator to call ansible-builder.)

ansible-navigator builder build -t internal.example.com/my-team/my-runner:latest --squash new --prune-images -v 3

And now you can reference your (local) custom image in your project's ansible-navigator.yml file:

ansible-navigator:
  execution-environment:
    container-engine: docker
    enabled: True
    image: internal.example.com/my-team/my-runner:latest
    pull:
      policy: missing

Local Development

The cldr-runner project can also be used to bootstrap a local development environment on the native host environment (as opposed to an Execution Environment image). This option is more involved, but can avoid issues with Docker, such as mount latencies and SSH agent forwarding, and improve overall Ansible collection development.

The local_development.yml playbook sets up a cldr-runner-like workspace for OSX and Ubuntu. The playbook will clone the Cloudera collections and cdpy for local work, install the external Ansible dependencies, update the Python venv, and create a convenient setup script for future work.

Note

The cloned Cloudera collections and cdpy project use the main branches by default. Manipulating the branches is outside the scope of the local_development.yml playbook.

Important

If you are using an M1 or M2 Macbook, enable the Rosetta within your terminal to ensure compatibility and prevent library incompatibilities between architectures. Go to Finder > Applications and then Terminal > <right-click on your Terminal> > Get Info > Enable "Open using Rosetta" (or whatever application you want to use).

Development in this manner starts with sourcing the setup script, activating the virtual environment, and then switching to and running cldr-runner-based applications, such as ansible-navigator-based cloudera-deploy definitions, using the development environment's collections and tools.

See Advanced Usage: Execution Modes for details on how you can use a host-based, local Ansible environment for execution and development.

Local setup instructions

Follow these steps to set up a local environment:

1. Create a new virtual environment (using your favorite venv app; example uses virtualenvwrapper):

   mkvirtualenv <your development directory>

2. Set up the bootstrap requirements:

   export ANSIBLE_COLLECTIONS_PATH=<your target development directory>
   pip install ansible-core==2.12.*
   ansible-galaxy collection install community.general

3. Make sure you are able to connect to public GitHub via SSH and then construct the development environment:

   ansible-playbook local_development.yml

   [!WARNING] For Ubuntu deployments, you will need to add the --ask-become-pass flag to the above command.

4. Source the setup-ansible-env.sh file to use this development environment.

   source <your development directory>/setup-ansible-env.sh

License and Copyright

Copyright 2024, Cloudera, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.