Eclipse Otterdog

Introduction

Otterdog is a tool to manage GitHub organizations at scale using a configuration as code approach. It is actively developed by the Eclipse Foundation and used to manage its numerous projects hosted on GitHub.

Quickstart

To install and use the cli part of otterdog you have to install the following:

• git (mandatory): install using apt install git
• otterdog (mandatory): install using pipx install otterdog
• bitwarden cli tool (optional): install using snap install bw
• pass cli tool (optional): install using apt install pass

Otterdog Presentation @ Open Source Summit 2023

Default Configuration used @ Eclipse Foundation

Documentation

The documentation is available at otterdog.readthedocs.io.

Build instructions

System requirements:

• python3.11+ (mandatory): e.g. install using apt install python3 or use pyenv install 3.12
• git (mandatory): install using apt install git
• poetry (mandatory): install using pipx install poetry
• dynamic versioning plugin (mandatory): install using pipx inject poetry "poetry-dynamic-versioning[plugin]"
• bitwarden cli tool (optional): install using snap install bw
• pass cli tool (optional): install using apt install pass

Building Steps

• Create a virtual python environment and install necessary python dependencies using poetry:

$ make init

Running make init will also install poetry and the dynamic versioning plugin if it is not installed yet.

• Testing build

$ ./otterdog.sh -h

Quick Setup

To start using the cli part of otterdog right away on a specific organization you have to set up the following:

• define a default configuration to use, or use the following default config right away
• create a otterdog.json file that contains a list of GitHub organizations to manager and their respective credentials
• start managing your organizations using the cli

Default configuration

The example default config has all supported features enabled and can be used right away. However, it is advised to use a released tag instead of main to avoid incompatibilities.

Otterdog configuration

Create a otterdog.json file with the following content (replace bracketed values according to your setup):

{
  "defaults": {
    "jsonnet": {
      "base_template": "https://github.com/eclipse-csi/otterdog#examples/template/otterdog-defaults.libsonnet@main",
      "config_dir": "orgs"
    }
  },
  "organizations": [
    {
      "name": "<project-name>",
      "github_id": "<github-id>",
      "credentials": {
        "provider": "plain",
        "api_token": "<GitHub PAT>",
        "username": "<Username>",
        "password": "<Password>",
        "2fa_seed": "<2FA TOTP seed>"
      }
    }
  ]
}

The name of the configuration file can be freely chosen (can be overridden with the -c flag). However, when named otterdog.json, the cli tool will automatically detect and use that file if it is in the current working directory.

Important

In this example the plain provider is being used to access credentials to avoid setting up a real credential provider (see below) for a quick setup. However, the plain provider should NOT be used for anything else to avoid leakage of data in case the otterdog.json file is shared with other users.

Credentials

Otterdog needs certain credentials to access information from an organization and its repositories on GitHub:

• username / password / 2FA seed
• API token

The login / username / 2FA seed are required to access the web interface of GitHub in order to retrieve certain settings that are not accessible via its rest / graphql API.

The GitHub api token needs to have the following scopes enabled:

• repo
• workflow
• admin:org
• admin:org_hook
• delete_repo

The credentials can be stored in different providers (bitwarden, pass).

Bitwarden

When using bitwarden to store the credentials, you need to enter a valid item id as additional credential data:

{
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "bitwarden",
        "item_id" : "<bitwarden item id>"
      }
    }
  ]
}

The item stored in bitwarden needs to contain the following information (a sample json output of such an item):

{
  "object": "item",
  "id": "<bitwarden item id>",
  "name": "<item name>",
  "fields": [
    {
      "name": "api_token_admin",
      "value": "<github API token>"
    }
  ],
  "login": {
    "username": "<github username>",
    "password": "<github password>",
    "totp": "<github TOTP text code>"
  }
}

Mandatory items:

• Field with name "api_token_admin" and as value the GitHub token to access the organization
• login.username of a user that can access the organization with enabled 2FA
• login.password the password of that user
• login.totp the TOTP text code

Pass

When using pass to store the credentials, you need to enter fully qualified pass names to access the various required credential data:

{
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "pass",
        "api_token": "<path/to/api_token>",
        "username": "<path/to/username>",
        "password": "<path/to/password>",
        "2fa_seed": "<path/to/2fa_seed>"
      }
    }
  ]
}

In case your password storage dir is not located at the default location, you can configurate that in the defaults:

{
  "defaults": {
    "pass": {
      "password_store_dir": "path/to/storage/dir"
    }
  }
}

As the password_store_dir might be different on different machines, you can also customize that in a separate .otterdog-defaults.json file:

{
  "pass": {
    "password_store_dir": "path/to/storage/dir"
  }
}

Typical Workflow

In general, all operations act on the local configuration stored in <cwd>/<config-dir>/<organization>/<organization>.jsonnet.

A typical workflow to handle changes to an organization are as follows:

1. (first time) run an initial import of the organization
2. (first time) run an apply operation to create all resources already inherited from the default config (e.g. config repo)
3. (regular) fetch the latest config from the config repo using fetch-config
4. (optional) make any local changes to the configuration
5. (optional) run the validate operation to see if the configuration is syntactically and semantically correct
6. (optional) run the plan operation to see which changes would be applied taking the current live configuration into account
7. (regular) run the apply operation to actually apply the changes (also runs validate and plan, so steps 6 & 7 are redundant)
8. (regular) push the local configuration to the config repo using the push-config operation

Note

It is not mandatory to store the configuration in the remote config repository (<org>/.otterdog by default). It could be stored anywhere else, however the operations fetch-config and push-config expect this repository to exist to function properly.