The Pipeline Builder is a collection of tools related to GitHub Actions and other GitHub-related concerns for Paketo-style buildpacks. This collection includes GitHub actions, a CLI that configures pipelines for repositories, and a collection of descriptors for existing repositories.
- Pipeline Builder
- Octo Pipelines
- Actions
- Adoptium Dependency
- Alibaba Dragonwell Dependency
- Amazon Corretto Dependency
- AppDynamics Dependency
- Aqua Security Dependency
- Azul Zulu Dependency
- Bellsoft Liberica Dependency
- CA APM Dependency
- CF Java Index Dependency
- Clojure Tools Dependency
- Compute Artifact Dependencies
- Draft Release
- Foojay Dependency
- GCS Dependency
- GitHub Release Dependency
- Google Stackdriver Profiler Dependency
- GraalVM Dependency
- Gradle Dependency
- IBM Semeru Dependency
- JProfiler Dependency
- JRebel Dependency
- Leiningen Dependency
- Liberty Dependency
- Maven Dependency
- New Relic Dependency
- NPM Dependency
- OverOps Dependency
- Paketo Deps Dependency
- Aternity Dependency
- Rustup Init Dependency
- Rust Dependency
- Skywalking Dependency
- Spring Generations
- Tomcat Dependency
- Tomee Dependency
- YourKit Dependency
- License
Octo Pipelines is a CLI that generates a collection of GitHub Workflows and other Github-related artifacts in a repository. The pipeline generator takes a simplified descriptor as input and transforms it into more complicated workflows and artifacts.
The command line can either be built or run directly using Go.
$: go run github.com/paketo-buildpacks/pipeline-builder/cmd/octo --descriptor <DESCRIPTOR>
The input is a YAML-based descriptor, examples of which can be found in .github/pipeline-descriptor.yml
in other repositories.
Regardless of the contents of the descriptor the pipeline builder creates a minimal set of workflows and artifacts.
A Dependendabot configuration file is added to all repositories. It will always add an update for github-actions
, and if the repository contains a go.mod
file, it will also add an update for gomod
.
In order to facilitate the automated creation of draft releases and notes, a set of semver-scope labels (semver:major
, semver:minor
, and semver:patch
) and type labels (type:bug
, type:dependency-upgrade
, type:documentation
, type:enhancement
, type:question
, type:task
) are added to the repository. In addition, a workflow that enforces exactly one of each label is attached to every PR.
A pipeline-updating workflow is added to the repository to ensure that workflows are kept up to date as the pipeline-builder
evolves.
Draft release notes are created on every commit to main
. These notes take into account every PR since the previous release in order to create a division of the types of changes that were made and the semver scope of the change to work out the next release number.
The descriptor is a YAML document with a number of top-level keys that correspond to new workflows, modified workflows, or artifacts. All top-level keys except github
are optional.
github:
username: ${{ secrets.JAVA_GITHUB_USERNAME }}
token: ${{ secrets.JAVA_GITHUB_TOKEN }}
mappers:
- '|tanzu-buildpacks|paketo-buildpacks|'
- '|tanzu-buildpacks\/|pivotal-cf/tanzu-|'
buildpack_toml_paths:
'tanzu-buildpacks/java-function': buildpacks/java
github
is an object the describes the GitHub identity that the pipeline should use in the many places that GitHub API access is required. The token must be granted the repo
, write:packages
, and workflow
scopes.
The mappers
list is optional and is used by the Draft Release workflow. Mappers are used to control how the action looks up metadata in Github for a buildpack. See the action notes for more information.
The buildpack_toml_paths
map allows you to specify a map of buildpack ids and map them to the location within the Github repository where the buildpack.toml file exists. The path does not need to contain buildpack.toml
, that is assumed and added automatically (although including it won't break anything).
codeowners:
- path: "*"
owner: "@paketo-buildpacks/java-buildpacks"
codeowners
is a list of objects test describe a path
and owner
. Each of these objects corresponds to a line in CODEOWNERS
file.
package:
repository: ""
repositories: ["index.docker.io/paketobuildpacks/adoptium","gcr.io/paketo-buildpacks/adoptium"]
include_dependencies: false
register: true
registry_token: ${{ secrets.JAVA_REGISTRY_TOKEN }}
source_path: subdir/adoptium
platform:
os: linux
package
is an object that describes the repositories
a buildpackage should be published to as well as whether to include the buildpackage's dependencies when creating it (false
by default). If defined, a create-package
workflow is created that creates and publishes a new package when a release is published as well as adds a create-package
job to the tests workflow that is run on each PR and each commit. It will also add additional content to the draft release notes about the contents of the build package and will update the digest of the buildpackage in the published release notes. If register
is true
, after the package is created, it is registered with the Buildpack Registry Index.
repository
is the primary repository. It is the repository registered with the Buildpack Registry Index. It defaults to the first entry in repositories
so if a value is specified, this will take preference over repositories[0]
.
repositories
is the list of repositories that the image should be published to. The list is an array of strings. If repository
is not set then the first registry address will be used for repository
, which is the address that's added to the Buildpack Registry Index. When multiple repositories are set, the image will be packaged once and then copied to subsequent repository addresses specified for efficiency.
If you have a single repository to which you want to publish, using repository
is preferred but it is the same as setting repositories
to a single repository. If you have multiple repositories, you may set both repository
and repositories
but that's only necessary if you need a repository registered with the Buildpack Registry Index that's not the first in the repositories
list.
source_path
is the optional path to the buildpack's directory relative to the repository's root. Defaults to the repository root.
platform
describes what platform the created package should be built for. os
can be set to linux
or windows
(linux
by default).
builder:
repository: gcr.io/projectriff/builder
builder
is an object that describes the repository
a builder should be published to. If defined, a create-builder
workflow is created that's creates and publishes a new builder when a release is published as well as adds a create-builder
job to the tests workflow that is run on each PR and each commit. It will also add additional content to the draft release notes about the contents of the build package and will update the digest of the builder in the published release notes. Finally it will add update workflows for the lifecycle and builder image elements of the builder.toml
file.
docker_credentials:
- registry: gcr.io
username: _json_key
password: ${{ secrets.JAVA_GCLOUD_SERVICE_ACCOUNT_KEY }}
docker_credentials
is a list of objects that describes a registry
, username
, and password
for authenticating against a Docker registry. In every job that registry access is required, a Docker Login Action is created and conditionalized so that it will only run on pushes to main
and on PRs that come from the same repository (to protect the secrets).
http_credentials:
- host: download.aquasec.com
username: ${{ secrets.AQUA_SECURITY_USERNAME }}
password: ${{ secrets.AQUA_SECURITY_PASSWORD }}
http_credentials
is a list of objects that describe a host
, username
, and password
for authenticating requests against an HTTP endpoint. In every job that HTTP access is required a .netrc
file is written and conditionalized so that it will only be written on pushes to main
and on PRs that come from the same repository (to protect the secrets).
dependencies:
- name: JRE 11
id: jre
version_pattern: "11\\.[\\d]+\\.[\\d]+"
purl_pattern: "version-11\\.[\\d]+\\.[\\d]+"
cpe_pattern: "patch[\\d]+"
uses: docker://ghcr.io/paketo-buildpacks/actions/adoptium-dependency:main
with:
implementation: hotspot
type: jre
version: "[11,12)"
dependencies
is a list of objects that define how dependencies are detected and updated by describing an optional name
(defaults to id
), id
that matches a buildpack.toml
defined dependency id, an optional version_pattern
that defines which dependency with a given id
to update, optional purl_pattern
and cpe_pattern
values which allow you to provide alternative patterns for updating the CPEs and PURL (they default to the version pattern), a uses
to define which GitHub Action to use to find the next version, and a with
used to configure the GitHub action to use to find the next version. If defined, each object will create an update
workflow that is responsible for detecting a new version, updating buildpack.toml
and opening a PR to include the change in the repository, if appropriate.
test:
steps:
- name: Install richgo
run: |
#!/usr/bin/env bash
set -euo pipefail
echo "Installing richgo ${RICHGO_VERSION}"
mkdir -p "${HOME}"/bin
echo "${HOME}/bin" >> "${GITHUB_PATH}"
curl \
--location \
--show-error \
--silent \
"https://github.com/kyoh86/richgo/releases/download/v${RICHGO_VERSION}/richgo_${RICHGO_VERSION}_linux_amd64.tar.gz" \
| tar -C "${HOME}"/bin -xz richgo
- name: Run Tests
run: |
#!/usr/bin/env bash
set -euo pipefail
richgo test ./...
env:
RICHGO_FORCE_COLOR: "1"
test
is an object that allows you to override the default test configuration by specifying a list of steps
as defined by the GitHub Actions reference documentation. The default value is described in the example above.
path: ..
path
is a pointer to a Git repository, on your local filesystem, where files will be created. If unspecified, the default is ..
.
offline_packages:
- source: paketo-buildpacks/adoptium
target: gcr.io/tanzu-buildpacks/adoptium
tag_prefix: my-buildpack/
source_path: subdir/adoptium
platform:
os: linux
offline_packages
is a list of objects that describe a source
GitHub repository and a target
Docker registry location. If defined, each object will create a create-package
workflow that is responsible for detecting a new online buildpackage release and creating a matching offline buildpackage release and publishing it.
source_path
is the optional path to the buildpack's directory relative to the repository's root. Defaults to the repository root.
tag_prefix
is the optional prefix to filter for when detecting the buildpack's version via tags. Defaults to empty string.
platform
describes what platform the created package should be built for. os
can be set to linux
or windows
(linux
by default).
republish_images:
- source: gcr.io/paketo-buildpacks/bellsoft-liberica
target: gcr.io/tanzu-buildpacks/bellsoft-liberica-lite
id: tanzu-buildpacks/bellsoft-liberica-lite
target_repo: paketo-buildpacks/bellsoft-liberica
tag_prefix: java/
republish_images
is a list of objects that describe a source
Docker registry location, a target
Docker registry location, and a new buildpack id. If defined, each object will create a republish-image
workflow that is responsible for detecting new source images, modifying the buildpack id of the image to the new id, and publishing a new target image with the modified buildpack id.
The target_repo
setting can be used to point the workflow to a different source control repository for the target image. It's assumed to be the current repository, but for cases where you have a utility repo which is republishing multiple images you need to point the workflow to are repository from which it can read the git tags and extract the target version. The example above, points it to the source image's code repository so it'll end up with a source and target version that are in sync.
tag_prefix
is the optional prefix to filter for when detecting the buildpack's version via tags. Defaults to empty string.
actions:
- source: adoptium-dependency
target: ghcr.io/paketo-buildpacks/actions/adoptium-dependency
actions
is a list of objects that describe a source
directory within this repository's actions/
directory, and a target
Docker registry location. If defined, each object will create a create-action
workflow that is responsible for building and publishing new versions of the action.
This repository contains the source code and build pipelines for a collection of GitHub actions that are used by buildpacks. Most of these actions are used to find the latest version of given dependencies.
All the of the dependency actions also accept a pre_release
configuration option (true
by default) of whether to return pre-release versions.
The Adoptium Dependency queries the Adoptium API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/adoptium-dependency:main
with:
implementation: hotspot
type: jre
version: "[11,12)"
The Alibaba Dependency watches Alibaba repositories for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/alibaba-dragonwell-dependency:main
with:
glob: Alibaba_Dragonwell_[\d\.]+_x64_linux.tar.gz$
repository: dragonwell<jdk-version>
token: ${{ secrets.JAVA_GITHUB_TOKEN }}
The Amazon Corretto Dependency watches Amazon Corretto repositories for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/amazon-corretto-dependency:main
with:
glob: amazon-corretto-11\.[\d.-]+-linux-x64.tar.gz
repository: corretto-11
The AppDynamics Dependency queries the AppDynamics API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/appdynamics-dependency:main
with:
type: sun-jvm
The Aqua Security Dependency queries the Aqua Security API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/aqua-security-dependency:main
with:
username: ${{ secrets.AQUA_SECURITY_USERNAME }}
password: ${{ secrets.AQUA_SECURITY_PASSWORD }}
The Azul Zulu Dependency queries the Azul Zulu API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/azul-zulu-dependency:main
with:
type: headfull
version: "11"
The Bellsoft Liberica Dependency queries the Bellsoft Liberica API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/bellsoft-liberica-dependency:main
with:
type: jre
version: "11"
The CA APM Dependency watches the CA APM Download Page for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/ca-apm-dependency:main
with:
type: java
The CF Java Index Dependency querys a CloudFoundry Java buildpack v2 index.yml
file for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/cf-java-index-dependency:main
with:
repository_root: <repo-root>
The Clojure Tools Dependency watches Clojure Tools repositories for new versions. It then filters based on the stable.properties file in their brew tap repo, allowing it to pick the most recent stable version.
uses: docker://ghcr.io/paketo-buildpacks/actions/clojure-tools-dependency:main
with:
token: ${{ secrets.JAVA_GITHUB_TOKEN }}
The Compute Artifact Dependencies takes a buildpack image and produces the list of included dependencies. It's used for generating release notes. If the image is a composite buildpack, then it will recursively load all of the referenced buildpacks, pulling metadata directly from Github, and give a compete, sorted and deduplicated list of dependencies.
A Github token is strongly recommended. This is for fetching information from Github, and without it you might get rate limited. It is required for private repositories.
The package
and version
entries are required. They are required to indiciate the specific buildpack that will be described.
A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of gcr.io/foo/bar:1.2.3
but the Github project is at github.com/org/foo-bar
then you can add a mapper of |foo\/bar|org/foo-bar|
to have compute-artifact-dependencies
try both. You can specify as many mappers as you want, with each mapper generating a new URI for compute-artifact-dependencies
to try. It will stop when it hits the first URL that successfully pulls back the buildpack.toml
file.Mappers may use regexp capture groups and reference the captured value with $<num>
(ex: $1
, $2
, etc) in the regular expression.
The location of the buildpack.toml
file in the Github project is assumed to be at /buildpack.toml
. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable in the format BP_TOML_PATH_<org>_<repo>=actual/path
where <org>
and <repo>
are the organization and repo pulled out of the image URI. For example, gcr.io/foo/bar:1.2.3
. The org is foo
and the repo is bar
. A leading /
is not necessary, nor is buildpack.toml
on the end. You may omit that to shorten the value and the action will automatically add /buildpack.toml
to the end. For example, with an org of vmware-tanzu
and a repo of function-buildpacks-for-knative
the environment variable would be BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE=buildpacks/java/buildpack.toml
, which would be the same as BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE=buildpacks/java
since the trailing buildpack.toml
is optional.
The output is artifact-reference-description
, which can be referenced by other jobs.
uses: docker://ghcr.io/paketo-buildpacks/actions/compute-artifact-description:main
with:
github_token: ${{ secrets.JAVA_GITHUB_TOKEN }}
package: "gcr.io/paketo-buildpacks/java"
version: "1.2.3"
mapper_1: '|tanzu-buildpacks|paketo-buildpacks|'
mapper_2: '|tanzu-buildpacks\/|pivotal-cf/tanzu-|'
The draft-release
action is used to augment release notes generated by the release-drafter
action. The release-drafter
pulls in notes based on PR changes. The draft-release
action is what augments those with all of the buildpack specific information. For example, dependencies included or order groups.
The draft-release
action needs to be able to look up and remotely fetch buildpack metadata. It does this by obtaining the buildpack.toml
file for any dependent buildpacks from Github. This requires mapping the image URI to the Github project URI, which is not always one-to-one. As such, the action supports mappers.
A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of gcr.io/foo/bar:1.2.3
but the Github project is at github.com/org/foo-bar
then you can add a mapper of |foo\/bar|org/foo-bar|
to have draft-release
try both. You can specify as many mappers as you want, with each mapper generating a new URI for draft-release
to try. It will stop when it hits the first URL that successfully pulls back the buildpack.toml
file. Mappers may use regexp capture groups and reference the captured value with $<num>
(ex: $1
, $2
, etc) in the regular expression.
The location of the buildpack.toml
file in the Github project is assumed to be at /buildpack.toml
. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable for the action in the format BP_TOML_PATH_<org>_<repo>=actual/path
where <org>
and <repo>
are the organization and repo pulled out of the image URI. For example, gcr.io/foo/bar:1.2.3
. The org is foo
and the repo is bar
.
A leading /
on the environment variable is not necessary, nor is buildpack.toml
on the end. You may omit both to shorten the value and the action adjust as required. For example, with an org of tanzu-buildpacks
and a repo of java-function
then the environment variable would be BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION=/buildpacks/java/buildpack.toml
, which would be the same as BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION=buildpacks/java
since the trailing buildpack.toml
is optional.
uses: docker://ghcr.io/paketo-buildpacks/actions/draft-release:main
with:
github_token: ${{ secrets.JAVA_GITHUB_TOKEN }}
mapper_1: '|tanzu-buildpacks|paketo-buildpacks|'
mapper_2: '|tanzu-buildpacks\/|pivotal-cf/tanzu-|'
release_body: ${{ steps.release-drafter.outputs.body }}
release_id: ${{ steps.release-drafter.outputs.id }}
release_name: ${{ steps.release-drafter.outputs.name }}
release_tag_name: ${{ steps.release-drafter.outputs.tag_name }}
env:
BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION: buildpacks/java
The Foojay Dependency queries the Foojay API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/foojay-dependency:main
with:
distro: microsoft
type: jdk
version: "11"
The GCS Dependency watches GCS Buckets for new versions. The first capture in glob
is used as the version.
uses: docker://ghcr.io/paketo-buildpacks/actions/gcs-dependency:main
with:
bucket: projectriff
glob: command-function-invoker/command-function-invoker-linux-amd64-([\d\.]+).tgz
The GitHub Release Dependency watches GitHub Releases for new versions. The first capture in glob
is used as the version.
uses: docker://ghcr.io/paketo-buildpacks/actions/github-release-dependency:main
with:
glob: sapmachine-jre-.+_linux-x64_bin.tar.gz
owner: SAP
repository: SapMachine
tag_filter: sapmachine-(11.*)
token: ${{ secrets.GITHUB_TOKEN }}
The Google Stackdriver Profiler Dependency watches the Google Cloud Profiler bucket for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/google-stackdriver-profiler-dependency:main
The GraalVM Dependency watches the GraalVM Repository for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/graalvm-dependency:main
with:
glob: graalvm-ce-java11-linux-amd64-.+.tar.gz
token: ${{ secrets.GITHUB_TOKEN }}
version: 11
The Gradle Dependency queries the Gradle API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/gradle-dependency:main
The IBM Semeru Dependency queries the Gradle API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/ibm-semeru-dependency:main
with:
glob: ibm-semeru-open-jdk_x64_linux_.+_openj9-.+.tar.gz
owner: ibmruntimes
repository: semeru8-binaries
tag_filter: jdk(.*8.*)
token: ${{ secrets.JAVA_GITHUB_TOKEN }}
The JProfiler Dependency watches the JProfiler Changelog for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/jprofiler-dependency:main
The JRebel Dependency watches the JRebel Download Page for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/jrebel-dependency:main
The Leiningen Dependency watches the Leiningen Repository for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/leiningen-dependency:main
with:
token: ${{ secrets.GITHUB_TOKEN }}
The Liberty Dependency watches a Maven Repository for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/liberty-dependency:main
with:
uri: https://repo1.maven.org/maven2
group_id: org.apache.maven
artifact_id: apache-maven
classifier: bin
packaging: tar.gz
The Maven Dependency queries a Maven Repository for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/maven-dependency:main
with:
uri: https://repo1.maven.org/maven2
group_id: org.apache.maven
artifact_id: apache-maven
classifier: bin
packaging: tar.gz
version_regex: ^3\.[\d]+\.[\d]+$
The New Relic Dependency watches the New Relic Download Page for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/new-relic-dependency:main
with:
type: php
The NPM Dependency queries the NPM API for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/npm-dependency:main
with:
package: "@google-cloud/debug-agent"
The OverOps Dependency watches the OverOps Download Page for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/overops-dependency:main
The Paketo Deps Dependency queries the Paketo Deps Server for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/paketo-deps-dependency:main
with:
name: rust
The Aternity Dependency watches the Aternity Bucket for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/aternity-dependency:main
The Rustup Init Dependency queries the Rustup Github Project for new versions. The target
specifies the target triple to download.
uses: docker://ghcr.io/paketo-buildpacks/actions/rustup-init-dependency:main
with:
target: x86_64-unknown-linux-musl
token: ${{ secrets.PAKETO_BOT_GITHUB_TOKEN }}
The Rust Dependency queries the Rust Github Project for the latest version. The target
specifies the target triple to download.
uses: docker://ghcr.io/paketo-buildpacks/actions/rust-dependency:main
with:
target: x86_64-unknown-linux-gnu
token: ${{ secrets.PAKETO_BOT_GITHUB_TOKEN }}
The Skywalking Dependency watches the Apache Skywalking Download Page for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/skywalking-dependency:main
The Spring Generations queries the Spring Generations API for the generation lifecycle of Spring projects.
uses: docker://ghcr.io/paketo-buildpacks/actions/spring-generations:main
with:
uri: https://spring.io/api
The Tomcat Dependency watches the Apache Tomcat Download Page for new versions. An optional version_regex
can be specified to filter on a specific minor versions. If set, the regex must include three capture groups. For example, ^v(10)\.(0)\.([\d]+)/$
to match 10.0.*
.
uses: docker://ghcr.io/paketo-buildpacks/actions/tomcat-dependency:main
with:
uri: https://downloads.apache.org/tomcat/tomcat-9
version_regex: ^v([\d]+)\.([\d]+)\.([\d]+)/$
The Tomee Dependency watches the Apache Tomee Download Page for new versions. Available distributions are microprofile
, webprofile
, plus
or plume
.
uses: docker://ghcr.io/paketo-buildpacks/actions/tomee-dependency:main
with:
uri: https://downloads.apache.org/tomee/
dist: webprofile
major: 9
The YourKit Dependency watches the YourKit Download Page for new versions.
uses: docker://ghcr.io/paketo-buildpacks/actions/yourkit-dependency:main
This library is released under version 2.0 of the Apache License.