Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to configure config extensions for repos #3349

Open
wants to merge 62 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
ab9f474
add repo plugins
anbraten Apr 5, 2022
f4c234f
use ed25519 for signing
anbraten May 14, 2022
180d767
revert unrelated
anbraten May 14, 2022
c4e8f2d
clean up code and fix tests
anbraten May 14, 2022
754a823
Merge remote-tracking branch 'upstream/master' into repo-plugins
anbraten May 14, 2022
0a479bb
clean up
anbraten May 14, 2022
ef13725
use async key pair for webhooks
anbraten May 14, 2022
ce7f026
fix tests
anbraten May 14, 2022
d0c8986
fix linter
anbraten May 14, 2022
a63433f
improve code
anbraten May 14, 2022
960bfac
add key pair to database
anbraten May 15, 2022
38d84e3
Merge remote-tracking branch 'upstream/master' into repo-plugins
anbraten May 17, 2022
bdb6887
Merge branch 'webhook-pubkey' into repo-plugins
anbraten May 17, 2022
5709122
add some extension docs
anbraten May 17, 2022
49396a9
Merge remote-tracking branch 'upstream/master' into repo-plugins
anbraten Jun 1, 2022
c1e1e4d
fix migration and undo some changes
anbraten Jun 1, 2022
c6257f3
Merge remote-tracking branch 'upstream/master' into repo-plugins
anbraten Jun 18, 2022
8bed39b
remove vendor folder
anbraten Jun 29, 2022
c603968
adjust names
anbraten Jun 29, 2022
1399009
fiy typo
anbraten Jun 29, 2022
889cc96
add extensions manager
anbraten Jul 2, 2022
554a2fc
Merge remote-tracking branch 'upstream/master' into repo-plugins
anbraten Jul 2, 2022
4bf532b
add extensions ui
anbraten Jul 2, 2022
dbd9d83
cleanup code
anbraten Jul 2, 2022
e497563
adjust spacings
anbraten Jul 2, 2022
84266ce
Merge branch 'master' into repo-plugins
anbraten Jul 2, 2022
12b8c54
fix colors
anbraten Jul 2, 2022
2539b6d
Merge branch 'repo-plugins' of github.com:anbraten/woodpecker into re…
anbraten Jul 2, 2022
2015d02
add docs links and fix types
anbraten Jul 2, 2022
bde2df0
Merge branch 'master' into repo-plugins
anbraten Jul 4, 2022
f37c392
fix tests
anbraten Jul 4, 2022
f43ff61
Merge remote-tracking branch 'upstream/main' into plugins
anbraten Feb 11, 2024
e384b7f
[pre-commit.ci] auto fixes from pre-commit.com hooks [CI SKIP]
pre-commit-ci[bot] Feb 11, 2024
9244c67
cleanup
anbraten Feb 11, 2024
381bcfb
Merge branch 'plugins' of github.com:anbraten/woodpecker into plugins
anbraten Feb 11, 2024
8af6ed5
use configured endpoints
anbraten Feb 11, 2024
0a600d9
improve
anbraten Feb 11, 2024
0a0a89a
Merge remote-tracking branch 'upstream/main' into plugins
anbraten Jun 5, 2024
bf73ef2
undo unrelated
anbraten Jun 5, 2024
15c0951
finish
anbraten Jun 5, 2024
b7fcc08
adjust urls and add debug loggin
anbraten Jun 5, 2024
4fc4608
Merge remote-tracking branch 'upstream/main' into plugins
anbraten Jun 5, 2024
072e1c2
update docs
anbraten Jun 5, 2024
17973b0
Merge branch 'main' into plugins
anbraten Jun 5, 2024
5ae18de
adjust docs
anbraten Jun 5, 2024
6b9bc26
Merge branch 'plugins' of github.com:anbraten/woodpecker into plugins
anbraten Jun 5, 2024
22ef60a
add endpoint check
anbraten Jun 5, 2024
98da4b4
prettier
anbraten Jun 5, 2024
c39f456
fix translations
anbraten Jun 5, 2024
d2b206c
update swagger
anbraten Jun 5, 2024
79cbe97
adjsut docs
anbraten Jun 5, 2024
826ea1f
Merge remote-tracking branch 'upstream/main' into plugins
anbraten Sep 30, 2024
93a8392
fix
anbraten Sep 30, 2024
d17793c
undo secret
anbraten Sep 30, 2024
399d877
review
anbraten Sep 30, 2024
9f61ad1
Update web/src/assets/locales/en.json
anbraten Sep 30, 2024
d760e18
fix
anbraten Sep 30, 2024
322b94f
Merge branch 'plugins' of github.com:anbraten/woodpecker into plugins
anbraten Sep 30, 2024
9afdde4
use const
anbraten Sep 30, 2024
4a4821c
fix
anbraten Sep 30, 2024
46f4c58
fix
anbraten Sep 30, 2024
5fc3096
Merge branch 'main' into plugins
anbraten Oct 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions cmd/server/docs/docs.go
Original file line number Diff line number Diff line change
Expand Up @@ -4179,6 +4179,9 @@ const docTemplate = `{
"clone_url_ssh": {
"type": "string"
},
"config_extension_endpoint": {
"type": "string"
},
"config_file": {
"type": "string"
},
Expand Down Expand Up @@ -4225,6 +4228,9 @@ const docTemplate = `{
"scm": {
"$ref": "#/definitions/SCMKind"
},
"secret_extension_endpoint": {
"type": "string"
},
"timeout": {
"type": "integer"
},
Expand All @@ -4251,6 +4257,9 @@ const docTemplate = `{
"$ref": "#/definitions/WebhookEvent"
}
},
"config_extension_endpoint": {
"type": "string"
},
"config_file": {
"type": "string"
},
Expand All @@ -4260,6 +4269,9 @@ const docTemplate = `{
"netrc_only_trusted": {
"type": "boolean"
},
"secret_extension_endpoint": {
"type": "string"
},
"timeout": {
"type": "integer"
},
Expand Down
5 changes: 5 additions & 0 deletions cmd/server/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,11 @@ var flags = append([]cli.Flag{
Name: "config-service-endpoint",
Usage: "url used for calling configuration service endpoint",
},
&cli.StringFlag{
EnvVars: []string{"WOODPECKER_ALLOWED_EXTENSIONS_HOSTS"},
Name: "allowed-extensions-hosts",
Usage: "Hosts that are allowed to be used by extensions",
},
&cli.StringFlag{
EnvVars: []string{"WOODPECKER_DATABASE_DRIVER"},
Name: "driver",
Expand Down
66 changes: 66 additions & 0 deletions docs/docs/20-usage/72-extensions/20-secrets-extension.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Secrets extensions (alpha state)

:::warning
The secret extension is still very expirimental and might change at any time. If you have any feedback or ideas please let us know.
:::

Woodpecker allows you to replace the internal secrets service with an external HTTP extension. This allows you to store secrets in a different system like Hashicorp Vault or AWS Secrets Manager.

## Security

:::warning
As Woodpecker will pass private information like tokens and will execute the returned configuration, it is extremely important to secure the external extension. Therefore Woodpecker signs every request. Read more about it in the [security section](./10-extensions.md#security).
:::

## How it works

For example if you open the secrets list in the repository settings, the following will happen:

- Woodpecker will send a HTTP GET request to the `<this-server-url>/repo/<repo-id>/secrets` endpoint of your extension
- The extension will first verify the request is coming from the actual Woodpecker instance and wasn't modified in transit
using a http signature. This is pretty important so no one can just send a request to this server and get all your
secrets for example.
- The extension server will then return a list of secrets that are available for this repository.
- The secrets will be displayed in the UI.

### Endpoints

The secrets extension needs the following endpoints to work.
All data is sent as JSON in the request body and the response should also be JSON.
A successful request should always return a 2xx status code.

The secret json object should look like this:

```ts
class Secret {
id: number;
org_id: number; // has to be 0 for repository secrets
repo_id: number;
name: string;
value: string;
images: string[]; // a list of docker images this secret can be used with
events: string[]; // a list of pipeline events this secret can be used with
}
```

Check out the [secret model](https://github.com/woodpecker-ci/woodpecker/blob/main/server/model/secret.go) for more information.

#### `GET /repo/:repoId/secrets`

Should return an array of secrets for the repository:

#### `GET /repo/:repoId/secrets/:secretName`

Should return a single secret for the repository.

#### `POST /repo/:repoId/secrets`

Create a new secret for the repository.

#### `PUT /repo/:repoId/secrets/:secretName`

Update an existing secret for the repository.

#### `DELETE /repo/:repoId/secrets/:secretName`

Delete a secret from the repository.
163 changes: 163 additions & 0 deletions docs/docs/20-usage/72-extensions/40-configuration-extension.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# Configuration extension

The configuration extension can be used to modify or generate Woodpeckers pipeline configurations. You can configure a HTTP
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The configuration extension can be used to modify or generate Woodpeckers pipeline configurations. You can configure a HTTP
The configuration extension can be used to modify or generate Woodpeckers pipeline configurations. You can configure an HTTP

endpoint in the repository settings in the extensions tab.

Using such an extension can be useful if you want to:

- Preprocess the original configuration file with something like go templating
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Preprocess the original configuration file with something like go templating
- Preprocess the original configuration file with something like Go template

- Convert custom attributes to Woodpecker attributes
- Add defaults to the configuration like default steps
- Convert configuration files from a totally different format like Gitlab CI config, Starlark, Jsonnet, ...
- Centralize configuration for multiple repositories in one place

## Security

:::warning
As Woodpecker will pass private information like tokens and will execute the returned configuration, it is extremely important to secure the external extension. Therefore Woodpecker signs every request. Read more about it in the [security section](./10-extensions.md#security).
:::

## Global configuration

In addition to the ability to configure the extension per repository, you can also configure a global endpoint in the Woodpecker server configuration. This can be useful if you want to use the extension for all repositories. Be careful if
you share your Woodpecker server with others as they will also use your configuration extension.

The global configuration will be called before the repository specific configuration extension if both are configured.

```ini title="Server"
WOODPECKER_CONFIG_SERVICE_ENDPOINT=https://example.com/ciconfig
```

## How it works

When a pipline is triggered Woodpecker will fetch the pipeline configuration from the repository, then make a HTTP POST request to the configured extension with a JSON payload containing some data like the repository, pipeline information and the current config files retrieved from the repository. The extension can then send back modified or even new pipeline configurations following Woodpeckers offical yaml format that should be used.

:::tip
The netrc data is pretty powerful as it contains credentials to access the repository. You can use this to clone the repository or even use the forge (Github or Gitlab, ...) api to get more information about the repository.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate?

:::

### Request

The extension receives an HTTP POST request with the following JSON payload:

```ts
class Request {
repo: Repo;
pipeline: Pipeline;
netrc: Netrc;
configuration: {
name: string; // filename of the configuration file
data: string; // content of the configuration file
}[];
}
```

Checkout the following models for more information:

- [repo model](https://github.com/woodpecker-ci/woodpecker/blob/main/server/model/repo.go)
- [pipeline model](https://github.com/woodpecker-ci/woodpecker/blob/main/server/model/pipeline.go)
- [netrc model](https://github.com/woodpecker-ci/woodpecker/blob/main/server/model/netrc.go)

:::tip
The `netrc` data is pretty powerful as it contains credentials to access the repository. You can use this to clone the repository or even use the forge (Github or Gitlab, ...) api to get more information about the repository.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The `netrc` data is pretty powerful as it contains credentials to access the repository. You can use this to clone the repository or even use the forge (Github or Gitlab, ...) api to get more information about the repository.
The `netrc` data is pretty powerful as it contains credentials to access the repository. You can use this to clone the repository or even use the forge (Github or Gitlab, ...) API to get more information about the repository.

:::

Example request:

```json
{
"repo": {
"id": 100,
"uid": "",
"user_id": 0,
"namespace": "",
"name": "woodpecker-testpipe",
"slug": "",
"scm": "git",
"git_http_url": "",
"git_ssh_url": "",
"link": "",
"default_branch": "",
"private": true,
"visibility": "private",
"active": true,
"config": "",
"trusted": false,
"protected": false,
"ignore_forks": false,
"ignore_pulls": false,
"cancel_pulls": false,
"timeout": 60,
"counter": 0,
"synced": 0,
"created": 0,
"updated": 0,
"version": 0
},
"pipeline": {
"author": "myUser",
"author_avatar": "https://myforge.com/avatars/d6b3f7787a685fcdf2a44e2c685c7e03",
"author_email": "my@email.com",
"branch": "main",
"changed_files": ["somefilename.txt"],
"commit": "2fff90f8d288a4640e90f05049fe30e61a14fd50",
"created_at": 0,
"deploy_to": "",
"enqueued_at": 0,
"error": "",
"event": "push",
"finished_at": 0,
"id": 0,
"link_url": "https://myforge.com/myUser/woodpecker-testpipe/commit/2fff90f8d288a4640e90f05049fe30e61a14fd50",
"message": "test old config\n",
"number": 0,
"parent": 0,
"ref": "refs/heads/main",
"refspec": "",
"clone_url": "",
"reviewed_at": 0,
"reviewed_by": "",
"sender": "myUser",
"signed": false,
"started_at": 0,
"status": "",
"timestamp": 1645962783,
"title": "",
"updated_at": 0,
"verified": false
},
"configs": [
{
"name": ".woodpecker.yaml",
"data": "steps:\n - name: backend\n image: alpine\n commands:\n - echo \"Hello there from Repo (.woodpecker.yaml)\"\n"
}
]
}
```

### Response

The extension should respond with a JSON payload containing the new configuration files in Woodpeckers official yaml format.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The extension should respond with a JSON payload containing the new configuration files in Woodpeckers official yaml format.
The extension should respond with a JSON payload containing the new configuration files in Woodpecker's official YAML format.

If the extension wants to keep the existing configuration files, it can respond with **HTTP 204**.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If the extension wants to keep the existing configuration files, it can respond with **HTTP 204**.
If the extension wants to keep the existing configuration files, it can respond with HTTP `204 No Content`.


```ts
class Response {
configs: {
name: string; // filename of the configuration file
data: string; // content of the configuration file
}[];
}
```

Example response:

```json
{
"configs": [
{
"name": "central-override",
"data": "steps:\n - name: backend\n image: alpine\n commands:\n - echo \"Hello there from ConfigAPI\"\n"
}
]
}
```
33 changes: 33 additions & 0 deletions docs/docs/20-usage/72-extensions/72-extensions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Extensions

Woodpecker allows you to replace internal logic with external extensions by using pre-defined http endpoints.

There are currently two types of extensions available:
Copy link
Contributor

@zc-devs zc-devs Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the second one? Is it possible to configure it globally like config extension?
There are 3 types on screenshot.


- [Configuration extension](./40-configuration-extension.md) to modify or generate Woodpeckers pipeline configurations.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [Configuration extension](./40-configuration-extension.md) to modify or generate Woodpeckers pipeline configurations.
- [Configuration extension](./40-configuration-extension.md) to modify or generate Woodpecker pipeline configurations.

- [Secrets extension (alpha state)](./20-secrets-extension.md) to receive and update secrets from an external system like Hashicorp Vault or AWS Secrets Manager.

## Security

:::warning
You need to trust the extensions as they are receiving private information like secrets and tokens and might return harmful
data like malicious pipeline configurations that could be executed.
:::

To prevent your extensions from such attaks, Woodpecker is signing all http-requests using [http signatures](https://tools.ietf.org/html/draft-cavage-http-signatures). Woodpecker therefore uses a public-private ed25519 key pair. To verify the requests your extension has to verify the signature of all request using the public key with some library like [httpsig](https://github.com/go-fed/httpsig). You can get the public Woodpecker key by opening `http://my-woodpecker.tld/api/signature/public-key` or by visiting the Woodpecker UI, going to you repo settings and opening the extensions page.

## Example extensions

A simplistic serivce providing endpoints for a config and secrets extension can be found here: [https://github.com/woodpecker-ci/example-extensions](https://github.com/woodpecker-ci/example-extensions)

## Configuration

To prevent extensions from calling local services by default only external hosts / ip-addresses are allowed. You can change this behavior by setting the `WOODPECKER_ALLOWED_EXTENSIONS_HOSTS` environment variable. You can use a comma separated list of:
anbraten marked this conversation as resolved.
Show resolved Hide resolved

- Built-in networks:
- `loopback`: 127.0.0.0/8 for IPv4 and ::1/128 for IPv6, localhost is included.
- `private`: RFC 1918 (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) and RFC 4193 (FC00::/7). Also called LAN/Intranet.
- `external`: A valid non-private unicast IP, you can access all hosts on public internet.
- `*`: All hosts are allowed.
- CIDR list: `1.2.3.0/8` for IPv4 and `2001:db8::/32` for IPv6
- (Wildcard) hosts: `mydomain.com`, `*.mydomain.com`, `192.168.100.*`
7 changes: 7 additions & 0 deletions docs/docs/20-usage/72-extensions/_category_.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
label: 'Extensions'
# position: 3
collapsible: true
collapsed: true
link:
type: 'doc'
id: 'extensions'
Loading