Configuration Extension

[s00500]

Clear and concise description of the problem

Drone uses a env var called DRONE_YAML_ENDPOINT to specify a rest service that can override the default pipeline config before a run is executed. this enables management of multiple repos that need to run the same configuration and centralized configuration of pipelines for repositories in a organisation.

Suggested solution

My Question is: does woodpecker already support something like this ? the closest I have found so fare was the Gating Service, but there is no documentation on it yet and it seems to only allow for prohibiting runs...

Else I am also happy to work on a contribution PR here myself :-) My solution would be to match drones current way of dealing with the problem, providing a new endpoint that will be fetched via http, using a token based authentication flow and retrieving a new yaml "override" configuration

Awesome Project btw, keep up the great work!!!

Alternative

No response

Additional context

No response

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[anbraten]

There isn't something similar like the config plugin / endpoint for WP yet. #53 requested something similar some time ago.

So having such a feature would be awesome. There are two things I am not quite sure about yet:

• What is the underlying usecase here?
  From your description it sounds that you want to solve some kind of templating, right? Would the endpoint really be the best solution here. Github Actions have a composition pipeline feature and Gitlab has templating support allowing to load configs from other repos or http sources. I think the Github Actions compositions feature is quite interesting for reusing configs, but while writing I had a look at Drone samples and really liked the idea of using the config endpoint to replace / add secrets to steps from something like Hashicorp Vault.
• How to add such a feature that will fit most usecases.
  Having a setting similar to Drone would then apply for the whole deployment and would not allow admins of repos to use their own endpoint which will only work for deployments where the instance admin is also the only repo admin. Bigger companies or orgs often have open registrations and could not use this feature then. I can currently think of two solutions: Setting the endpoint in the repo settings (UI) or setting it via pipeline config (probably impossible as it could call external services and could be misused on public repos with PRs).

Sorry for this speech 🙈

tldr: I would currently propose adding an option to the repo settings (UI) which allows to add extensions which are simply http endpoints receiving the config and sending back an adjusted config and adding a server option to set allowed ip-subnets for the external calls to prevent internal network attacks.

[s00500]

Hey @anbraten, great inputs
I started today adding the feature just like it is done in drone.... and I must say this is very straight forward (to add to woodpecker). I have used it like this with drone myself to manage pipelines for about 40 repos. In this case the external system is used to set overrides for individual repos. In theory a permission or management sytem for these could be implemented on the external system....

But I also like the idea of providing repo level customization of the feature from repo settings. Generally: why not have both? The globally configured system would still be able to overrule everything, while the entry in repo settings would overrule the yaml file provided by the repo

I also think that providing a rest endpoint solution is much simpler than any kind of template...

The final question is how the security aspect can be handled best. Drone uses a token sent to the external service to authenticate there. When using https and normal certificate verification the drone server itself should be able to verify that it has hit the correct server. I think we could just add a second field to the repo settings that would allow to set something like a Bearer Token on the request ?

So I guess there is my speech =D

Let me know what you think, I am quite motivated to make a PR for this in the course of the next 2 weekends and play a little bit with it :-)
I really like the stack of this project, go+vue, vite, very clean codestructure in most places, looks really nice!

[anbraten]

Sounds great. Having a global option overriding the per repo settings for those endpoint sounds fine to me.

For security I like your approach of having a Bearer Token, similar to how Github, Gitea, etc have it for their webhook settings.
In addition we should then add a filter so the webhook is only allowed to call specific destionations, so a user would be prevented from doing GET request in the Woodpecker servers private network. Something similar to how they implemented it for Gitea: https://github.com/go-gitea/gitea/pull/17482/files

Just start with the implementation part you like most 😉 and please feel free to ask again if something is unclear or if you need any help.

[s00500]

Hm... when setting up my testsetup I hit a weird issue, maybe you can give a quick tipp....

I am running agent and server on my M1 mac, all good, connecting to my own gitea instance at git.lbsfilm.at, using a ssh tunnel to get the webhook back to my local... all good...

I could not get pipelines to run though, they would keep getting stuck in pending. I realized that I can make them run by commenting out the worker filtering on the server side in server/queue/fifio.go func (q *fifo) assignToWorker() (*list.Element, *worker)

When I do that it runs fine... but why would it not execute otherwise ?

The filter created by the agent looks like this: {map[platform:darwin/amd64] }
Exactly what I would expect...

My Pipeline:

pipeline:
  backend:
    image: alpine
    commands:
      - echo "Hello there"

I also tried adding

    when:
      platform: darwin/amd64

Same result... I am using current master, any Ideas ?

Some logs from the server:
queue: trying to assign task: 25 with deps []

Failed task 25 ([]) - map[]

(I did added a else to the filter function that logs the task)

			log.Info().Msgf("Trying worker %+v", w.filter)
			if w.filter(task) {
				log.Info().Msgf("OK worker %v", w)
				log.Debug().Msgf("queue: assigned task: %v with deps %v", task.ID, task.Dependencies)
				return e, w
			} else {
				log.Info().Msgf("Failed task %+v", task)
			}

ServerConfig:

WOODPECKER_DEBUG_PRETTY=true
WOODPECKER_LOG_LEVEL=debug

WOODPECKER_OPEN=false
WOODPECKER_ADMIN=lbsadmin
WOODPECKER_SERVER_ADDR=:8123
WOODPECKER_GRPC_ADDR=:9001
WOODPECKER_HOST=http://testing.lbs.sh

WOODPECKER_GITEA=true
WOODPECKER_GITEA_URL=https://git.lbsfilm.at
WOODPECKER_GITEA_CLIENT=***
WOODPECKER_GITEA_SECRET=***
WOODPECKER_AGENT_SECRET=***

Agent Config:

WOODPECKER_DEBUG_PRETTY=true
WOODPECKER_LOG_LEVEL=trace

WOODPECKER_GRPC_SECURE=false
WOODPECKER_SERVER=localhost:9001
WOODPECKER_AGENT_SECRET=***
WOODPECKER_KEEPALIVE_TIME=5s

I also noticed this message every now and then on the agent

5:58PM TRC grpc: to many keepalive pings without sending data error="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: EOF", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: "too_many_pings""

But I guess that is fine, it does work correctly when I skip the filter on the server...

Maybe you can spot my issue easily ? Otherwise I will keep digging...

[s00500]

Hm ok... tracked it down to the SetPlatform function that defaults to linux/amd64 when the pipeline does not set a value.... feels a little strange... maybe it would make more sense to loosen the match in the filter function and not provide a default ?

[anbraten]

I absolutley agree that we should not use a specific arch as fallback.

[s00500]

Ok, I can make that a separate PR

Another question: When using drone I found it useful that the build config would get fetched again from the rest endpoint when I restarted an existing build. This allowed me to fix bugs in configuration and then rerunning ci without having to push a new change or tag to the actual repository...

When implementing my change here I found that the retrieved config is stored into the database for "for historical correctness, restarts, etc" to quote the comments...

What do you think ? On a pipeline restart should the config get reloaded from the endpoint or should the stored one be used ?

[anbraten]

Ok, I can make that a separate PR

👍🏾

For the other question I would suggest following way: Save repo config as is to database (like we currently do) => after that send config to endpoint and execute the return. For restarting we should be able to simply fetch the stored config from db again, send it to the endpoint again and executed it after that. Only problem that we need to think about is how we show the actual config in the UI.