Implement repo level, user level and global environment variables support

[dvjn]

What?

• Add support for variables at user, org, repo and admin level which will be exposed to the pipelines.
• This is similar to secrets, but they will not be limited to specific events and plugins and will not be masked.

Why?

• There are use cases when we need to set some org level or repo level variables, which need not be a secret.
• A workaround is to use secrets, but then it masks the variable from all the output which is annoying sometimes.

[dvjn]

Hi @6543 @anbraten @qwerty287 I have done initial UI changes for this, please review the proposal and screenshots and let me know if I should continue working on this and implement the rest.

Screenshots:

Repo Settings:

User Settings:

Admin Settings:

Add Variable Screen:

[anbraten]

As variables are not secret anyways there was a suggestion somewhere to have a special config file like .woodpecker/config.yaml in the repo which can be used for such settings. This would be quite handy as it would also allow to use those variables in the cli exec command as well and allows easy copies of that config to other woodpecker instances etc. Should those variables then be shown in the UI as well as read-only or should only the config file be the source of them? What do you think?

[qwerty287]

should only the config file be the source of them

I'd prefer this way, but it's not that easy to have a good way to implement this at org/global level.

[zc-devs]

JFYI, how it is implemented in Gitea.

[dvjn]

As variables are not secret anyways there was a suggestion somewhere to have a special config file like .woodpecker/config.yaml in the repo which can be used for such settings.

I think this is not as useful as it seems, since you can already use variables in a workflow file. This config.yaml approach is mainly beneficial when you need these variables in multiple workflows within the same repository.
However, the most important use-cases of this PR, in my opinion, are: first, the ability to manage global variables using the admin UI, and second, the capability to share variables across repositories under an organization or user.

FYI, the solution I'm proposing is heavily inspired by Gitea.

[dvjn]

@anbraten @qwerty287 @zc-devs
Do let me know your final thoughts on this and let me know if I should move forward with implementing it.

[dvjn]

I can also add agent variables to address #3311

[zc-devs]

your final thoughts

I like the idea.

if I should move forward with implementing it

I am not a right guy :) Let's hear from maintainers.

I can also add agent variables

That would be great!

[qwerty287]

Yes, the way you proposed would be fine for me.

However, you write "Fixes #3661" in the desc which is not correct (I believe there's an issue about your thing too). #3661 is about workflow-level vars and this should definitely be defined in the yaml.

[dvjn]

However, you write "Fixes #3661" in the desc which is not correct (I believe there's an issue about your thing too). #3661 is about workflow-level vars and this should definitely be defined in the yaml.

My bad, will remove that issue from the description.

[itsTurnip]

I'm really interested in this PR. Keep moving! Thanks for your work.

[dvjn]

I have completed the code changes in the server and web frontend. This means the feature is now in a fully usable state.
The docs and cli changes are still pending. Planning to do those after an initial round of code review.

It would be great if someone could review the code.

Summary of Changes:

• Duplicated code for secrets (removing events and images).
• Added support for from_variable and variables (similar to from_secret and secrets) in pipeline config.
• Added new table in database variables, with migrations.
• Copied some secrets test cases and used for variables.

[lafriks]

Wasn't there an idea to drop secrets section from step? Maybe let's do not add variables section and keep only one way to use them with from_variable?

[dvjn]

Wasn't there an idea to drop secrets section from step?

I am not aware of it. If it is just an idea without a proper discussion, maybe let's not act on that?

Maybe let's do not add variables section and keep only one way to use them with from_variable?

I have kept feature parity with secrets, to not cause confusion. If we want to remove it, wouldn't it be better to remove both secrets and variables together in a separate PR.

Just my thoughts, let me know if we still want to remove it.

[zc-devs]

If I set some variable

and run pipeline

skip_clone: true
steps:
  glob-vars:
    image: alpine
    commands:
      - echo $$COMPANY

what would I see in steps (UI) logs?

---

Maybe add build_pr_images label?

[dvjn]

what would I see in steps (UI) logs?

It would be empty as you haven't explicitly added those variables in your step either by using variables or from_variable.

[zc-devs]

As this is not a secret value, it doesn't make sense to restrict this values by a step (to me). Like you mentioned

they will not be limited to specific events and plugins and will not be masked

So, I would expect it to be printed without explicitly added those variables in your step.
In case if I want to map "global" var to different name in a step, I can use from_variable.

[dvjn]

@zc-devs

That is a good point. That would make this feature much easier to use.

Although, I feel that this kind of implicit behaviour might cause some confusion. It won't always be obvious where an environment variable is coming from.

Also, it could be confusing for a user who has push permissions on a repo, but cannot change the repo settings. This kind of user won't be able to view where the variables are automatically getting set from.

[zc-devs]

much easier to use

That is what I meant 👍

It won't always be obvious where an environment variable is coming from.
user won't be able to view where the variables are automatically getting set from.

From Server or Agent 😄 I can say that about all those variables, that comes from Agent/Server :) (example)

[dvjn]

I understand your point, but I'm still afraid of adding this new implicit behavior.

Can we get another opinion if that's okay?

[dvjn]

Hi maintainers, it would be great to get a review from y'all.

[6543]

uh damed i got requested but it fall under my todo list :?

[6543]

Review process:

• docs
• pipeline engine
• api
• server datastore
• cli
• webui

[6543]

NOTE: write into docs that it is case-sensitive

[6543]

TODO:

• The Variable views should addopt KeyValueEditor (Use KeyValueEditor for DeployPipelinePopup too #4412)
• variable creation via WebUI should work again

[6543]

@dvjn using secrets as template is fine ... but there are a lot of copy-paste errors :/

[pat-s]

It would be empty as you haven't explicitly added those variables in your step either by using variables or from_variable.

If that would be the case, most of the implementation would be meaningless as then one could simply use secrets, as already mentioned.

The real value of this PR IMO lies in the possibility to (easily) set global/org-wide variables which are not sensitive. Without having to repeat them within the YAML again.
For every variable there could/should be an informative log line telling that this variable has been set by a global/org-wide one.

Also, I am not sure if "variables" is the best wording if the values are mapped as env vars in the end. Then, it could directly be named "Environment". This might avoid confusion if there is a difference between a "variable" and an "env var".