Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An internal CI/CD system compatible with Github actions workflow yaml syntax, action yaml syntax and most action plugins #13539

Closed
13 of 20 tasks
lunny opened this issue Nov 13, 2020 · 96 comments · Fixed by #21937
Closed
13 of 20 tasks
Labels
topic/gitea-actions related to the actions of Gitea type/proposal The new feature has not been accepted yet but needs to be discussed first. type/summary This issue aggregates a bunch of other issues

Comments

@lunny
Copy link
Member

lunny commented Nov 13, 2020

Internal CI/CD system

I propose to implement an internal CI/CD system that is compatible with Github workflows yaml syntax, action yaml syntax, and most action plugins. And this is in fact not only a CI/CD system, but you can also choose to not check out the code for some events for example, when an issue is posted, you want to reply to them, then you don't need to check out the code. You can just reply to that issue. So the sub-system name is botsactions, if you have a better name than that, please let me know.

Pros

  • Migrated repositories from Github could easily reuse the actions workflow files
  • Plugins could be copied from https://github.com/actions or https://gitea.com/actions to your self-hosted Gitea instances or any git service repository address
  • Transparent for the CI/CD process. You can check out all workflow files and plugin codes you are using.
  • Bring version management for your CI/CD scripts and plugins
  • Customize your image. Github's image for ubuntu-latest is very big but Gitea now will use node:16-buster as the default image which is about 150MB and of course you could change it yourself.

Cons

  • Images are maybe very big for those small devices and have to be changed when you have new dependencies.
  • Many Gitea users are familiar with Drone/Woodpecker
  • More feedbacks are welcome

Reference documentation

At the moment, we lack documents, but if you want to know more details about what Github Actions looks like, please visit https://docs.github.com/en/actions.

Names chosen

Different from Github's related sub-system's name, we call the plugins action or actions. One action in fact is a repository with a file named action.yml in the root, the repository is hosted in any git service server which could be gitea.com, your self-hosted Gitea instance, and even github.com. There is a default configuration but we should also support absolute git URL as a plugin repository.

Task summary

There are many things need to do

Although we will have an internal CI/CD system Gitea will still be open to integrating external systems and will provide more features to make the integration smooth. The below tasks do not belong to the internal CI/CD system but are very important for integrating the external CI/CD system

@a1012112796
Copy link
Member

a1012112796 commented Nov 15, 2020

tips about api for ci tools to repot detail message:

  1. lint messages, can show them like code comment (but can marked as wrong by manager)
    the stype of data that provide by api
{
    commit: xxxxxxx
    path: "xxxxx/xxxxx",
    line: -1,
    type:  bug/ security/ codesmell/ info,
    detail: xxxxxxx,
   provider: xxxxx,
}

example in gitee
image

  1. code coverage data like codecov
{
    commit: xxxxxxx
    path: "xxxxx/xxxxx",
    group: golang,    // only data in same group will be calculate together
   provider: xxxxx,
   start_line: xx,
   start_cols: xx,
   end_line: xx,
  end_cols: xx,
  matched: 0 -- not, .>1 -- yes
}

do some calculate and show some usefull data on ui.

  1. detal log in ci steps, should show them in an individual page like gh actions.
    then user can design a sample ci tools which not need it's own ui :)
    style:
{
    commit: xxxxxxx
    step_name: aaa,
    parent_step_name: bbb,
   previous_step_name: ccc,
    log: xxxxx, or link to log data
   status: pending or runing or success ....
}

example on gh
image

@lunny lunny added the type/proposal The new feature has not been accepted yet but needs to be discussed first. label Nov 15, 2020
@Be-ing
Copy link

Be-ing commented Mar 12, 2021

Support github actions files and allow external agent to execute the action via Gitea's API and returned result as above results.

Can you clarify what this will entail? Would this mean GitHub Actions could run on a github.com mirror of a repo and the results would appear on a Gitea server?

@lunny
Copy link
Member Author

lunny commented Mar 14, 2021

That means Github actions could be running on Gitea(with this CI feature) servers(actually on user's agent, Gitea will only a coordinator and result saver).
Then mirrors/migrations of repositories from github/github enterprise could run the ci directly and no(or small) modification needed.

@moqmar
Copy link
Contributor

moqmar commented Mar 30, 2021

Would this be implemented by explicitly checking for a .github directory and sending those repos to the agent automatically on push using something like a global webhook? Because I think it would be great to not limit this to just GitHub Actions, but rather have a "global webhook" for pushes to an agent that can then decide what to do with that push - either using the GitHub Actions definitions, or for example running other CI implementations.

@lunny
Copy link
Member Author

lunny commented Mar 30, 2021

@moqmar Global webhooks have been implemented. The issue is to be compatible with actions for those migrations from github.

@ChristopherHX
Copy link
Contributor

ChristopherHX commented May 4, 2021

I have been working on an external webhook endpoint for github actions self-hosted runners to use with my gitea instance or locally on any pc. This is basically a fork of the runner to be able to reuse their code for a missing aspnetcore server. During my tests I was able to use gitea almost like a ghes instance, some actions like actions/checkout@v2 have had minor incompatibilities with authentication.

I would be glad to see endpoints for CI logs and have an integrated UI. Currently I have a basic react UI to display my logs, added as an external tab to gitea.

Experimental Usage
Download and extract https://github.com/ChristopherHX/runner.server/releases/latest for your system

Setup Gitea as an ghes endpoint, file bin/appsettings.json, change the url from http://localhost:5000 to a different http url / port

  "Runner.Server": {
    "GitServerUrl": "https://gitea:3042",
    "GitApiServerUrl": "https://gitea:3042/api/v1",
    "GitGraphQlServerUrl": null,
    "GITHUB_TOKEN": "Add your personal access token here",
    "ActionDownloadUrls": [
      {
        "TarballUrl": "https://gitea:3042/{0}/archive/{1}.tar.gz",
        "ZipballUrl": "https://gitea:3042/{0}/archive/{1}.zip"
      }
    ]
  }

Add <url of Runner.Server>/runner/server/_apis/v1/Message (http://localhost:5000/runner/server/_apis/v1/Message) as a webhook (select gitea) for your repo, to enable actions.

Setup https and use openid connect with gitea (Optional for Security, only working if the github actions server is using https)

  • Create an oauth2 app (User settings)
  • You will need to add the clientid and secret into bin/appsettings.json
  • You will need a pem certificate pair (cert.pem (only a single certificate will work, no cert chain) , key.pem)
"Kestrel": {
    "Endpoints": {
      "HttpsFromPem": {
        "Url": "https://*:5001",
        "Certificate": {
          "Path": "./cert.pem",
          "KeyPath": "./key.pem"
        }
      }
    }
  },
  "ClientId": "ClientId of your Oauth app",
  "ClientSecret": "Client secret of your Oauth app",
  "Authority": "https://gitea:3042",

Add <url of Runner.Server>/signin-oidc (https://localhost:5001/signin-oidc) as redirect url for the OAuth app in gitea.

Start bin/Runner.Server(.exe)
Open <url of Runner.Server> (http://localhost:5000, https://localhost:5001) to see if a job was received

Configure self-hosted runners, https://github.com/ChristopherHX/runner.server#advanced-usage

EDIT 2021/10/26
I added a ( temporary ) public test instance of runner.server connected to try.gitea.io https://try.gitea.io/ChristopherHX/actions-on-gitea/commit/7ac850a3a96a7c60417c1306f50d49ea08d2e7d7

Press rerun workflow to see in progress job runs
EDIT 2021/02/02
Fix openid sample config to actually have working https with pem

@techknowlogick
Copy link
Member

@ChristopherHX whoa! That's amazing!!

@Bobby99as
Copy link

This seems nice

@lunny lunny added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Oct 28, 2022
@lunny lunny added this to the 1.19.0 milestone Oct 28, 2022
@lunny lunny changed the title Support github actions An internal CI/CD system compatible with Github actions yaml syntax and most action plugins Nov 17, 2022
@lunny lunny changed the title An internal CI/CD system compatible with Github actions yaml syntax and most action plugins An internal CI/CD system compatible with Github workflow yaml syntax, action yaml syntax and most action plugins Nov 17, 2022
@lunny lunny added type/summary This issue aggregates a bunch of other issues topic/gitea-actions related to the actions of Gitea and removed type/feature Completely new functionality. Can only be merged if feature freeze is not active. labels Nov 17, 2022
@lunny lunny changed the title An internal CI/CD system compatible with Github workflow yaml syntax, action yaml syntax and most action plugins An internal CI/CD system compatible with Github actions workflow yaml syntax, action yaml syntax and most action plugins Nov 17, 2022
@kbingham
Copy link

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

Being able to store the configuration in a custom branch would help a lot for adding automation to open source projects that don't want to have specific CI files added to the core repo. The linux kernel for instance.

@lunny
Copy link
Member Author

lunny commented Mar 30, 2023

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

Being able to store the configuration in a custom branch would help a lot for adding automation to open source projects that don't want to have specific CI files added to the core repo. The linux kernel for instance.

Maybe we can have some options:

  1. .github/workflows or .gitea/workflows
  2. <repo_name>.actions.git like <repo_name>.wiki.git
  3. A setting of the repository to store multiple workflow files.
  4. A global workflows feature, which you can create a workflow to includes files and repositories.

@siegenthalerroger
Copy link

I actually really like the idea of having a seperate repo analogous to the wiki repository for this. Would be interesting to explore other functionality that could be implemented in this way.

@kbingham
Copy link

kbingham commented Apr 1, 2023

Yup now I've been exploring more, I can see a separated repo for the workflows could be a very helpful way of doing this too. Then multiple repos (mirrors, forks, or separate repos for separate developers) can point to common workflow actions.

@dsseng
Copy link
Contributor

dsseng commented Apr 8, 2023

Maybe after artifact upload is done, basic code intelligence (not as powerful as SourceGraph, but just goto definitions/references + browse file symbols) could be implemented. After finishing builds/tests the action might do something like rust-analyzer lsif . and upload it as a special artifact Gitea will build navigation from. This approach gives the most detailed code navigation, just as in IDE using the same language server as used to generate LSIF. See https://lsif.dev/ for more details regarding the format.

Didn't open a new issue because of dependency on artifacts

@prologic
Copy link

Are there any docs on built-in actions in 1.19.0? Even just a quick bullet point here in this thread? 🙏

@prologic
Copy link

Answering my own question, getting this up and running with Gitea v1.19.0 (from upstream Docker images) was pretty straight forward:

  1. Add the following to your app.ini config (since if you're running an older version, it won't be there by default and disable by default?):
[actions]
ENABLED = true
  1. Restart Gitea
  2. Go to Site Administration -> Runners
  3. Create new Runner and Copy the token
  4. Setup and register a Runner
$ git clone https://gitea.com/gitea/act_runner.git
$ cd act_runner
$ make build
$ ./act_runner register
# follow the instructions
$ ./act_runner daemon

That's it! 🥳 Now just add workflows in .gitea/workflows/build.yml which use the same syntax as Github actions? (I guess so since the runner being used is the act_runner).

@prologic
Copy link

Happy to continue beta testing this (I also run an older Drone CI setup too!) and provide feedback.

One thing that springs to mind (which I'm hoping one of the developers can answer):

  • How do I set-up Gitea + actions in a way where I can spawn runners on demand? (like Github does)

Basically, instead of having a static pool of runners (however I choose to run them), it would be nice to have a webhook (globally) that I can listen on and spawn a new runner on demand (say in a virtual machine or similar). Thanks! 🙏

@aminify
Copy link

aminify commented Apr 18, 2023

Apparently in version 19.0.1 we should put something different in the app.ini file. Can someone help with that?
I put the below in it and actions weren't enabled. I used docker image by the way... I don't know if that's the source of issue. Really poor documentation so far...

[repo.actions]
ENABLED = true

@jimafisk
Copy link

jimafisk commented Apr 18, 2023

@aminify I think it's just:

[actions]
ENABLED = true

https://docs.gitea.io/en-us/config-cheat-sheet/#actions-actions

@delvh
Copy link
Member

delvh commented Apr 18, 2023

No, the cheatsheet is still true.
You simply misunderstood something here:
the key DEFAULT_REPO_UNITS/DEFAULT_FORK_REPO_UNITS in the [repository] section used the value actions instead of repo.actions.
The section to enable actions is [actions].

@lunny
Copy link
Member Author

lunny commented Apr 18, 2023

All units configuration should use repo.actions, but the ini section is [actions].

@RynoM
Copy link

RynoM commented Apr 18, 2023

  • How do I set-up Gitea + actions in a way where I can spawn runners on demand? (like Github does)

This would be nice! I've been tinkering with using it to deploy/restart my docker services. Would be nice if it could just do dynamic generation / access from outside for runner tokens. That way the container is more ephemeral.

Also its probably been said before, but an environment variable for enabling actions would be nice, instead of having to update app.ini.

@aminify
Copy link

aminify commented Apr 19, 2023

I have setup the runner and everything and I have a docker build command to run in my action. Unfortunately I get /var/run/act/workflow/1: line 2: docker: command not found. Anyone knows the best way to enable docker inside the runner?

@panekj
Copy link

panekj commented Apr 19, 2023

I have setup the runner and everything and I have a docker build command to run in my action. Unfortunately I get /var/run/act/workflow/1: line 2: docker: command not found. Anyone knows the best way to enable docker inside the runner?

You need to install it or use a docker image runner that has it installed

@aminify
Copy link

aminify commented Apr 20, 2023

thanks @panekj
for others facing the same issue. I managed to fix this docker: command not found issue using ubuntu-latest:docker://catthehacker/ubuntu:act-22.04 as label argument for act-runner (in register phase)

@sorenisanerd
Copy link

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

@prologic
Copy link

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

Wait?! Why are you logging secrets in the first place? 🤔

@vquie
Copy link

vquie commented Apr 21, 2023

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

I can confirm this. I use the 1password action to retrieve passwords. They are logged in all their beauty. Their code says add-mask though.

@JakobDev
Copy link
Contributor

Wait?! Why are you logging secrets in the first place?

There are Programs who print API Keys for debug reasons with no Way to disable it. It also possible, that this may be leaked in a error message when something crashes, so it's generally a good Idea to mask this, just in case.

@kbingham
Copy link

Yup now I've been exploring more, I can see a separated repo for the workflows could be a very helpful way of doing this too. Then multiple repos (mirrors, forks, or separate repos for separate developers) can point to common workflow actions.

FWIW - Today I discovered gitlab can do this and it's all documented here:

@prologic
Copy link

Wait?! Why are you logging secrets in the first place?

There are Programs who print API Keys for debug reasons with no Way to disable it. It also possible, that this may be leaked in a error message when something crashes, so it's generally a good Idea to mask this, just in case.

Those programs should be loaded up on a rocket and shot into sun! 😆 talk about security law practice 🤦‍♂️

@delvh
Copy link
Member

delvh commented Apr 30, 2023

I wonder how much more we want to abuse this one issue?
As far as I'm concerned the basic functionality exists, and the remains should be extracted into separate issues.
I don't think it's good that this issue is still open despite the main functionality being implemented already.
Issues should focus on one specific thing, otherwise they'll be pretty much never closed.

techknowlogick added a commit that referenced this issue May 5, 2023
A new documentation section for Gitea Actions.

Some content comes from:

- [Hacking on Gitea
Actions](https://blog.gitea.io/2023/03/hacking-on-gitea-actions/)
- The README of [act_runner](https://gitea.com/gitea/act_runner)
- @ChristopherHX's excellent overview of the differences between Gitea
Actions and GitHub Actions in [this
comment](#13539 (comment)).

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
topic/gitea-actions related to the actions of Gitea type/proposal The new feature has not been accepted yet but needs to be discussed first. type/summary This issue aggregates a bunch of other issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.