diff --git a/docs/best-practices/gitops.md b/docs/best-practices/gitops.md index d0e6f1e6a..8311d3946 100644 --- a/docs/best-practices/gitops.md +++ b/docs/best-practices/gitops.md @@ -1,179 +1,362 @@ -# GitOps Toolchain +# GitOps -// TODO(daniel-hutao): to update according helm-installer plugin added. +## 0 Goal -If you are interested in watching a video demo, see the youtube video below: +In this tutorial, we will try to use DevStream to achieve the following: + +1. create a new repository for a Python web application written with [Flask](https://flask.palletsprojects.com/en/2.2.x/); +2. setup basic CI pipelines for the repo we created with GitHub Actions; +3. install [Argo CD](https://argo-cd.readthedocs.io/en/stable/) for GitOps in _an existing Kubernetes cluster_; +4. create an Argo CD application that deploys the web application generated in step 1. + +> Note: +> +> in step 3, Argo CD is installed in an existing Kubernetes cluster. Setting up infrastructure like a Kubernetes cluster is something DevStream chooses not to do. +> +> If you want to follow this tutorial and give it a try yourself but don't know how to get a Kubernetes cluster up and running locally, maybe the following blogs (also from DevStream) would help: +> +> - [Creating a Local Kubernetes Cluster from the Ground Up - a Tutorial of "Kind"](https://blog.devstream.io/posts/creating-a-local-k8s-cluster-with-kind/) +> - [Getting Started with minikube](https://blog.devstream.io/posts/getting-started-with-minikube/) + +--- + +## 1 TL;DR: See It in Action + +If you prefer to see this GitOps doc in action, check out the video demo below: -[https://www.youtube.com/watch?v=q7TK3vFr1kg](https://www.youtube.com/watch?v=q7TK3vFr1kg) +It's from an older version of DevStream with slightly different configuration files, but you get the gist. We will update the video with the latest version of DevStream soon; bear with us for a little while. -For Chinese readers, watch this one instead: +For Chinese readers, watch this one: +However, if you are like us, who prefer to do things hands-on and get their hands dirty, read on, follow the steps, and have a go yourself! +--- -[https://www.bilibili.com/video/BV1W3411P7oW/](https://www.bilibili.com/video/BV1W3411P7oW/) +## 2 Overview -## Plugins needed +DevStream will use the following plugins to achieve the goal described in [Section 0](#0-goal): 1. [repo-scaffolding](../plugins/repo-scaffolding.md) -2. [jira-github](../plugins/jira-github-integ.md) -3. [githubactions-golang](../plugins/githubactions-golang.md) -4. [argocd](../plugins/helm-installer/helm-installer.md) -5. [argocdapp](../plugins/argocdapp.md) +2. [githubactions-golang](../plugins/githubactions-golang.md) +3. [argocd](../plugins/helm-installer/helm-installer.md) (TODO: this doc needs to be updated) +4. [argocdapp](../plugins/argocdapp.md) -The dependencies of these plugins are(`a -> b` means for `a depends on b`): +However, you do not have to worry about these plugins because DevStream will manage them automatically for you. -- `jira-github` -> `repo-scaffolding` -- `githubactions-golang` -> `repo-scaffolding` -- `argocdapp` -> `argocd`, `githubactions-golang` and `repo-scaffolding` +--- -Note: These dependencies are optional; you can use dependency to make sure a certain tool is installed before another. We should use dependency according to the actual usage situation. +## 3 Getting Started: Download DevStream (`dtm`) -## 1 Download DevStream (`dtm`) +Create a temporary working directory for this tutorial: -In your working directory, run: +```bash +mkdir test +cd test/ +``` + +Then, under the newly created directory, execute the following command: -```shell +```bash sh -c "$(curl -fsSL https://raw.githubusercontent.com/devstream-io/devstream/main/hack/install/download.sh)" ``` -This will download the corresponding `dtm` binary to your working directory according to your OS and chip architecture, and grant the binary execution permission. +This script checks your system and downloads the corresponding `dtm` binary. Then the binary will be granted execution permission. -> Optional: you can then move `dtm` to a place which is in your PATH. For example: `mv dtm /usr/local/bin/`. +If you do an `ls`, you can see the binary `dtm` has been downloaded: -_For more details on how to install, see [install dtm](../install.md)._ +```bash +tiexin@mbp ~/work/devstream-io/test $ ls +dtm +``` -## 2 Prepare the Config File +And as a test, try to execute it, and you will get similar output to the following: -Run the following command to generate a template configuration file for gitops `gitops.yaml`. +```bash +tiexin@mbp ~/work/devstream-io/test $ ./dtm +DevStream is an open-source DevOps toolchain manager + +###### ##### +# # ###### # # # # ##### ##### ###### ## # # +# # # # # # # # # # # # ## ## +# # ##### # # ##### # # # ##### # # # ## # +# # # # # # # ##### # ###### # # +# # # # # # # # # # # # # # # +###### ###### ## ##### # # # ###### # # # # + +Usage: + dtm [command] + +Available Commands: + apply Create or update DevOps tools according to DevStream configuration file + completion Generate the autocompletion script for dtm for the specified shell + delete Delete DevOps tools according to DevStream configuration file + destroy Destroy DevOps tools deployment according to DevStream configuration file & state file + develop Develop is used for develop a new plugin + help Help about any command + init Download needed plugins according to the config file + list This command only supports listing plugins now + show Show is used to print plugins' configuration templates or status + upgrade Upgrade dtm to the latest release version + verify Verify DevOps tools according to DevStream config file and state + version Print the version number of DevStream + +Flags: + --debug debug level log + -h, --help help for dtm + +Use "dtm [command] --help" for more information about a command. +``` -```shell -./dtm show config -t gitops > gitops.yaml +> Optional: you can move `dtm` to a directory which is in your $PATH. For example: `mv dtm /usr/local/bin/`. This will allow you to run `dtm` directly without having to prefix it with the dot and slash (`./dtm`). +> +> For more methods on how to install DevStream, see [install dtm](../install.md). + +--- + +## 4 Config File + +Create a file named `config.yaml` and paste the following content into it: + +```yaml +config: + state: + backend: local + options: + stateFile: devstream.state +vars: + githubUser: IronCore864 + dockerUser: ironcore864 + app: helloworld + +tools: +- name: repo-scaffolding + instanceID: myapp + options: + destinationRepo: + owner: [[ githubUser ]] + repo: [[ app ]] + branch: main + repoType: github + sourceRepo: + org: devstream-io + repo: dtm-scaffolding-python + repoType: github + vars: + imageRepo: [[ dockerUser ]]/[[ app ]] +- name: githubactions-python + instanceID: default + dependsOn: [ repo-scaffolding.myapp ] + options: + owner: [[ githubUser ]] + repo: [[ app ]] + language: + name: python + branch: main + docker: + registry: + type: dockerhub + username: [[ dockerUser ]] + repository: [[ app ]] +- name: helm-installer + instanceID: argocd +- name: argocdapp + instanceID: default + dependsOn: [ "helm-installer.argocd", "githubactions-python.default" ] + options: + app: + name: [[ app ]] + namespace: argocd + destination: + server: https://kubernetes.default.svc + namespace: default + source: + valuefile: values.yaml + path: helm/[[ app ]] + repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}} ``` -Then modify the `gitops.yaml` file accordingly. +Then modify the `vars` section in the `config.yaml` file accordingly. Please update the values for `githubUser` and `dockerUser` to your real users. -For me I can set these variables like: +In the example above, I set these vars like the following: | Variable | Example | Note | | ------------------------------ | ----------------- | ------------------------------------------------------------ | -| defaultBranch | main | The branch name you want to use | -| githubUsername | daniel-hutao | It should be case-sensitive here; strictly use your GitHub username | -| repoName | go-webapp | As long as it doesn't exist in your GitHub account and the name is legal | -| dockerhubUsername | exploitht | It should be case-sensitive here; strictly use your DockerHub username | -| jiraID | merico | This is a domain name prefix like merico in https://merico.atlassian.net | -| jiraProjectKey | DT | A descriptive prefix for your project’s issue keys to recognize work from this project | -| jiraUserEmail | tao.hu@merico.dev | The email you use to log in to Jira | -| argocdNameSpace | argocd | The namespace used by ArgoCD | -| argocdDeployTimeout | 10m | How long does ArgoCD deployment timeout | +| githubUser | IronCore864 | case-sensitive, use your GitHub username strictly here | +| dockerUser | ironcore864 | case-sensitive, use your DockerHub username strictly here| +## 5 Environment Variables -These plugins require some environment variables to work, so let's set them: +The following environment variables are required for this to work: ```bash export GITHUB_TOKEN="YOUR_GITHUB_TOKEN_HERE" -export JIRA_API_TOKEN="YOUR_JIRA_API_TOKEN_HERE" export DOCKERHUB_TOKEN="YOUR_DOCKERHUB_TOKEN_HERE" ``` -If you don't know how to create these three tokens, check out: +> Note: +> +> if you don't know how to create these three tokens, check out: +> +> - GITHUB_TOKEN: [Manage API tokens for your Atlassian account](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) +> - DOCKERHUB_TOKEN: [Manage access tokens](https://docs.docker.com/docker-hub/access-tokens/) -- GITHUB_TOKEN: [Manage API tokens for your Atlassian account](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) -- JIRA_API_TOKEN: [Creating a personal access token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) -- DOCKERHUB_TOKEN: [Manage access tokens](https://docs.docker.com/docker-hub/access-tokens/) +--- -## 3 Initialize +## 6 Init Run: ```bash -dtm init -f gitops.yaml +./dtm init +``` + +This downloads the required plugins, according to the config file, automatically. + +You'll get some outputs similar to the following: + +```bash +2022-12-05 17:46:01 ℹ [INFO] Using dir to store plugins. +2022-12-05 17:46:01 ℹ [INFO] -------------------- [ repo-scaffolding-darwin-arm64_0.10.1 ] -------------------- +... (omitted) +... (omitted) +2022-12-05 17:46:51 ✔ [SUCCESS] Initialize finished. ``` -## 4 Apply +--- + +## 7 Apply Run: ```bash -dtm apply -f gitops.yaml +./dtm apply -y ``` -and confirm to continue, then you should see similar output to: +You will see similar outputs as the following: ``` -... -2022-03-11 13:36:11 ✔ [SUCCESS] All plugins applied successfully. -2022-03-11 13:36:11 ✔ [SUCCESS] Apply finished. +2022-12-05 17:49:49 ℹ [INFO] Apply started. +2022-12-05 17:49:49 ℹ [INFO] Using local backend. State file: devstream.state. +2022-12-05 17:49:49 ℹ [INFO] Tool (repo-scaffolding/myapp) found in config but doesn't exist in the state, will be created. +2022-12-05 17:49:49 ℹ [INFO] Tool (helm-installer/argocd) found in config but doesn't exist in the state, will be created. +2022-12-05 17:49:49 ℹ [INFO] Tool (githubactions-python/default) found in config but doesn't exist in the state, will be created. +2022-12-05 17:49:49 ℹ [INFO] Tool (argocdapp/default) found in config but doesn't exist in the state, will be created. +2022-12-05 17:49:49 ℹ [INFO] Start executing the plan. +2022-12-05 17:49:49 ℹ [INFO] Changes count: 4. +... (omitted) +... (omitted) +2022-12-05 17:51:51 ℹ [INFO] -------------------- [ Processing progress: 4/4. ] -------------------- +2022-12-05 17:51:51 ℹ [INFO] Processing: (argocdapp/default) -> Create ... +2022-12-05 17:51:52 ℹ [INFO] application.argoproj.io/helloworld created +2022-12-05 17:51:52 ✔ [SUCCESS] Tool (argocdapp/default) Create done. +2022-12-05 17:51:52 ℹ [INFO] -------------------- [ Processing done. ] -------------------- +2022-12-05 17:51:52 ✔ [SUCCESS] All plugins applied successfully. +2022-12-05 17:51:52 ✔ [SUCCESS] Apply finished. ``` -## 5 Check the Results +--- + +## 8 Check the Results Let's continue to look at the results of the `apply` command. -### 5.1 Repository Scaffolding +### 8.1 Repository -- The repository scaffolding we got looks like this: +The repository is created automatically by DevStream with scaffolding code: ![](gitops/a.png) -### 5.2 Jira-Github Integration +### 8.2 CI Pipelines with GitHub Actions -- How do Jira and Github integrate? Let's create a new issue: +GitHub Actions pipelines are created and executed: ![](gitops/b.png) -- The issue will be renamed automatically like this: +### 8.3 ArgoCD Installation -![](gitops/c.png) +Argo CD is installed in your Kubernetes cluster: -- We can find this auto-synced `Story` in Jira: - -![](gitops/d.png) - -- If we continue to leave a comment on this issue: - -![](gitops/d1.png) - -- The comment will also be automatically synced to Jira: - -![](gitops/e.png) - -### 5.3 GitHub Actions CI for Golang - -- What does CI do here? - -![](gitops/f.png) - -- The CI processes also build an image, and this image is automatically pushed to DockerHub: +```bash +tiexin@mbp ~/work/devstream-io/test $ kubectl get namespaces +NAME STATUS AGE +argocd Active 5m42s +default Active 6m28s +kube-node-lease Active 6m29s +kube-public Active 6m29s +kube-system Active 6m29s +local-path-storage Active 6m25s +tiexin@mbp ~/work/devstream-io/test $ kubectl get pods -n argocd +NAME READY STATUS RESTARTS AGE +argocd-application-controller-0 1/1 Running 0 5m43s +argocd-applicationset-controller-66687659f-dsrtd 1/1 Running 0 5m43s +argocd-dex-server-6944757486-clshl 1/1 Running 0 5m43s +argocd-notifications-controller-7944945879-b9878 1/1 Running 0 5m43s +argocd-redis-7887bbdbbb-xzppj 1/1 Running 0 5m43s +argocd-repo-server-d4f5cc7cb-8gj24 1/1 Running 0 5m43s +argocd-server-5bb75c4bd9-g948r 1/1 Running 0 5m43s +``` -![](gitops/g.png) +### 8.4 Continuous Deployment with ArgoCD -### 5.4 ArgoCD Deployment +The CI pipelines build a Docker image and push it into Dockerhub, and an Argo CD application created by DevStream deploys the app already: -- Of course, the ArgoCD must have been installed as expected. +```bash +tiexin@mbp ~/work/devstream-io/test $ kubectl get deployment -n default +NAME READY UP-TO-DATE AVAILABLE AGE +helloworld 1/1 1 1 5m16s +tiexin@mbp ~/work/devstream-io/test $ kubectl get pods -n default +NAME READY STATUS RESTARTS AGE +helloworld-69b5586b94-wjwd9 1/1 Running 0 5m18s +tiexin@mbp ~/work/devstream-io/test $ kubectl get services -n default +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +helloworld ClusterIP 10.96.73.97 80/TCP 5m27s +kubernetes ClusterIP 10.96.0.1 443/TCP 8m2s +``` -![](gitops/h.png) +If you do a port-forwarding: -### 5.5 ArgoCD Application Deployment +```bash +kubectl port-forward -n default svc/helloworld 8080:8 +``` -- Our code has just been built into an image, at this time the image is automatically deployed to our k8s as a Pod: +And accesses `localhost:8080` in your browser, you can see the deployed app return a "helloworld" to you. Hooray! -![](gitops/i.png) +--- -## 6 Clean Up +## 9 Clean Up Run: ```bash -dtm destroy -f gitops.yaml +./dtm delete -y ``` -and you should see similar output: +And you will get similar outputs to the following: +```bash +2022-12-05 17:59:25 ℹ [INFO] Delete started. +2022-12-05 17:59:26 ℹ [INFO] Using local backend. State file: devstream.state. +2022-12-05 17:59:26 ℹ [INFO] Tool (argocdapp/default) will be deleted. +2022-12-05 17:59:26 ℹ [INFO] Tool (githubactions-python/default) will be deleted. +2022-12-05 17:59:26 ℹ [INFO] Tool (repo-scaffolding/myapp) will be deleted. +2022-12-05 17:59:26 ℹ [INFO] Tool (helm-installer/argocd) will be deleted. +2022-12-05 17:59:26 ℹ [INFO] Start executing the plan. +2022-12-05 17:59:26 ℹ [INFO] Changes count: 4. +... (omitted) +... (omitted) +2022-12-05 17:59:35 ℹ [INFO] -------------------- [ Processing done. ] -------------------- +2022-12-05 17:59:35 ✔ [SUCCESS] All plugins deleted successfully. +2022-12-05 17:59:35 ✔ [SUCCESS] Delete finished. ``` -2022-03-11 13:39:11 ✔ [SUCCESS] All plugins destroyed successfully. -2022-03-11 13:39:11 ✔ [SUCCESS] Destroy finished. + +Then you can delete what we created: + +```bash +cd ../ +rm -rf test/ +rm -rf ~/.devstream/ ``` diff --git a/docs/best-practices/gitops/a.png b/docs/best-practices/gitops/a.png index 9454bd3b3..e3cdabeca 100644 Binary files a/docs/best-practices/gitops/a.png and b/docs/best-practices/gitops/a.png differ diff --git a/docs/best-practices/gitops/b.png b/docs/best-practices/gitops/b.png index d153c4161..c17318970 100644 Binary files a/docs/best-practices/gitops/b.png and b/docs/best-practices/gitops/b.png differ diff --git a/docs/best-practices/gitops/c.png b/docs/best-practices/gitops/c.png deleted file mode 100644 index 0b44caf4e..000000000 Binary files a/docs/best-practices/gitops/c.png and /dev/null differ diff --git a/docs/best-practices/gitops/d.png b/docs/best-practices/gitops/d.png deleted file mode 100644 index a89cdcfa3..000000000 Binary files a/docs/best-practices/gitops/d.png and /dev/null differ diff --git a/docs/best-practices/gitops/d1.png b/docs/best-practices/gitops/d1.png deleted file mode 100644 index ff4b1c849..000000000 Binary files a/docs/best-practices/gitops/d1.png and /dev/null differ diff --git a/docs/best-practices/gitops/e.png b/docs/best-practices/gitops/e.png deleted file mode 100644 index 876f5f71f..000000000 Binary files a/docs/best-practices/gitops/e.png and /dev/null differ diff --git a/docs/best-practices/gitops/f.png b/docs/best-practices/gitops/f.png deleted file mode 100644 index 91fffb91b..000000000 Binary files a/docs/best-practices/gitops/f.png and /dev/null differ diff --git a/docs/best-practices/gitops/g.png b/docs/best-practices/gitops/g.png deleted file mode 100644 index 638bc0445..000000000 Binary files a/docs/best-practices/gitops/g.png and /dev/null differ diff --git a/docs/best-practices/gitops/h.png b/docs/best-practices/gitops/h.png deleted file mode 100644 index 7b9cba87b..000000000 Binary files a/docs/best-practices/gitops/h.png and /dev/null differ diff --git a/docs/best-practices/gitops/i.png b/docs/best-practices/gitops/i.png deleted file mode 100644 index aae94aad1..000000000 Binary files a/docs/best-practices/gitops/i.png and /dev/null differ