Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: the logic refactor with name/instance/plugin #430

Merged
merged 11 commits into from
Apr 23, 2022
Merged

refactor: the logic refactor with name/instance/plugin #430

merged 11 commits into from
Apr 23, 2022

Conversation

daniel-hutao
Copy link
Member

@daniel-hutao daniel-hutao commented Apr 21, 2022 •
edited
Loading

Summary

The logic refactor with name/instance/plugin

Related Issues

See #431 for more info

Test

  • config-gitops.yaml
tools:
- name: github-repo-scaffolding-golang
  instanceID: default
  options:
    owner: [[ githubUsername ]]
    org: ""
    repo: [[ repoName ]]
    branch: [[ defaultBranch ]]
    image_repo: [[ dockerhubUsername ]]/[[ repoName ]]
- name: jira-github-integ
  instanceID: default
  dependsOn: [ "github-repo-scaffolding-golang.default" ]
  options:
    owner: [[ githubUsername ]]
    repo: [[ repoName ]]
    jiraBaseUrl: https://[[ jiraID ]].atlassian.net
    jiraUserEmail: [[ jiraUserEmail ]]
    jiraProjectKey: [[ jiraProjectKey ]]
    branch: main
- name: githubactions-golang
  instanceID: default
  dependsOn: [ "github-repo-scaffolding-golang.default" ]
  options:
    owner: ${{github-repo-scaffolding-golang.default.outputs.owner}}
    org: ""
    repo: ${{github-repo-scaffolding-golang.default.outputs.repo}}
    language:
      name: go
      version: "1.17"
    branch: [[ defaultBranch ]]
    build:
      enable: True
      command: "go build ./..."
    test:
      enable: True
      command: "go test ./..."
      coverage:
        enable: True
        profile: "-race -covermode=atomic"
        output: "coverage.out"
    docker:
      enable: False
      repo: ${{github-repo-scaffolding-golang.default.outputs.repo}}
- name: argocd
  instanceID: default
  options:
    create_namespace: true
    repo:
      name: argo
      url: https://argoproj.github.io/argo-helm
    chart:
      chart_name: argo/argo-cd
      release_name: argocd
      namespace: [[ argocdNameSpace ]]
      wait: true
      timeout: [[ argocdDeployTimeout ]]
      upgradeCRDs: true
- name: argocdapp
  instanceID: default
  dependsOn: ["argocd.default", "github-repo-scaffolding-golang.default"]
  options:
    app:
      name: ${{github-repo-scaffolding-golang.default.outputs.repo}}
      namespace: [[ argocdNameSpace ]]
    destination:
      server: https://kubernetes.default.svc
      namespace: default
    source:
      valuefile: values.yaml
      path: helm/${{github-repo-scaffolding-golang.default.outputs.repo}}
      repoURL: ${{github-repo-scaffolding-golang.default.outputs.repoURL}}
  • gitops-variables.yaml
defaultBranch: main
githubUsername: daniel-hutao
repoName: dtm-test-go
jiraID: merico
jiraUserEmail: tao.hu@merico.dev
jiraProjectKey: DT
dockerhubUsername: exploitht
argocdNameSpace: argocd
argocdDeployTimeout: 10m
  • apply
$ ./dtm apply -f config-gitops.yaml --var-file=gitops-variables.yaml         
2022-04-22 16:29:14 ℹ [INFO]  Apply started.
2022-04-22 16:29:14 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-04-22 16:29:15 ℹ [INFO]  Tool github-repo-scaffolding-golang (default) found in config but doesn't exist in the state, will be created.
2022-04-22 16:29:15 ℹ [INFO]  Tool argocd (default) found in config but doesn't exist in the state, will be created.
2022-04-22 16:29:15 ℹ [INFO]  Tool jira-github-integ (default) found in config but doesn't exist in the state, will be created.
2022-04-22 16:29:15 ℹ [INFO]  Tool githubactions-golang (default) found in config but doesn't exist in the state, will be created.
2022-04-22 16:29:15 ℹ [INFO]  Tool argocdapp (default) found in config but doesn't exist in the state, will be created.
Continue? [y/n]
Enter a value (Default is n): y

2022-04-22 16:29:16 ℹ [INFO]  Start executing the plan.
2022-04-22 16:29:16 ℹ [INFO]  Changes count: 5.
2022-04-22 16:29:16 ℹ [INFO]  -------------------- [  Processing progress: 1/5.  ] --------------------
2022-04-22 16:29:16 ℹ [INFO]  Processing: github-repo-scaffolding-golang (default) -> Create ...
2022-04-22 16:29:20 ℹ [INFO]  The repo dtm-test-go has been created.
2022-04-22 16:29:34 ✔ [SUCCESS]  Tool github-repo-scaffolding-golang(default) Create done.
2022-04-22 16:29:34 ℹ [INFO]  -------------------- [  Processing progress: 2/5.  ] --------------------
2022-04-22 16:29:34 ℹ [INFO]  Processing: argocd (default) -> Create ...
2022-04-22 16:29:36 ℹ [INFO]  Creating or updating helm chart ...
2022/04/22 16:29:41 creating 1 resource(s)
2022/04/22 16:29:41 creating 1 resource(s)
2022/04/22 16:29:41 CRD applicationsets.argoproj.io is already present. Skipping.
2022/04/22 16:29:41 creating 1 resource(s)
2022/04/22 16:29:41 CRD argocdextensions.argoproj.io is already present. Skipping.
2022/04/22 16:29:41 creating 1 resource(s)
2022/04/22 16:29:41 CRD appprojects.argoproj.io is already present. Skipping.
2022/04/22 16:29:41 Clearing discovery cache
2022/04/22 16:29:41 beginning wait for 1 resources with timeout of 1m0s
2022/04/22 16:29:43 creating 43 resource(s)
2022/04/22 16:29:43 beginning wait for 43 resources with timeout of 10m0s
2022/04/22 16:29:44 Deployment is not ready: argocd/argocd-applicationset-controller. 0 out of 1 expected pods are ready
2022/04/22 16:29:46 Deployment is not ready: argocd/argocd-applicationset-controller. 0 out of 1 expected pods are ready
2022/04/22 16:29:48 Deployment is not ready: argocd/argocd-applicationset-controller. 0 out of 1 expected pods are ready
2022/04/22 16:29:50 Deployment is not ready: argocd/argocd-notifications-controller. 0 out of 1 expected pods are ready
2022/04/22 16:29:52 Deployment is not ready: argocd/argocd-repo-server. 0 out of 1 expected pods are ready
2022/04/22 16:29:54 Deployment is not ready: argocd/argocd-repo-server. 0 out of 1 expected pods are ready
2022/04/22 16:29:56 Deployment is not ready: argocd/argocd-repo-server. 0 out of 1 expected pods are ready
2022/04/22 16:29:58 Deployment is not ready: argocd/argocd-repo-server. 0 out of 1 expected pods are ready
2022/04/22 16:30:00 Deployment is not ready: argocd/argocd-repo-server. 0 out of 1 expected pods are ready
2022/04/22 16:30:02 Deployment is not ready: argocd/argocd-repo-server. 0 out of 1 expected pods are ready
2022/04/22 16:30:04 release installed successfully: argocd/argo-cd-4.5.5
2022-04-22 16:30:04 ✔ [SUCCESS]  Tool argocd(default) Create done.
2022-04-22 16:30:04 ℹ [INFO]  -------------------- [  Processing progress: 3/5.  ] --------------------
2022-04-22 16:30:04 ℹ [INFO]  Processing: jira-github-integ (default) -> Create ...
2022-04-22 16:30:05 ℹ [INFO]  Creating GitHub Actions workflow jira-github-integ.yml ...
2022-04-22 16:30:06 ✔ [SUCCESS]  Github Actions workflow jira-github-integ.yml created.
2022-04-22 16:30:08 ✔ [SUCCESS]  Tool jira-github-integ(default) Create done.
2022-04-22 16:30:08 ℹ [INFO]  -------------------- [  Processing progress: 4/5.  ] --------------------
2022-04-22 16:30:08 ℹ [INFO]  Processing: githubactions-golang (default) -> Create ...
2022-04-22 16:30:09 ℹ [INFO]  Creating GitHub Actions workflow pr-builder.yml ...
2022-04-22 16:30:10 ✔ [SUCCESS]  Github Actions workflow pr-builder.yml created.
2022-04-22 16:30:10 ℹ [INFO]  Creating GitHub Actions workflow main-builder.yml ...
2022-04-22 16:30:11 ✔ [SUCCESS]  Github Actions workflow main-builder.yml created.
2022-04-22 16:30:11 ✔ [SUCCESS]  Tool githubactions-golang(default) Create done.
2022-04-22 16:30:11 ℹ [INFO]  -------------------- [  Processing progress: 5/5.  ] --------------------
2022-04-22 16:30:11 ℹ [INFO]  Processing: argocdapp (default) -> Create ...
2022-04-22 16:30:12 ℹ [INFO]  application.argoproj.io/dtm-test-go created
2022-04-22 16:30:12 ✔ [SUCCESS]  Tool argocdapp(default) Create done.
2022-04-22 16:30:12 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-04-22 16:30:12 ✔ [SUCCESS]  All plugins applied successfully.
2022-04-22 16:30:12 ✔ [SUCCESS]  Apply finished.
  • devstream.state
argocd_default:
  name: argocd
  instanceid: default
  dependson: []
  options:
    chart:
      chart_name: argo/argo-cd
      namespace: argocd
      release_name: argocd
      timeout: 10m
      upgradeCRDs: true
      wait: true
    create_namespace: true
    repo:
      name: argo
      url: https://argoproj.github.io/argo-helm
  resource:
    workflows: |
      deployments:
        - name: argocd-applicationset-controller
          ready: true
        - name: argocd-dex-server
          ready: true
        - name: argocd-notifications-controller
          ready: true
        - name: argocd-redis
          ready: true
        - name: argocd-repo-server
          ready: true
        - name: argocd-server
          ready: true
argocdapp_default:
  name: argocdapp
  instanceid: default
  dependson:
    - argocd.default
    - github-repo-scaffolding-golang.default
  options:
    app:
      name: dtm-test-go
      namespace: argocd
    destination:
      namespace: default
      server: https://kubernetes.default.svc
    source:
      path: helm/dtm-test-go
      repoURL: https://github.com/daniel-hutao/dtm-test-go.git
      valuefile: values.yaml
  resource:
    app:
      name: dtm-test-go
      namespace: argocd
    dest:
      namespace: default
      server: https://kubernetes.default.svc
    src:
      path: helm/dtm-test-go
      repoURL: https://github.com/daniel-hutao/dtm-test-go.git
      valueFile: values.yaml
github-repo-scaffolding-golang_default:
  name: github-repo-scaffolding-golang
  instanceid: default
  dependson: []
  options:
    branch: main
    image_repo: exploitht/dtm-test-go
    org: ""
    owner: daniel-hutao
    repo: dtm-test-go
  resource:
    org: ""
    outputs:
      org: ""
      owner: daniel-hutao
      repo: dtm-test-go
      repoURL: https://github.com/daniel-hutao/dtm-test-go.git
    owner: daniel-hutao
    repoName: dtm-test-go
githubactions-golang_default:
  name: githubactions-golang
  instanceid: default
  dependson:
    - github-repo-scaffolding-golang.default
  options:
    branch: main
    build:
      command: go build ./...
      enable: true
    docker:
      enable: false
      repo: dtm-test-go
    language:
      name: go
      version: "1.17"
    org: ""
    owner: daniel-hutao
    repo: dtm-test-go
    test:
      command: go test ./...
      coverage:
        enable: true
        output: coverage.out
        profile: -race -covermode=atomic
      enable: true
  resource:
    workflowDir: /repos/daniel-hutao/dtm-test-go/contents/.github/workflows
jira-github-integ_default:
  name: jira-github-integ
  instanceid: default
  dependson:
    - github-repo-scaffolding-golang.default
  options:
    branch: main
    jiraBaseUrl: https://merico.atlassian.net
    jiraProjectKey: DT
    jiraUserEmail: tao.hu@merico.dev
    owner: daniel-hutao
    repo: dtm-test-go
  resource:
    workflowDir: /repos/daniel-hutao/dtm-test-go/contents/.github/workflows
  • delete
$ ./dtm delete -f config-gitops.yaml --var-file=gitops-variables.yaml
2022-04-22 16:30:47 ℹ [INFO]  Delete started.
2022-04-22 16:30:47 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-04-22 16:30:47 ℹ [INFO]  Tool jira-github-integ (default) will be deleted.
2022-04-22 16:30:47 ℹ [INFO]  Tool githubactions-golang (default) will be deleted.
2022-04-22 16:30:47 ℹ [INFO]  Tool argocdapp (default) will be deleted.
2022-04-22 16:30:47 ℹ [INFO]  Tool github-repo-scaffolding-golang (default) will be deleted.
2022-04-22 16:30:47 ℹ [INFO]  Tool argocd (default) will be deleted.
Continue? [y/n]
Enter a value (Default is n): y

2022-04-22 16:30:49 ℹ [INFO]  Start executing the plan.
2022-04-22 16:30:49 ℹ [INFO]  Changes count: 5.
2022-04-22 16:30:49 ℹ [INFO]  -------------------- [  Processing progress: 1/5.  ] --------------------
2022-04-22 16:30:49 ℹ [INFO]  Processing: jira-github-integ (default) -> Delete ...
2022-04-22 16:30:50 ℹ [INFO]  Deleting GitHub Actions workflow jira-github-integ.yml ...
2022-04-22 16:30:51 ✔ [SUCCESS]  GitHub Actions workflow jira-github-integ.yml removed.
2022-04-22 16:30:51 ℹ [INFO]  Prepare to delete 'jira-github-integ_default' from States.
2022-04-22 16:30:51 ✔ [SUCCESS]  Tool jira-github-integ (default) delete done.
2022-04-22 16:30:51 ℹ [INFO]  -------------------- [  Processing progress: 2/5.  ] --------------------
2022-04-22 16:30:51 ℹ [INFO]  Processing: githubactions-golang (default) -> Delete ...
2022-04-22 16:30:51 ℹ [INFO]  Deleting GitHub Actions workflow pr-builder.yml ...
2022-04-22 16:30:52 ✔ [SUCCESS]  GitHub Actions workflow pr-builder.yml removed.
2022-04-22 16:30:52 ℹ [INFO]  Deleting GitHub Actions workflow main-builder.yml ...
2022-04-22 16:30:53 ✔ [SUCCESS]  GitHub Actions workflow main-builder.yml removed.
2022-04-22 16:30:53 ℹ [INFO]  Prepare to delete 'githubactions-golang_default' from States.
2022-04-22 16:30:53 ✔ [SUCCESS]  Tool githubactions-golang (default) delete done.
2022-04-22 16:30:53 ℹ [INFO]  -------------------- [  Processing progress: 3/5.  ] --------------------
2022-04-22 16:30:53 ℹ [INFO]  Processing: argocdapp (default) -> Delete ...
2022-04-22 16:30:54 ℹ [INFO]  application.argoproj.io "dtm-test-go" deleted
2022-04-22 16:30:54 ℹ [INFO]  Prepare to delete 'argocdapp_default' from States.
2022-04-22 16:30:54 ✔ [SUCCESS]  Tool argocdapp (default) delete done.
2022-04-22 16:30:54 ℹ [INFO]  -------------------- [  Processing progress: 4/5.  ] --------------------
2022-04-22 16:30:54 ℹ [INFO]  Processing: github-repo-scaffolding-golang (default) -> Delete ...
2022-04-22 16:30:55 ✔ [SUCCESS]  GitHub repo dtm-test-go removed.
2022-04-22 16:30:55 ℹ [INFO]  Prepare to delete 'github-repo-scaffolding-golang_default' from States.
2022-04-22 16:30:55 ✔ [SUCCESS]  Tool github-repo-scaffolding-golang (default) delete done.
2022-04-22 16:30:55 ℹ [INFO]  -------------------- [  Processing progress: 5/5.  ] --------------------
2022-04-22 16:30:55 ℹ [INFO]  Processing: argocd (default) -> Delete ...
2022-04-22 16:30:58 ℹ [INFO]  Uninstalling argocd helm chart.
2022/04/22 16:30:58 uninstall: Deleting argocd
2022/04/22 16:30:59 Starting delete for "argocd-redis" Service
2022/04/22 16:30:59 Starting delete for "argocd-application-controller" Service
2022/04/22 16:30:59 Starting delete for "argocd-server" Service
2022/04/22 16:30:59 Starting delete for "argocd-dex-server" Service
2022/04/22 16:30:59 Starting delete for "argocd-applicationset-controller" Service
2022/04/22 16:30:59 Starting delete for "argocd-repo-server" Service
2022/04/22 16:30:59 Starting delete for "argocd-application-controller" StatefulSet
2022/04/22 16:30:59 Starting delete for "argocd-redis" Deployment
2022/04/22 16:30:59 Starting delete for "argocd-repo-server" Deployment
2022/04/22 16:30:59 Starting delete for "argocd-applicationset-controller" Deployment
2022/04/22 16:30:59 Starting delete for "argocd-notifications-controller" Deployment
2022/04/22 16:30:59 Starting delete for "argocd-server" Deployment
2022/04/22 16:30:59 Starting delete for "argocd-dex-server" Deployment
2022/04/22 16:30:59 Starting delete for "argocd-application-controller" RoleBinding
2022/04/22 16:30:59 Starting delete for "argocd-dex-server" RoleBinding
2022/04/22 16:30:59 Starting delete for "argocd-notifications-controller" RoleBinding
2022/04/22 16:30:59 Starting delete for "argocd-server" RoleBinding
2022/04/22 16:30:59 Starting delete for "argocd-applicationset-controller" RoleBinding
2022/04/22 16:30:59 Starting delete for "argocd-repo-server" RoleBinding
2022/04/22 16:30:59 Starting delete for "argocd-applicationset-controller" Role
2022/04/22 16:30:59 Starting delete for "argocd-repo-server" Role
2022/04/22 16:30:59 Starting delete for "argocd-dex-server" Role
2022/04/22 16:30:59 Starting delete for "argocd-notifications-controller" Role
2022/04/22 16:30:59 Starting delete for "argocd-application-controller" Role
2022/04/22 16:30:59 Starting delete for "argocd-server" Role
2022/04/22 16:30:59 Starting delete for "argocd-server" ClusterRoleBinding
2022/04/22 16:30:59 Starting delete for "argocd-application-controller" ClusterRoleBinding
2022/04/22 16:30:59 Starting delete for "argocd-server" ClusterRole
2022/04/22 16:30:59 Starting delete for "argocd-application-controller" ClusterRole
2022/04/22 16:30:59 Starting delete for "argocd-gpg-keys-cm" ConfigMap
2022/04/22 16:30:59 Starting delete for "argocd-cm" ConfigMap
2022/04/22 16:30:59 Starting delete for "argocd-tls-certs-cm" ConfigMap
2022/04/22 16:30:59 Starting delete for "argocd-notifications-controller-cm" ConfigMap
2022/04/22 16:30:59 Starting delete for "argocd-rbac-cm" ConfigMap
2022/04/22 16:30:59 Starting delete for "argocd-ssh-known-hosts-cm" ConfigMap
2022/04/22 16:30:59 Starting delete for "argocd-secret" Secret
2022/04/22 16:30:59 Starting delete for "argocd-notifications-controller-secret" Secret
2022/04/22 16:30:59 Starting delete for "argocd-repo-server" ServiceAccount
2022/04/22 16:30:59 Starting delete for "argocd-application-controller" ServiceAccount
2022/04/22 16:30:59 Starting delete for "argocd-dex-server" ServiceAccount
2022/04/22 16:30:59 Starting delete for "argocd-applicationset-controller" ServiceAccount
2022/04/22 16:30:59 Starting delete for "argocd-server" ServiceAccount
2022/04/22 16:30:59 Starting delete for "argocd-notifications-controller" ServiceAccount
2022/04/22 16:30:59 purge requested for argocd
2022/04/22 16:30:59 release uninstalled, response: &{0x14000136000 }
2022-04-22 16:30:59 ℹ [INFO]  Prepare to delete 'argocd_default' from States.
2022-04-22 16:30:59 ✔ [SUCCESS]  Tool argocd (default) delete done.
2022-04-22 16:30:59 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-04-22 16:30:59 ✔ [SUCCESS]  All plugins deleted successfully.
2022-04-22 16:30:59 ✔ [SUCCESS]  Delete finished.

@daniel-hutao
chore: make 'githubactions' package have a consistent alias 'ga'
56862aa 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao
docs: update gitops example to use the variables
13605c8 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao daniel-hutao self-assigned this Apr 21, 2022
@daniel-hutao daniel-hutao changed the title docs: gitops improvement refactor: the logic refactor with name/instance/plugin Apr 22, 2022
@daniel-hutao
fix: missing gitops-variables
e38f62d 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao
refactor: change "Name", "Plugin" to "InstanceID", "Name" in Config a…
5c542aa 
…nd State

Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao
refactor: update all configs with dtm show config
3968527 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao
refactor: modify all instanceID to "default"
a6c9148 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao
refactor: update the dependsOn at al config
26757c0 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao
refactor: topological sort updated according the dependOn refactor
04696e4 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao
refactor: outputs update according the refactor with outputs reference
f6388fa 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao
fix: some bugfix according test
550049c 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@daniel-hutao daniel-hutao marked this pull request as ready for review April 22, 2022 08:35
@daniel-hutao
Copy link
Member Author

// TODO update docs in website

@daniel-hutao
test: fix ut error according name/instanceID refactor
fb11c2f 
Signed-off-by: Daniel Hu <tao.hu@merico.dev>
@IronCore864 IronCore864 merged commit 353c5ac into devstream-io:main Apr 23, 2022
@IronCore864 IronCore864 deleted the feat-e2e branch April 23, 2022 03:02
@daniel-hutao daniel-hutao mentioned this pull request Apr 23, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@IronCore864 IronCore864 IronCore864 approved these changes

Assignees

@daniel-hutao daniel-hutao

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@daniel-hutao @IronCore864