docs: udpate core concepts and config for app centric config

docs/commands/verify.md

@@ -23,7 +23,7 @@ If not, it tries to give you a hint that maybe you forgot to run `dtm init` firs
 
 ## 4 Config / State / ResourceStatus
 
-For definitions of _Config_, _State_, and _ResourceStatus_, see [Core Concepts](../core-concepts/core-concepts.md).
+For definitions of _Config_, _State_, and _ResourceStatus_, see [Core Concepts](../core-concepts/overview.md).
 
 `dtm verify` tries to see if the _Config_ matches the _State_ and the _Resource_ or not. If not, it tells you what exactly is not the same, and what would happen if you run `dtm apply`.
 

docs/commands/verify.zh.md

@@ -23,7 +23,7 @@
 
 ## 4 Config / State / ResourceStatus
 
-关于 _Config_ 、 _State_ 和 _ResourceStatus_ 的定义，见[核心概念](../core-concepts/core-concepts.zh.md)。
+关于 _Config_ 、 _State_ 和 _ResourceStatus_ 的定义，见[核心概念](../core-concepts/overview.zh.md)。
 
 `dtm verify` 尝试判断 _Config_ 是否与 _State_ 和 _ResourceStatus_ 匹配。如果不匹配，它会告诉你到底是什么不一样，以及如果你运行 `dtm apply` 会发生什么。
 

docs/core-concepts/config.md

@@ -1,42 +1,56 @@
-# DevStream Configuration
+# Config
 
-DevStream uses YAML files to describe your DevOps toolchain configuration.
+Now let's have a look at some config examples.
 
-## Main Config File
+TL;DR: see [the last section](./config.md#8-min-config) of this documentation.
 
-By default, `dtm` tries to use `./config.yaml` (under your current directory) as the main config.
+---
 
-The main config contains three sections:
+## 1 The Main Config File
 
-- `varFile`: the path/to/your/variables file
-- `toolFile`: the path/to/your/tools configuration file
-- `pluginDir`: the path/to/your/plugins directory, default: `~/.devstream/plugins`, or use `-d` flag to specify a directory
-- `state`: configuration of DevStream state. For example, 
+As mentioned in the overview section, the main config contains many settings, like:
 
-### Example Main Config File
+- `pluginDir`
+- `state`
+- `varFile`
+- `toolFile`, see [here](./tools-apps.md).
+- `appFile`
+- `templateFile`
 
-See the `config.yaml` example below:
+Here's an example of the main config file:
 
 ```yaml
-varFile: variables.yaml
-
-toolFile: tools.yaml
-
-pluginDir: /usr/local/devstream/plugins # optional
-
-state:
+varFile: "./variables.yaml"
+toolFile: "./tools.yaml"
+pluginDir: ""    # empty, use the default value: ~/.devstream/plugins
+appFile: "./apps.yaml"
+templateFile: "./templates.yaml"
+state:           # state config, backend can be "local", "s3", or "k8s"
   backend: local
   options:
     stateFile: devstream.state
 ```
 
-### Variables File
+By default, `dtm` tries to use `./config.yaml` (under your working directory) as the main config.
+
+You can override the default value with `-f` or `--config-file`. Examples:
+
+```shell
+dtm apply -f path/to/your/config.yaml
+dtm apply --config-file path/to/your/config.yaml
+```
+
+No default values are provided for `varFile` and `toolFile`. If `varFile` isn't specified in the main config, `dtm` will not use any var files, even if there is already a file named `variables.yaml` under the current directory. Similarly, if `toolFile` isn't specified in the main config, `dtm` will throw an error, even if there is a `tools.yaml` file under the current directory.
 
-The var file is a YAML file containing key-value pairs.
+---
 
-_At the moment, nested/composite values (for example, the value is a list/dictionary) are not supported yet._
+## 2 Variables File
 
-See the `variables.yaml` example below:
+To not repeat yourself (Don't repeat yourself, DRY, see [here](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself),) we can define some variables in a var file and use the vars in the config file.
+
+The var file is a YAML file containing key-value pairs, which can be nested/composite values (for example, the value is a list/dictionary).
+
+Example:
 
 ```yaml
 githubUsername: daniel-hutao
@@ -45,9 +59,60 @@ defaultBranch: main
 dockerhubUsername: exploitht
 ```
 
-### Tool File
+To use these variables in a config file, use the following syntax:
+
+```yaml
+[[ varNameHere ]]
+```
+
+---
+
+## 3 State Config
+
+The `state` section specifies where to store the DevStream state.
+
+### 3.1 Local
+
+```yaml
+# part of the main config file
+state:
+  backend: local
+  options:
+    stateFile: devstream.state
+```
+
+_Note: The `stateFile` under the `options` section is mandatory for the local backend._
+
+### 3.2 Kubernetes
+
+```yaml
+# part of the main config file
+state:
+  backend: k8s
+  options:
+    namespace: devstream # optional, default is "devstream", will be created if not exists
+    configmap: state     # optional, default is "state", will be created if not exists
+```
+
+### 3.3 S3
 
-Tool file contains a list of tools.
+```yaml
+# part of the main config file
+state:
+  backend: s3
+  options:
+    bucket: devstream-remote-state
+    region: ap-southeast-1
+    key: devstream.state
+```
+
+_Note: the `bucket`, `region`, and `key` under the `options` section are all mandatory fields for the s3 backend._
+
+---
+
+## 4 Tool File
+
+The tool file contains a list of tools.
 
 The tool file contains:
 
@@ -61,7 +126,7 @@ The tool file contains:
 - Each dictionary (tool) has an optional field which is `options`, which in turn is a dictionary containing parameters for that specific plugin. For plugins' parameters, see the "plugins" section of this document.
 - Each directory (tool) has an optional field which is `dependsOn`. Continue reading for detail about dependencies.
 
-See the `tools.yaml` example down below:
+Example:
 
 ```yaml
 tools:
@@ -92,31 +157,186 @@ tools:
     branch: main
 ```
 
-### State
 
-The `state` section specifies where to store DevStream state. As of now (v0.5.0), we only support local backend.
+If you want tool A to be installed before tool B, you can let tool B depend on tool A.
+
+The syntax for dependency is:
+
+```yaml
+dependsOn: [ "ToolName.ToolInstanceID" ]
+```
+
+Since `dependsOn` is a list, a tool can have multiple dependencies:
+
+```
+dependsOn: [ "ToolName1.ToolInstanceID1", "ToolName2.ToolInstanceID2", "..." ]
+```
 
-From v0.6.0 on, we will support both "local" and "s3" backend store the DevStream state.
+The example config above shows that the second tool depends on the first.
 
-Read the section [The State Section in the Main Config](./stateconfig.md) for more details.
+---
 
-## Default Values
+## 5 App File
 
-By default, `dtm` uses `config.yaml` as the main config file.
+A full app file looks like the following:
 
-### Specifying a Main Config File Explicitly 
+```yaml
+apps:
+- name: service-A
+  spec:
+    language: python
+    framework: django
+  repo:
+    scmType: github
+    owner: devstream-io
+    org: devstream-io # either owner or org must exist
+    name: service-A   # defaults to "app.name"
+    url: github.com/devstream-io/repo-name   # optional. if exists, no need for the scm/owner/org/name sections
+    apiURL: gitlab.com/some/path/to/your/api # optional, in case of self-managed git
+  repoTemplate:   # optional. if repoTemplate isn't empty, create repo according to the template
+    scmType: github
+    owner: devstream-io
+    org: devstream-io # either owner or org must exist
+    name: dtm-scaffolding-golang
+    url: github.com/devstream-io/repo-name   # optional. if exists, no need for the scm/owner/org/name sections
+  ci:
+  - type: template              # type template means it's a reference to the pipeline template. read the next section.
+    templateName: ci-pipeline-1
+    options: # optional, used to override defaults in the template
+    vars:    # optional, key/values to be passed to the template
+      key: value
+  cd:
+  - type: template              # type template means it's a reference to the pipeline template. read the next section.
+    templateName: cd-pipeline-1
+    options: # optional, used to override defaults in the template
+    vars:    # optional, key/values to be passed to the template
+      key: value
+```
 
-You can override the default value with `-f` or `--config-file`. Examples:
+Since many key/values have default values, it's possible to use the following minimum config for the apps section:
 
-```shell
-dtm apply -f path/to/your/config.yaml
-dtm apply --config-file path/to/your/config.yaml
+```yaml
+apps:
+- name: service-B # minimum config demo
+  spec:
+    language: python
+    framework: django
+  repo:
+    url: github.com/devstream-io/repo-name
+  repoTemplate:
+    url: github.com/devstream-io/repo-name
+  ci:
+    - type: githubactions
+  cd:
+    - type: argocdapp
+```
+
+---
+
+## 6 Template File
+
+You can define some pipeline templates in the template file, like the following:
+
+```yaml
+pipelineTemplates:
+- name: ci-pipeline-1
+  type: githubactions # corresponds to a DevStream plugin
+  options:
+    branch: main      # optional
+    docker:
+      registry:
+        type: dockerhub
+        username: [[ dockerUser ]]
+        repository: [[ app ]]
+- name: cd-pipeline-1
+  type: argocdapp
+  options:
+    app:
+      namespace: argocd
+    destination:
+      server: https://kubernetes.default.svc
+      namespace: default
+    source:
+      valuefile: values.yaml
+      path: helm/[[ app ]]
+      repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}}
+```
+
+Then, you can refer to these pipeline templates in the apps file.
+
+---
+
+## 7 Putting It All Together
+
+Here's a complete example.
+
+Main config file:
+
+```yaml
+toolFile: "./tools.yaml"
+appFile: "./apps.yaml"
+templateFile: "./templates.yaml"
+state:
+  backend: local
+  options:
+    stateFile: devstream.state
+```
+
+Tool file:
+
+```yaml
+tools:
+- name: argocd
+  instanceID: default
 ```
 
-### No Defaults for varFile and toolFile
+App file:
 
-For `varFile` and `toolFile`, no default values are provided.
+```yaml
+apps:
+- name: service-B
+  spec:
+    language: python
+    framework: django
+  repo:
+    url: github.com/devstream-io/repo-name
+  repoTemplate:
+    url: github.com/devstream-io/dtm-scaffolding-python
+  ci:
+    - type: githubactions
+  cd:
+    - type: argocdapp
+```
+
+In this example, we didn't use a var file, or template file, since they are optional and we don't need them in this example.
 
-If `varFile` isn't specified in the main config, `dtm` will not use any var files, even if there is already a file named `variables.yaml` under the current directory.
+## 8 Min Config
 
-Similarly, if `toolFile` isn't specified in the main config, `dtm` will throw an error, even if there is a `tools.yaml` file under the current directory.
+Alternatively, you can merge multiple files into one to achieve a minimum config:
+
+```yaml
+---
+state:
+  backend: local
+  options:
+    stateFile: devstream.state
+---
+tools:
+- name: argocd
+  instanceID: default
+---
+serviceName: test-service
+apps:
+- name: [[ serviceName ]]
+  spec:
+    language: python
+    framework: django
+  repo:
+    url: github.com/devstream-io/[[ serviceName ]]
+  repoTemplate:
+    url: github.com/devstream-io/dtm-scaffolding-python
+  ci:
+    - type: githubactions
+  cd:
+    - type: argocdapp
+```

docs/core-concepts/config.zh.md

@@ -1,5 +1,7 @@
 # DevStream配置
 
+TODO: 需要跟英文文档同步。请暂且阅读英文文档。
+
 DevStream使用YAML文件来描述你的DevOps工具链配置。
 
 ## 主配置文件
@@ -95,7 +97,7 @@ tools:
 
 从v0.6.0开始，我们将支持`local`和`s3`两种存储。
 
-更多状态存储细节请参见[DevStream状态存储](./stateconfig.md)
+更多状态存储细节请参见[DevStream状态存储](./state.zh.md)
 
 ## 默认值