Docs updates for v0.7 (#159)

* Fixing deactivate route bug in webhook gateway * Fixing deactivate route bug in storage gateway * fixing event source deactivation -> activation bug * Started adding support for NATS streaming * Started working on supporting nats standard and streaming for communication between gateway and sensor * Changing nats subscription to queue subscription * Fire K8s event on nats connection/subscription updates * Updating gateway and sensor controller validate method * Fixing nats-streaming issue * Fix storage grid channel issue * Removing premature return in transformer * Renaming dispatchProtocol to eventProtocol * Moving common structs in apis package * Adding deepcopy and proto files for common api * making trigger errors more explicit * Updating docs for v0.7 * Updating docs for v0.7 * Updating docs for v0.7 * Updating README * Updating gateway installation docs * Updating makefile * Updating makefile
argoproj · Jan 29, 2019 · 349ff86 · 349ff86
1 parent 3461eda
commit 349ff86
Show file tree

Hide file tree

Showing 110 changed files with 1,908 additions and 2,921 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,13 @@
 # Changelog
 
+## v0.7
++ Refactor gateways #147
++ Fixed sensor error state recovery bug #145
++ Ability to add annotations on sensor and gateway pods #143
++ Github gateway
++ Added support for NATS standard and streaming as communication protocol between gateway
+  and sensor #99
+
 ## v0.6
 + Gitlab Gateway #120
 + If sensor is repeatable then deploy it as deployment else job #109

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -48,7 +48,7 @@ make all
 Follow the rest of the [quickstart guide](docs/quickstart.md) to install components.
 
 ## Changing Types
-If you're making a change to the `pkg/apis/sensor/v1alpha1` package, please ensure you re-run the K8 code-generator scripts found in the `/hack` folder. First, ensure you have the `generate-groups.sh` script at the path: `vendor/k8s.io/code-generator/`. Next run the following commands in order:
+If you're making a change to the `pkg/apis`  package, please ensure you re-run the K8 code-generator scripts found in the `/hack` folder. First, ensure you have the `generate-groups.sh` script at the path: `vendor/k8s.io/code-generator/`. Next run the following commands in order:
 ```
 $ make codegen
 ```
diff --git a/Makefile b/Makefile
@@ -194,33 +194,33 @@ mqtt-image: mqtt-linux
 
 # Custom gateways
 storage-grid:
-	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/storagegrid-gateway ./gateways/custom/storagegrid/cmd
+	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/storagegrid-gateway ./gateways/community/storagegrid/cmd
 
 storage-grid-linux:
 	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make storage-grid
 
 storage-grid-image: storage-grid-linux
-	 docker build -t $(IMAGE_PREFIX)storage-grid-gateway:$(IMAGE_TAG) -f ./gateways/custom/storagegrid/Dockerfile .
+	 docker build -t $(IMAGE_PREFIX)storage-grid-gateway:$(IMAGE_TAG) -f ./gateways/community/storagegrid/Dockerfile .
 	@if [ "$(DOCKER_PUSH)" = "true" ] ; then  docker push $(IMAGE_PREFIX)storage-grid-gateway:$(IMAGE_TAG) ; fi
 
 gitlab:
-	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/gitlab-gateway ./gateways/custom/gitlab/cmd
+	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/gitlab-gateway ./gateways/community/gitlab/cmd
 
 gitlab-linux:
 	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make gitlab
 
 gitlab-image: gitlab-linux
-	 docker build -t $(IMAGE_PREFIX)gitlab-gateway:$(IMAGE_TAG) -f ./gateways/custom/gitlab/Dockerfile .
+	 docker build -t $(IMAGE_PREFIX)gitlab-gateway:$(IMAGE_TAG) -f ./gateways/community/gitlab/Dockerfile .
 	@if [ "$(DOCKER_PUSH)" = "true" ] ; then  docker push $(IMAGE_PREFIX)gitlab-gateway:$(IMAGE_TAG) ; fi
 
 github:
-	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/github-gateway ./gateways/custom/github/cmd
+	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/github-gateway ./gateways/community/github/cmd
 
 github-linux:
 	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make github
 
 github-image: github-linux
-	 docker build -t $(IMAGE_PREFIX)github-gateway:$(IMAGE_TAG) -f ./gateways/custom/github/Dockerfile .
+	 docker build -t $(IMAGE_PREFIX)github-gateway:$(IMAGE_TAG) -f ./gateways/community/github/Dockerfile .
 	@if [ "$(DOCKER_PUSH)" = "true" ] ; then  docker push $(IMAGE_PREFIX)github-gateway:$(IMAGE_TAG) ; fi
 
 test:

diff --git a/README.md b/README.md
@@ -3,14 +3,14 @@
 [![Go Report Card](https://goreportcard.com/badge/github.com/argoproj/argo-events)](https://goreportcard.com/report/github.com/argoproj/argo-events)
 
 ## What is Argo Events?
-Argo Events is an open source event-based dependency manager for Kubernetes. The core concept of the project are
- * `gateways` which are implemented as a Kubernetes-native Custom Resource Definition that either produce the events internally or process the events that originate from outside the gateways
+Argo Events is an event-based dependency manager for Kubernetes. The core concept of the project are
+ * `gateways` which are implemented as a Kubernetes-native Custom Resource Definition that process events from event source.
 
- * `sensors` which are implemented as a Kubernetes-native Custom Resource Definition that define a set of dependencies and trigger actions.
+ * `sensors` which are implemented as a Kubernetes-native Custom Resource Definition that define a set of event dependencies and trigger actions.
 
-    - Define multiple dependencies from a variety of gateway sources
+    - Define multiple dependencies from a variety of gateway event sources
     - Build custom gateways to support business-level constraint logic
-    - Trigger messages and Kubernetes object creation after successful dependency resolution
+    - Trigger Kubernetes object creation after successful dependency resolution
     - Trigger escalation after errors, or dependency constraint failures
     - Build and manage a distributed, cross-team, event-driven architecture
     - Easily leverage Kubernetes-native APIs to monitor dependencies
@@ -23,10 +23,36 @@ Argo Events is an open source event-based dependency manager for Kubernetes. The
 - Scalable & Resilient.
 - Simple or Complex dependencies. Manage everything from simple, linear, real-time dependencies to complex, multi-source, batch job dependencies.
 
+<br/>
+<br/>
+
+![](docs/architecture.png)
+
+<br/>
+<br/>
+
 ## Getting Started
 [![asciicast](https://asciinema.org/a/207973.png)](https://asciinema.org/a/207973)
 
-- [Installation](./docs/quickstart.md)
-- [Guide](./docs/tutorial.md)
-- [Want to contribute or develop/run locally?](./CONTRIBUTING.md)
-- See where the project is headed in the [roadmap](./ROADMAP.md)
+<br/>
+<br/>
+
+1. [Installation](./docs/quickstart.md)
+2. [Sensor and gateway controllers](docs/controllers-guide.md)
+3. [Learn about gateways](docs/gateway-guide.md)
+4. [Learn about sensors](docs/sensor-guide.md)
+5. [Learn about triggers](docs/trigger-guide.md)
+6. Install Gateways and Sensors
+    1. [Webhook](gateways/core/webhook/install.md)
+    2. [Artifact](gateways/core/artifact/install.md)
+    3. [Calendar](gateways/core/calendar/install.md)
+    4. [Resource](gateways/core/resource/install.md)
+    5. [File](gateways/core/file/install.md)
+    6. Streams
+        1. [NATS](gateways/core/stream/nats/install.md)
+        2. [KAFKA](gateways/core/stream/kafka/install.md)
+        3. [AMQP](gateways/core/stream/amqp/install.md)
+        4. [MQTT](gateways/core/stream/mqtt/install.md)
+7. [Write your own gateway](docs/custom-gateway.md)
+8. [Want to contribute or develop/run locally?](./CONTRIBUTING.md)
+9. See where the project is headed in the [roadmap](./ROADMAP.md)
diff --git a/ROADMAP.md b/ROADMAP.md
@@ -1,9 +1,8 @@
 # Roadmap
 
 ## Development Phase
+- Boolean logic for event dependencies in sensor
 - E2E testing
-- Add NATS Streaming as internal pub-sub system
-- Add SNS & SQS gateway Support
 
 ## Idea Phase
 - Use [hashicorp/go-getter](https://github.com/hashicorp/go-getter) for the `ArtifactReader` interface.

diff --git a/docs/architecture.png b/docs/architecture.png
diff --git a/docs/artifact-guide.md b/docs/artifact-guide.md
@@ -6,60 +6,14 @@ Inlined artifacts are included directly within the sensor resource and decoded a
 
 ## S3
 Amazon Simple Storage Service (S3) is a block/file/object store for the internet. The standardized [API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) allows storage and retrieval of data at any time from anywhere on the web. There are a number of S3 providers that include, but are not limited to:
-* [Amazon S3](https://aws.amazon.com/s3/?nc2=h_m1)
-* [Minio](https://minio.io/)
-* [NetApp](https://www.netapp.com/us/products/data-management-software/object-storage-grid-sds.aspx)
-* [CEPH](http://docs.ceph.com/docs/master/radosgw/s3/)
-* [Rook](https://rook.io/)
-
 ### Minio
-Argo Events uses the [minio-go](https://github.com/minio/minio-go) client for access to any Amazon S3 compatible object store. [Minio](https://www.minio.io/) is an distributed object storage server. Follow the Minio [Bucket Notification Guide](https://docs.minio.io/docs/minio-bucket-notification-guide) for help with configuring your minio server to listen and monitor for bucket event notifications. Note that you will need to setup a supported message queue for configuring your notification targets (i.e. NATS, WebHooks, Kafka, etc.). 
-
-#### Installation on Kubernetes
-The [Minio Deployment Quickstart Guide](https://docs.minio.io/docs/minio-deployment-quickstart-guide.html) is useful for help in getting Minio up & running on your orchestration platform. We've also outlined additional steps below to use Minio for signal notifications and as an object store for trigger resources.
-
-1. Install the Helm chart
-```
-$ helm init
-...
-$ helm install stable/minio --name artifacts --set service.type=LoadBalancer
-...
-
-$ #Verify that the minio pod, the minio service and minio secret are present
-$ kubectl get all -n default -l app=minio
-
-NAME                                   READY     STATUS    RESTARTS   AGE
-pod/artifacts-minio-85547b6bd9-bhtt8   1/1       Running   0          21m
-
-NAME                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
-service/artifacts-minio   ClusterIP   None         <none>        9000/TCP   21m
-
-NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
-deployment.apps/artifacts-minio   1         1         1            1           21m
-
-NAME                                         DESIRED   CURRENT   READY     AGE
-replicaset.apps/artifacts-minio-85547b6bd9   1         1         1         21m
-```
-
-2. Create a bucket in Minio and upload the hello-world.yaml into that bucket.
-Download the hello-world.yaml from https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml
-```
-$ kubectl port-forward `kubectl get pod -l app=minio -o name` 9000:9000
-```
-Open the browser at http://localhost:9000
-Create a new bucket called 'workflows'.
-Upload the hello-world.yaml into that bucket
-
+Argo Events uses the [minio-go](https://github.com/minio/minio-go) client for access to any Amazon S3 compatible object store. [Minio](https://www.minio.io/) is an distributed object storage server. Follow the Minio [Name Notification Guide](https://docs.minio.io/docs/minio-bucket-notification-guide) for help with configuring your minio server to listen and monitor for bucket event notifications. Note that you will need to setup a supported message queue for configuring your notification targets (i.e. NATS, WebHooks, Kafka, etc.). 
 
-#### Enabling bucket notifications
-Once the Minio server is configured with a notification target and you have restarted the server to put the changes into effect, you now need to explicitely enable event notifications for a specified bucket. Enabling these notifications are out of scope of Argo Events since bucket notifications are a construct within Minio that exists at the `bucket` level. To avoid multiple sensors on the same S3 bucket conflicting with each other, creating, updating, and deleting Minio bucket notifications should be delegated to a separate process with knowledge of all notification targets including those outside of the Argo Events.
-```
-$ k edit configmap artifacts-minio
-$ k delete pod artifacts-minio
-```
+## File
+Artifacts are defined in a file that is mounted via a [PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) within the `sensor-controller` pod. 
 
-## File (future enhancement)
-This will enable access to file artifacts via a filesystem mounted as a [PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) within the `sensor-controller` pod. 
+## URL
+Artifacts are accessed from web via RESTful API.
 
-## URL (future enhancement)
-This will enable access to web artifacts via RESTful API.
+## Configmap
+Artifact stored in Kubernetes configmap are accessed using the key.
diff --git a/docs/controllers-guide.md b/docs/controllers-guide.md
@@ -1,6 +1,6 @@
 ## Controllers
 
-* Sensor and Gateway controller are the components which manage Sensor and Gateway resources. 
+* Sensor and Gateway controllers are the components which manage Sensor and Gateway resources. 
 * Sensor and Gateway are Kubernetes Custom Resources. For more information on K8 CRDs visit, https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
 
 
@@ -20,9 +20,9 @@ data:
     namespace: my-custom-namespace # optional
 ```
 
-If you don't provide namespace, controller will watch all namespaces for gateway resource. 
+<b>`namespace`</b>: If you don't provide namespace, controller will watch all namespaces for gateway resource.
 
-<b>Note on `instance-id`</b>: it is used to map a gateway or sensor to a controller. 
+<b>`instance-id`</b>: it is used to map a gateway or sensor to a controller. 
 e.g. when you create a gateway with label `gateways.argoproj.io/gateway-controller-instanceid: argo-events`, a
  controller with label `argo-events` will process that gateway. `instance-id` for controller are managed using [controller-configmap](https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/gateway-controller-configmap.yaml)
 Basically `instance-id` is used to horizontally scale controllers, so you won't end up overwhelming a controller with large
@@ -32,73 +32,6 @@ Basically `instance-id` is used to horizontally scale controllers, so you won't
 
 ### Gateway controller
 Gateway controller watches gateway resource and manages lifecycle of a gateway.
-```yaml
-# The gateway-controller listens for changes on the gateway CRD and creates gateway
-apiVersion: apps/v1beta1
-kind: Deployment
-metadata:
-  name: gateway-controller
-spec:
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        app: gateway-controller
-    spec:
-      serviceAccountName: argo-events-sa
-      containers:
-      - name: gateway-controller
-        image: argoproj/gateway-controller:latest
-        imagePullPolicy: Always
-        env:
-          - name: GATEWAY_NAMESPACE
-            valueFrom:
-              fieldRef:
-                fieldPath: metadata.namespace
-          - name: GATEWAY_CONTROLLER_CONFIG_MAP
-            value: gateway-controller-configmap
-```
-
-* <b>Lifecycle of Gateway in gateway controller</b>
-
-  ![](gateway-controller-fsm.png)
 
 ### Sensor controller
 Sensor controller watches sensor resource and manages lifecycle of a sensor.
-```yaml
-# The sensor-controller listens for changes on the sensor CRD and creates sensor executor jobs
-apiVersion: apps/v1beta1
-kind: Deployment
-metadata:
-  name: sensor-controller
-spec:
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        app: sensor-controller
-    spec:
-      serviceAccountName: argo-events-sa
-      containers:
-      - name: sensor-controller
-        image: argoproj/sensor-controller:latest
-        imagePullPolicy: Always
-        env:
-          - name: SENSOR_NAMESPACE
-            valueFrom:
-              fieldRef:
-                fieldPath: metadata.namespace
-          - name: SENSOR_CONFIG_MAP
-            value: sensor-controller-configmap
-```
-
-* <b>Lifecycle of a Sensor managed by sensor controller</b>
-
-   ![](sensor-fsm.png) 
-
- <br/>
-
-* <b>Lifecycle of  Sensor within sensor pod</b>
-
-   ![](sensor-pod-fsm.png)
-
diff --git a/docs/core-gateway-style.png b/docs/core-gateway-style.png