Slack gateway (#177)

* Adding argo-events logo

* Update readme with logo

* Adding top level diagram

* docs(README): update features

* Added install steps

* docs(README): update readme

* docs(README): update further reading section

* docs(README): adding contribute guideline and license

* docs(gateway): adding gateway architecture diagram

* docs(gateway): updating gateway guide

* docs(): update readme and gateway guide

* Provide ability to define multi-fields/multi-values data filter

* docs(): update sensor docs

* docs(): adding gateway and sensor communication docs

* docs(): update gateway and sensor communication docs

* docs(): update gateway and sensor trigger docs

* docs(): update custom gateway docs

* docs(): update custom gateway docs

* docs(): update gateway docs

* docs(): update gateway docs

* docs(): fix links

* docs(): update trigger guide

* refactor(): restructuring gateway event source validation

* refactor(): updating logs

* refactor(): adding tests to gcp gateway and modifying the validation

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## v0.8
++ Support for arbitrary boolean logic to resolve event dependencies #12
++ Ability to add timezone and extra user payload to calendar gateway #164
++ Data filter bug fix #165
++ Added multi-fields/multi-values data filter #167
++ Added support for backoff option when making connections in stream gateway #168
++ AWS SNS Gateway #169
++ GCP PubSub Gateway #176
++ Support for git as trigger source #171
++ Enhance Gitlan & Github gateway with in-build http server #172
+
+
 ## v0.7
 + Refactor gateways #147
 + Fixed sensor error state recovery bug #145

Gopkg.toml

@@ -25,6 +25,10 @@ required = [
   name = "github.com/aws/aws-sdk-go"
   branch = "master"
 
+[[constraint]]
+  name = "github.com/nlopes/slack"
+  branch = "master"
+
 [[constraint]]
   name = "cloud.google.com/go"
   branch = "master"

README.md

@@ -1,60 +1,135 @@
 # Argo Events - The Event-Based Dependency Manager for Kubernetes
 
 [![Go Report Card](https://goreportcard.com/badge/github.com/argoproj/argo-events)](https://goreportcard.com/report/github.com/argoproj/argo-events)
+[![slack](https://img.shields.io/badge/slack-argoproj-brightgreen.svg?logo=slack)](https://argoproj.github.io/community/join-slack)
+
+<p align="center">
+  <img src="https://github.com/argoproj/argo-events/blob/update-docs/docs/argo-events-logo.png?raw=true" alt="Logo"/>
+</p>
 
 ## What is Argo Events?
-Argo Events is an event-based dependency manager for Kubernetes. The core concepts of the project are
- * `gateways` which are implemented as a Kubernetes Custom Resource Definition (CRD) that process events from an event source.
-
- * `sensors` which are implemented as a CRD that define a set of event dependencies and trigger actions.
-
-Argo Events allows you to - 
-    - Define multiple dependencies from a variety of gateway event sources
-    - Build custom gateways to support business-level constraint logic
-    - Trigger Kubernetes object creation after successful dependency resolution
-    - Trigger escalations after errors or dependency constraint failures
-    - Build and manage a distributed, cross-team, event-driven architecture
-    - Easily leverage Kubernetes-native APIs to monitor dependencies
-
-## Why Argo Events?
-- Runtime agnostic. The first runtime and package agnostic event framework for Kubernetes.
-- Containers. Designed from the ground-up to be Kubernetes-native. 
-- Extremely lightweight. All gateways, with the exception of calendar-based gateways, are event-driven, meaning that there is no polling involved.
-- Configurable. Configure gateways at runtime
-- Scalable & Resilient.
-- Simple or Complex dependencies. Manage everything from simple, linear, real-time dependencies to complex, multi-source, batch job dependencies.
-- Extensible with custom gateways
+**Argo Events** is an event-based dependency manager for Kubernetes which helps you define multiple dependencies from a variety of event sources like webhook, s3, schedules, streams etc.
+and trigger Kubernetes objects after successful event dependencies resolution.
 
 <br/>
 <br/>
 
-![](docs/architecture.png)
+<p align="center">
+  <img src="https://github.com/argoproj/argo-events/blob/update-docs/docs/argo-events-top-level.png?raw=true" alt="High Level Overview"/>
+</p>
 
-<br/>
 <br/>
 
-## Getting Started
-[![asciicast](https://asciinema.org/a/207973.png)](https://asciinema.org/a/207973)
+## Features 
+* Manage dependencies from a variety of event sources.
+* Ability to customize business-level constraint logic for event dependencies resolution.
+* Manage everything from simple, linear, real-time dependencies to complex, multi-source, batch job dependencies.
+* Ability to extends framework to add your own event source listener.
+* Define arbitrary boolean logic to resolve event dependencies.
+* CloudEvents compliant.
+* Ability to manage event sources at runtime.
 
-<br/>
-<br/>
+## Core Concepts
+The framework is made up of two components: 
+
+ 1. **`Gateway`** which is implemented as a Kubernetes-native Custom Resource Definition processes events from event source.
+
+ 2. **`Sensor`** which is implemented as a Kubernetes-native Custom Resource Definition defines a set of event dependencies and triggers K8s resources.
+
+## Install
+
+* ### Requirements
+  * Kubernetes cluster >v1.9
+  * Installed the [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) command-line tool >v1.9.0
+
+* ### Helm Chart
+
+    Make sure you have helm client installed and Tiller server is running. To install helm, follow https://docs.helm.sh/using_helm/
+
+    1. Add `argoproj` repository
+
+    ```bash
+    helm repo add argo https://argoproj.github.io/argo-helm
+    ```
+
+    2. Install `argo-events` chart
+
+    ```bash
+    helm install argo/argo-events
+    ```   
+
+* ### Using kubectl
+  * Deploy Argo Events SA, Roles, ConfigMap, Sensor Controller and Gateway Controller
+
+    ```
+    kubectl create namespace argo-events
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/argo-events-sa.yaml
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/argo-events-cluster-roles.yaml
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/sensor-crd.yaml
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/gateway-crd.yaml
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/sensor-controller-configmap.yaml
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/sensor-controller-deployment.yaml
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/gateway-controller-configmap.yaml
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/gateway-controller-deployment.yaml
+    ```
+
+**Note**: If you have already deployed the argo workflow controller in another namespace
+and the controller is namespace scoped, make sure to deploy a new controller in `argo-events` namespace.  
+
+## Get Started
+Lets deploy a webhook gateway and sensor,
+
+ * First, we need to setup event sources for gateway to listen. The event sources for any gateway are managed using K8s configmap.
+
+   ```bash
+   kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/gateways/webhook-gateway-configmap.yaml 
+   ```
+
+ * Create webhook gateway, 
+
+   ```bash
+    kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/gateways/webhook-http.yaml
+   ```
+
+   After running above command, gateway controller will create corresponding gateway pod and a LoadBalancing service.
+
+ * Create webhook sensor,
+
+    ```bash
+    kubectl apply -n argo-events https://raw.githubusercontent.com/argoproj/argo-events/master/examples/sensors/webhook-http.yaml
+    ```
+
+   Once sensor resource is created, sensor controller will create corresponding sensor pod and a ClusterIP service. 
+
+ * Once the gateway and sensor pods are running, trigger the webhook via a http POST request to `/foo` endpoint.
+   Note: the `WEBHOOK_SERVICE_URL` will differ based on the Kubernetes cluster.
+   ```
+    export WEBHOOK_SERVICE_URL=$(minikube service -n argo-events --url <gateway_service_name>)
+    echo $WEBHOOK_SERVICE_URL
+    curl -d '{"message":"this is my first webhook"}' -H "Content-Type: application/json" -X POST $WEBHOOK_SERVICE_URL/foo
+   ```
+ 
+   <b>Note</b>: 
+     * If you are facing an issue getting service url by running `minikube service -n argo-events --url <gateway_service_name>`, you can use `kubectl port-forward`
+     * Open another terminal window and enter `kubectl port-forward -n argo-events <name_of_the_webhook_gateway_pod> 9003:<port_on_which_gateway_server_is_running>`
+     * You can now use `localhost:9003` to query webhook gateway
+   
+   Verify that the Argo workflow was run when the trigger was executed.
+   ```
+   argo list -n argo-events
+   ```
+
+ * More examples can be found at [examples](./examples)
+
+## Further Reading
+1. [Gateway](docs/gateway-guide.md)
+2. [Sensor](docs/sensor-guide.md)
+3. [Trigger](docs/trigger-guide.md)
+4. [Communication between gateway and sensor](docs/communication.md)
+5. [Controllers](docs/controllers-guide.md)
+
+## Contribute
+[Contributions](https://github.com/argoproj/argo-events/issues) are more than welcome, if you are interested please take a look at our [Contributing Guidelines](./CONTRIBUTING.md).
 
-1. [Installation](./docs/quickstart.md)
-2. [Sensor and gateway controllers](docs/controllers-guide.md)
-3. [Learn about gateways](docs/gateway-guide.md)
-4. [Learn about sensors](docs/sensor-guide.md)
-5. [Learn about triggers](docs/trigger-guide.md)
-6. Install Gateways and Sensors
-    1. [Webhook](gateways/core/webhook/install.md)
-    2. [Artifact](gateways/core/artifact/install.md)
-    3. [Calendar](gateways/core/calendar/install.md)
-    4. [Resource](gateways/core/resource/install.md)
-    5. [File](gateways/core/file/install.md)
-    6. Streams
-        1. [NATS](gateways/core/stream/nats/install.md)
-        2. [KAFKA](gateways/core/stream/kafka/install.md)
-        3. [AMQP](gateways/core/stream/amqp/install.md)
-        4. [MQTT](gateways/core/stream/mqtt/install.md)
-7. [Write your own gateway](docs/custom-gateway.md)
-8. [Want to contribute or develop/run locally?](./CONTRIBUTING.md)
-9. See where the project is headed in the [roadmap](./ROADMAP.md)
+## License
+Apache License Version 2.0, see [LICENSE](./LICENSE)

docs/communication.md

@@ -0,0 +1,55 @@
+# Internal Communication
+
+1. [How gateway forwards events to sensor](#how-gateway-forwards-events-to-sensor)
+2. [HTTP](#http)
+3. [NATS Standard & Streaming](#nats-standard--streaming)
+
+## How gateway forwards events to sensor?
+There are two ways an event is dispatched from gateway to sensor:
+
+  1. **HTTP**
+  2. **NATS standard or streaming service**
+
+
+<br/>
+
+<p align="center">
+  <img src="https://github.com/argoproj/argo-events/blob/update-docs/docs/communication.png?raw=true" alt="Sensor"/>
+</p>
+
+<br/>
+
+## HTTP
+* To use HTTP as communication channel between gateway and sensor, you need to configure the `eventProtocol` in gateway as HTTP. Then, you need to specify
+the port on which the HTTP server in sensor will be running. The HTTP server is spun up automatically with the port configured in sensor spec when
+you create the sensor with `eventProtocol` as HTTP.
+
+* You don't need to specify address of sensor pod. The sensor pod is exposed through a ClusterIP service. This is taken care by the sensor controller.
+The name of the sensor service is formatted in a specific way by sensor controller so that gateway can create the service name from sensor name.
+This is how gateway gets the name of the service exposing sensor. Using the port defined in the spec, gateway makes HTTP POST requests to sensor service.  
+
+  * [**Gateway Example**](https://github.com/argoproj/argo-events/blob/master/examples/gateways/webhook-http.yaml)
+  * [**Sensor Example**](https://github.com/argoproj/argo-events/blob/master/examples/sensors/webhook-http.yaml)
+
+## NATS Standard & Streaming
+* To use NATS standard or streaming as communication channel between gateway and sensor, you need to configure the `eventProtocol` in gateway as NATS and type as either `Standard` or `Streaming`.
+You can read more about NATS [here](https://nats.io/documentation/)
+
+* In case of NATS, gateway doesn't need to be aware of sensors because the gateway acts as a publisher and sensors act as subscriber. 
+
+* You can store events in external persistent volume. This gives you ability to replay events in future for any reasons.
+Read more about storing NATS messages [here](https://nats.io/blog/use-cases-for-persistent-logs-with-nats-streaming/)
+
+*  NATS also facilitates the components that are not part of Argo-Events to consume events generated by gateway.
+
+* For a sensor to consume the events from NATS, the `eventProtocol` needs to specified as NATS. You can then configure the Standard or Streaming connection detail in `eventProtocol`.
+
+  1. Standard NATS example
+      * [**Gateway Example**](https://github.com/argoproj/argo-events/blob/master/examples/gateways/webhook-nats-standard.yaml)
+      * [**Sensor Example**](https://github.com/argoproj/argo-events/blob/master/examples/sensors/webhook-nats.yaml)
+
+  2. Streaming NATS example
+      * [**Gateway Example**](https://github.com/argoproj/argo-events/blob/master/examples/gateways/webhook-nats-streaming.yaml)
+      * [**Sensor Example**](https://github.com/argoproj/argo-events/blob/master/examples/sensors/webhook-nats-streaming.yaml)
+
+ **Note**: The framework **_does not_** provide a NATS installation. You can follow [this guide](https://github.com/nats-io/nats-streaming-operator) to install NATS onto your cluster.

docs/controllers-guide.md

@@ -5,8 +5,7 @@
 
 
 ### Controller configmap
-Provide the `instance-id` and the namespace for controller
-controller configmap
+Defines the `instance-id` and the `namespace` for controller configmap
 e.g. 
 ```yaml
 # The gateway-controller configmap includes configuration information for the gateway-controller
@@ -24,7 +23,7 @@ data:
 
 <b>`instance-id`</b>: it is used to map a gateway or sensor to a controller. 
 e.g. when you create a gateway with label `gateways.argoproj.io/gateway-controller-instanceid: argo-events`, a
- controller with label `argo-events` will process that gateway. `instance-id` for controller are managed using [controller-configmap](https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/gateway-controller-configmap.yaml)
+ controller with label `argo-events` will process that gateway. `instance-id` for controller are managed using [controller-configmap](https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/gateway-controller-configmap.yaml).
 Basically `instance-id` is used to horizontally scale controllers, so you won't end up overwhelming a controller with large
  number of gateways or sensors. Also keep in mind that `instance-id` has nothing to do with namespace where you are
  deploying controllers and gateways/sensors.