diff --git a/README.md b/README.md index 348950b..71a8b39 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,29 @@ -# The Event Gateway +![The Event Gateway](docs/assets/event_gateway_repo.png) -[![serverless](http://public.serverless.com/badges/v3.svg)](http://www.serverless.com) -[![Build Status](https://travis-ci.org/serverless/event-gateway.svg?branch=master)](https://travis-ci.org/serverless/event-gateway) +The Event Gateway combines both API Gateway and Pub/Sub functionality into a single event-driven experience. It's +dataflow for event-driven, serverless architectures. It routes Events (data) to Functions (serverless compute). Everything it cares about is an event! Even calling a function. It makes it easy to share events across different systems, teams and organizations! + +Features: +- **Platform agnostic** - All your cloud services are now compatible with one another: share cross-cloud functions and events with AWS Lambda, Microsoft Azure, IBM OpenWhisk and Google Cloud Platform. +- **Send events from any cloud** - Data streams in your application become events. Centralize events from any cloud provider to get a bird’s eye view of all the data flowing through your cloud. +- **React to cross-cloud events** - You aren’t locked in to events and functions being on the same provider: Any event, on any cloud, can trigger any function. Set events and functions up like dominoes and watch them fall. +- **Expose events to your team** - Share events and functions to other parts of the application. Your teammates can find them and utilize them in their own services. +- **Extendable through middleware** - Perform data transforms, authorizations, serializations, and other custom computes straight from the Event Gateway. -[Website](http://www.serverless.com) • [Newsletter](http://eepurl.com/b8dv4P) • [Gitter](https://gitter.im/serverless/serverless) • [Forum](http://forum.serverless.com) • [Meetups](https://github.com/serverless-meetups/main) • [Twitter](https://twitter.com/goserverless) +The Event Gateway is a L7 proxy and realtime dataflow engine, intended for use with Functions-as-a-Service on AWS, +Azure, Google & IBM. + +[![Build Status](https://travis-ci.org/serverless/event-gateway.svg?branch=master)](https://travis-ci.org/serverless/event-gateway) -The Event Gateway combines both API Gateway and Pub/Sub functionality into a single event-driven experience. It's dataflow for event-driven, serverless architectures. It routes Events (data) to Functions (serverless compute). + [Website](http://www.serverless.com) • [Newsletter](http://eepurl.com/b8dv4P) • [Gitter](https://gitter.im/serverless/serverless) • [Forum](http://forum.serverless.com) • [Meetups](https://github.com/serverless-meetups/main) • [Twitter](https://twitter.com/goserverless) -The Event Gateway is a layer-7 proxy and realtime dataflow engine, intended for use with Functions-as-a-Service on AWS, Azure, Google & IBM. ## Contents 1. [Installation](#installation) 1. [Quick Start](#quick-start) -1. [Philosophy](#philosophy) 1. [Motivation](#motivation) -1. [Features](#features) +1. [Components](#components) 1. [Function Discovery](#function-discovery) 1. [Subscriptions](#subscriptions) 1. [Events API](#events-api) @@ -23,6 +31,8 @@ The Event Gateway is a layer-7 proxy and realtime dataflow engine, intended for 1. [Client Libraries](#client-libraries) 1. [Comparison](#comparison) 1. [Architecture](#architecture) + 1. [System Overview](#system-overview) + 1. [Clustering](#clustering) 1. [Background](#background) ## Installation @@ -33,10 +43,12 @@ On macOS or Linux run the following: curl -sfL https://raw.githubusercontent.com/serverless/event-gateway/master/install.sh | sh ``` -On Windows download [binary](https://github.com/serverless/event-gateway/releases), then add to PATH. +On Windows download [binary](https://github.com/serverless/event-gateway/releases). ## Quick Start +_Check out [`event-gateway-examples` repo](https://github.com/serverless/event-gateway-examples) for more examples._ + ### Running Locally Run `event-gateway` in `dev` mode: @@ -76,16 +88,11 @@ curl --request POST \ --url http://127.0.0.1:4000/ \ --header 'content-type: application/json' \ --header 'event: pageVisited' \ - --data '{"foo": "bar"}' + --data '{"userId": "123"}' ``` After emitting the event subscribed function is called asynchronously. -## Philosophy - -- Everything we care about is an event! (even calling a function) -- Make it easy to share events across different systems, teams and organizations! - ## Motivation - It is cumbersome to plug things into each other. This should be easy! Why do I need to set up a queue system to @@ -96,16 +103,17 @@ After emitting the event subscribed function is called asynchronously. - Using new functions is risky without the ability to incrementally deploy them. - The AWS API Gateway is frequently cited as a performance and cost-prohibitive factor for using AWS Lambda. -## Features +## Components ### Function Discovery -Discover and call serverless functions from anything that can reach the Event Gateway. Function Discovery supports the -following function types: +Discover and call serverless functions from anything that can reach the Event Gateway. Function Discovery supports the following function types: - FaaS functions (AWS Lambda, Google Cloud Functions, Azure Functions, OpenWhisk Actions) - HTTP endpoints (e.g. GET http://example.com/function) +Function Discovery stores information about functions allowing the Eveng Gateway to call them as a reaction to received event. + #### Example: Register An AWS Lambda Function ##### curl example @@ -145,7 +153,6 @@ eventGateway.registerFunction({ ```http curl --request POST \ --url http://localhost:4000/ \ - --header ': ' \ --header 'content-type: application/json' \ --header 'event: invoke' \ --header 'function-id: createUser' \ @@ -286,26 +293,37 @@ The MIME type of the data block can be specified using the `Content-Type` header to. In case of `application/json` type the event gateway passes JSON payload to the target functions. In any other case the data block is base64 encoded. -### Emit a Custom Event (Async Function Invocation) +### Emit a Custom Event + +**Endpoint** -`POST /` +`POST /` -Request headers: +**Request Headers** - `Event` - `string` - required, event name - `Content-Type` - `MIME type string` - payload type -Request: arbitrary payload, subscribed function receives an event in above schema, where request payload is passed as `data` field +**Request** + +arbitrary payload, subscribed function receives an event in above schema, where request payload is passed as `data` field -Response: `202 Accepted` in case of success +**Response** + +Status code: +- `202 Accepted` ### Emit an HTTP Event Creating HTTP subscription requires `method` and `path` properties. Those properties are used to listen for HTTP events. -` /` +**Endpoint** + +` /` + +**Request** -Request: arbitrary payload, subscribed function receives an event in above schema. `data` field has the following fields: +arbitrary payload, subscribed function receives an event in above schema. `data` field has the following fields: ``` { @@ -321,32 +339,46 @@ Request: arbitrary payload, subscribed function receives an event in above schem } ``` -Response: function response +**Response** -### Invoking a Registered Function (Sync Function Invocation) +Status code: +- `200 OK` with payload with function response -`POST /` +### Invoking a Registered Function - Sync Function Invocation -Request headers: +**Endpoint** + +`POST /` + +**Request Headers** - `Event` - `string` - `"invoke"` - `Function-ID` - `string` - ID of a function to call -Request: arbitrary payload, invoked function receives an event in above schema, where request payload is passed as `data` field +**Request** + +arbitrary payload, invoked function receives an event in above schema, where request payload is passed as `data` field + +**Response** -Response: function response +Status code: +- `200 OK` with payload with function response ## Configuration API -The Event Gateway exposes a RESTful configuration API. By default Configuration API runs on `:4001` port. +The Event Gateway exposes a RESTful JSON configuration API. By default Configuration API runs on `:4001` port. + +### Function Discovery + +#### Register Function -### Function discovery +**Endpoint** -#### Register function +`POST /v1/functions` -`POST /v1/functions` +**Request** -Request: +JSON object: - `functionId` - `string` - required, function name - `provider` - `object` - required, provider specific information about a function, depends on type: @@ -354,24 +386,34 @@ Request: - `type` - `string` - required, provider type: `awslambda` - `arn` - `string` - required, AWS ARN identifier - `region` - `string` - required, region name - - `awsAccessKeyID` - `string` - optional, AWS API key ID - - `awsSecretAccessKey` - `string` - optional, AWS API key + - `awsAccessKeyID` - `string` - optional, AWS API key ID. By default credentials from the [environment](http://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials) are used. + - `awsSecretAccessKey` - `string` - optional, AWS API key. By default credentials from the [environment](http://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials) are used. - for HTTP function: - `type` - `string` - required, provider type: `http` - `url` - `string` - required, the URL of an http or https remote endpoint -Response: +**Response** + +Status code: +- `200 OK` on success +- `400 Bad Request` on validation error +JSON object: - `functionId` - `string` - function name - `provider` - `object` - provider specific information about a function -#### Update function +------ -`PUT /v1/functions/` +#### Update Function -Request: +**Endpoint** + +`PUT /v1/functions/` + +**Request** + +JSON object: -- `functionId` - `string` - required, function name - `provider` - `object` - required, provider specific information about a function, depends on type: - for AWS Lambda: - `type` - `string` - required, provider type: `awslambda` @@ -383,61 +425,107 @@ Request: - `type` - `string` - required, provider type: `http` - `url` - `string` - required, the URL of an http or https remote endpoint -Response: +**Response** + +Status code: +- `200 OK` on success +- `400 Bad Request` on validation error +- `404 Not Found` if function doesn't exists +JSON object: - `functionId` - `string` - function name - `provider` - `object` - provider specific information about a function -#### Delete function +------ + +#### Delete Function + +Delete all types of functions. This operation fails if the function is currently in-use by a subscription. + +**Endpoint** + +`DELETE /v1/functions/` + +**Response** -`DELETE /v1/functions/` +Status code: +- `204 No Content` on success +- `404 Not Found` if function doesn't exists -Notes: +------ -- used to delete all types of functions -- fails if the function ID is currently in-use by a subscription +#### Get Functions -#### Get functions +**Endpoint** -`GET /v1/functions` +`GET /v1/functions` -Response: +**Response** +Status code: +- `200 OK` on success + +JSON object: - `functions` - `array` of `object` - functions: - `functionId` - `string` - function name - `provider` - `object` - provider specific information about a function ### Subscriptions -#### Create subscription +#### Create Subscription + +**Endpoint** -`POST /v1/subscriptions` +`POST /v1/subscriptions` -Request: +**Request** - `event` - `string` - event name - `functionId` - `string` - ID of function to receive events - `method` - `string` - optionally, in case of `http` event, uppercase HTTP method that accepts requests - `path` - `string` - optionally, in case of `http` event, path that accepts requests, it starts with "/" -Response: +**Response** + +Status code: +- `200 OK` on success +- `400 Bad Request` on validation error -- `subscriptionId` - `string` - subscription ID, which is event name + function ID, e.g. `newusers-userProcessGroup` +JSON object: +- `subscriptionId` - `string` - subscription ID - `event` - `string` - event name - `functionId` - ID of function - `method` - `string` - optionally, in case of `http` event, HTTP method that accepts requests - `path` - `string` - optionally, in case of `http` event, path that accepts requests, starts with `/` -#### Delete subscription +------ + +#### Delete Subscription + +**Endpoint** + +`DELETE /v1/subscriptions/` -`DELETE /v1/subscriptions/` +**Response** -#### Get subscriptions +Status code: +- `204 No Content` on success +- `404 Not Found` if function doesn't exists -`GET /v1/subscriptions` +------ -Response: +#### Get Subscriptions +**Endpoint** + +`GET /v1/subscriptions` + +**Response** + +Status code: +- `200 OK` on success + +JSON object: - `subscriptions` - `array` of `object` - subscriptions - `subscriptionId` - `string` - subscription ID - `event` - `string` - event name @@ -447,9 +535,11 @@ Response: ### Status -Dummy endpoint (always returning 200 status code) for checking if the event gateway instance is running. +Dummy endpoint (always returning `200 OK` status code) for checking if the event gateway instance is running. + +**Endpoint** -`GET /v1/status` +`GET /v1/status` ## Client Libraries @@ -473,48 +563,91 @@ across different providers. ## Architecture +### System Overview + ``` - AWS us-east-1 (main ─┐ - region) │ - │ ┌─────────────┐ │ - │ │ │ │ - ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─│─ ▶│ etcd │◀ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ - │ │ │ │ │ - │ │ └─────────────┘ │ - │ ▲ │ │ - │ │ │ │ - │ ▼ │ │ - │ │ ┌──────────────┐ │ - │ │ │ │ │ - │ │ │ Gateway │ │ - │ │ instance │ │ │ - │ │ │ │ │ - │ └──────────────┘ │ │ - │ │ ▲ │ - │ │ │ │ - │ │ ▼ │ - │ ┌───┐ │ │ -GCloud us-c│ntral1───┐ │ │ λ ├┐ │ Azure West US────────┐ -│ ▼ │ │ └┬──┘│ │ │ ▼ │ -│ ┌──────────────┐ │ │ └───┘ │ │ ┌──────────────┐ │ -│ │ │ │ └────────────────────┘ │ │ │ │ -│ │ Gateway │ │ │ │ Gateway │ │ -│ │ instance │ │ │ │ instance │ │ -│ │ │ │ │ │ │ │ -│ └──────────────┘ │ │ └──────────────┘ │ -│ ▲ │ │ ▲ │ -│ │ │ │ │ │ -│ ▼ │ │ ▼ │ -│ ┌───┐ │ │ ┌───┐ │ -│ │ λ ├┐ │ │ │ λ ├┐ │ -│ └┬──┘│ │ │ └┬──┘│ │ -│ └───┘ │ │ └───┘ │ -└────────────────────┘ └────────────────────┘ + ┌──────────────┐ + │ │ + │ Client │ + │ │ + └──────────────┘ + ▲ + │ + Event + │ + ▼ + ┌───────────────────────────────────────────────────────────┐ + │ │ + │ Event Gateway Cluster │ + │ │ + └───────────────────────────────────────────────────────────┘ + ▲ + │ + │ + ┌─────────────────────────────┼─────────────────────────────┐ + │ │ │ + │ │ │ + ▼ ▼ ▼ + ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ + │ AWS Lambda │ │ Google Cloud │ │Azure Function │ + │ Function │ │ Function │ │ │ + │ │ │ │ │ Region: │ + │ Region: │ │ Region: │ │ West US │ + │ us-east-1 │ │ us-central1 │ │ │ + └───────────────┘ └───────────────┘ └───────────────┘ ``` -The Event Gateway instances use a strongly consistent, subscribable DB (initially etcd, with support for Consul, -Zookeeper, and Dynamo planned) to store and broadcast configuration. The instances locally cache configuration used to -drive low-latency event routing. +### Clustering + +The Event Gateway instances use a strongly consistent, subscribable DB (initially [etcd](https://coreos.com/etcd), +with support for Consul, and Zookeeper planned) to store and broadcast configuration. The instances locally +cache configuration used to drive low-latency event routing. The instance local cache is built asynchronously based on +events from backing DB. + +The Event Gateway is a horizontally scalable system. It can be scaled by adding instances to the cluster. A cluster is +a group of instances sharing the same database. A cluster can be created in one cloud region, across multiple regions, +across multiple cloud provider or even in both cloud and on-premise data centers. + +The Event Gateway is a stateless service and there is no direct communication between different instances. All +configuration data is shared using backing DB. If the instance from region A needs to call a function from region B the +invocation is not routed through the instance in region 2. The instance from region A invokes the function from region 2 +directly. + +``` +┌─────────────────────────────────────────────Event Gateway Cluster──────────────────────────────────────────────┐ +│ │ +│ │ +│ Cloud Region 1───────┐ │ +│ │ │ │ +│ │ ┌─────────────┐ │ │ +│ │ │ │ │ │ +│ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│─ ▶│etcd cluster │◀ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ +│ │ │ │ │ │ │ +│ │ │ └─────────────┘ │ │ +│ │ ▲ │ │ │ +│ │ │ │ │ +│ Cloud Region 2───────┐ │ │ │ Cloud Regio│ 3───────┐ │ +│ │ │ │ │ │ │ │ │ +│ │ ▼ │ │ ▼ │ │ ▼ │ │ +│ │ ┌───────────────┐ │ │ ┌──────────────┐ │ │ ┌──────────────┐ │ │ +│ │ │ │ │ │ │ │ │ │ │ │ │ │ +│ │ │ Event Gateway │ │ │ │Event Gateway │ │ │ │Event Gateway │ │ │ +│ │ │ instance │◀┼──────────┐ │ │ instance │◀─┼──────────┐ │ │ instance │ │ │ +│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ +│ │ └───────────────┘ │ │ │ └──────────────┘ │ │ │ └──────────────┘ │ │ +│ │ ▲ │ │ │ ▲ │ │ │ ▲ │ │ +│ │ │ │ │ │ │ │ │ │ │ │ │ +│ │ │ │ │ │ │ │ │ │ │ │ │ +│ │ ▼ │ │ │ ▼ │ │ │ ▼ │ │ +│ │ ┌───┐ │ │ │ ┌───┐ │ │ │ ┌───┐ │ │ +│ │ │ λ ├┐ │ └───┼───────▶│ λ ├┐ │ └────┼───────▶│ λ ├┐ │ │ +│ │ └┬──┘│ │ │ └┬──┘│ │ │ └┬──┘│ │ │ +│ │ └───┘ │ │ └───┘ │ │ └───┘ │ │ +│ └────────────────────┘ └────────────────────┘ └────────────────────┘ │ +│ │ +│ │ +└────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ +``` ## Background @@ -562,4 +695,4 @@ Service discovery problems may be relevant to serverless architectures, especial - [Serverless Meetups](http://www.meetup.com/serverless/) - [Facebook](https://www.facebook.com/serverless) - [Twitter](https://twitter.com/goserverless) -- [Contact Us](mailto:hello@serverless.com) \ No newline at end of file +- [Contact Us](mailto:hello@serverless.com) diff --git a/docs/assets/event_gateway_repo.png b/docs/assets/event_gateway_repo.png new file mode 100644 index 0000000..390adde Binary files /dev/null and b/docs/assets/event_gateway_repo.png differ diff --git a/docs/assets/title.png b/docs/assets/title.png new file mode 100644 index 0000000..fd5846a Binary files /dev/null and b/docs/assets/title.png differ