From 4fdbac350609fb7297de795c1c4a153b575ea996 Mon Sep 17 00:00:00 2001 From: Maciej Winnicki Date: Wed, 16 Aug 2017 00:23:37 +0200 Subject: [PATCH 1/2] readme improvements --- README.md | 178 ++++++++++++++++-------------- install.sh | 309 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 406 insertions(+), 81 deletions(-) create mode 100644 install.sh diff --git a/README.md b/README.md index 83844eb..d1edf9f 100644 --- a/README.md +++ b/README.md @@ -5,29 +5,50 @@ [Website](http://www.serverless.com) • [Newsletter](http://eepurl.com/b8dv4P) • [Gitter](https://gitter.im/serverless/serverless) • [Forum](http://forum.serverless.com) • [Meetups](https://github.com/serverless-meetups/main) • [Twitter](https://twitter.com/goserverless) -The Event Gateway combines both API Gateway and Pub/Sub functionality into a single event-driven experience, intended for use with Functions-as-a-Service on AWS, Azure, Google & IBM. It's dataflow for event-driven, serverless architectures. It routes Events (data) to Functions (serverless compute). The Event Gateway is a layer-7 proxy and realtime dataflow engine. +The Event Gateway combines both API Gateway and Pub/Sub functionality into a single event-driven experience. It's dataflow for event-driven, serverless architectures. It routes Events (data) to Functions (serverless compute). -## Quick Start +The Event Gateway is a layer-7 proxy and realtime dataflow engine, intended for use with Functions-as-a-Service on AWS, Azure, Google & IBM. -### Running Locally +## Contents + +1. [Installation](#installation) +1. [Quick Start](#quick-start) +1. [Philosophy](#philosophy) +1. [Motivation](#motivation) +1. [Features](#features) + 1. [Function Discovery](#function-discovery) + 1. [Subscriptions](#subscriptions) +1. [Events API](#events-api) +1. [Configuration API](#configuration-api) +1. [Client Libraries](#client-libraries) +1. [FAQ](#faq) +1. [Architecture](#architecture) +1. [Background](#background) -Download a binary file from the latest [release page](https://github.com/serverless/event-gateway/releases) and run `event-gateway` in `dev` mode: +## Installation + +On macOS or Linux run the following: ``` -event-gateway -dev +curl -sfL https://raw.githubusercontent.com/serverless/event-gateway/master/install.sh | sh ``` -Alternatively, run in Docker container: +On Windows download [binary](https://github.com/serverless/event-gateway/releases), then add to PATH. + +## Quick Start + +### Running Locally + +Run `event-gateway` in `dev` mode: ``` -git clone git@github.com:serverless/event-gateway.git -cd event-gateway -docker build -t event-gateway . -docker run -p 4000:4000 -p 4001:4001 event-gateway -dev +event-gateway -dev ``` ### Register a Function +Register an AWS Lambda function in the Function Discovery. + ``` curl --request POST \ --url http://127.0.0.1:4001/v1/functions \ @@ -37,6 +58,8 @@ curl --request POST \ ### Subscribe to an Event +Once the function is register you can subscribe it to you custom event. + ``` curl --request POST \ --url http://127.0.0.1:4001/v1/subscriptions \ @@ -46,6 +69,8 @@ curl --request POST \ ### Emit an Event +An event can be emitted using [Events API](#events-api). + ``` curl --request POST \ --url http://127.0.0.1:4000/ \ @@ -54,20 +79,7 @@ curl --request POST \ --data '{"foo": "bar"}' ``` -## Contents - -1. [Philosophy](#philosophy) -1. [Motivation](#motivation) -1. [Features](#features) - 1. [Function Discovery](#function-discovery) - 1. [Subscriptions](#subscriptions) -1. [Client Libraries](#client-libraries) -1. [Events API](#events-api) -1. [Configuration API](#configuration-api) -1. [Architecture](#architecture) -1. [What The Event Gateway is NOT](#what-the-event-gateway-is-not) -1. [Background](#background) -1. [Comparison](#comparison) +After emitting the event subscribed function is called asynchronously. ## Philosophy @@ -88,10 +100,11 @@ curl --request POST \ ### Function Discovery -Discover and call serverless functions from anything that can reach the Event Gateway. Function Discovery supports the following function types: +Discover and call serverless functions from anything that can reach the Event Gateway. Function Discovery supports the +following function types: -- FaaS functions (AWS Lambda, Google Cloud Functions) -- HTTP endpoints with an HTTP method specified (e.g. GET http://example.com/function) +- FaaS functions (AWS Lambda, Google Cloud Functions, Azure Functions, OpenWhisk Actions) +- HTTP endpoints (e.g. GET http://example.com/function) #### Example: Register An AWS Lambda Function @@ -190,7 +203,7 @@ eventGateway.subscribe({ curl --request POST \ --url http://localhost:4000/ \ --header 'content-type: application/json' \ - --header 'event: usercreated' \ + --header 'event: user.created' \ --data '{ "name": "Max" }' ``` @@ -199,15 +212,15 @@ curl --request POST \ ```javascript const eventGateway = fdk.eventGateway({ url: 'http://localhost' }) eventGateway.emit({ - event: "userCreated", + event: "user.created", data: { name: "Max" } }) ``` #### Sync subscriptions via HTTP event -Custom event subscriptions are async. There is a special `http` event type for creating sync subscriptions. `http` event is -a HTTP request received on specified path and for specified HTTP method. +Custom event subscriptions are asynchronous. There is a special `http` event type for creating synchronous subscriptions. +`http` event is a HTTP request received on specified path and for specified HTTP method. #### Example: Subscribe to an "http" Event @@ -237,16 +250,12 @@ eventGateway.subscribe({ }) ``` -## Client Libraries - -- [FDK for Node.js](https://github.com/serverless/fdk) - ## Events API -The Event Gateway exposes an API for emitting events. By default Events API runs on `:4000` port. Events API can be used for -emitting both custom and HTTP events. +The Event Gateway exposes an API for emitting events. Events API can be used for emitting custom event, HTTP events and +for invoking function. By default Events API runs on `:4000` port. -### How We Define Events +### Event Definition All data that passes through the Event Gateway is formatted as an Event, based on our default Event schema: @@ -260,7 +269,7 @@ Example: ```json { - "event": "myapp.subscription.created", + "event": "myapp.user.created", "id": "66dfc31d-6844-42fd-b1a7-a489a49f65f3", "receivedAt": 1500897327098, "data": {"foo": "bar"}, @@ -268,13 +277,23 @@ Example: } ``` +When an event occurs, all subscribers are called with the event in above schema as its argument. + #### Event Data Type -The MIME type of the data block can be specified using the `Content-Type` header (by default it's `application/octet-stream`). This allows the event gateway to understand how to deserialize the data block if it needs to. In case of `application/json` type the event gateway passes JSON payload to the target functions. In any other case the data block is Base64 encoded. +The MIME type of the data block can be specified using the `Content-Type` header (by default it's +`application/octet-stream`). This allows the event gateway to understand how to deserialize the data block if it needs +to. In case of `application/json` type the event gateway passes JSON payload to the target functions. In any other case +the data block is base64 encoded. ### Emit a Custom Event (Async Function Invocation) -`POST /` with `Event` header set to an event name. Optionally `Content-Type: ` header can be set to specify payload encoding. +`POST /` + +Request headers: + +- `Event` - `string` - required, event name +- `Content-Type` - `MIME type string` - payload type Request: arbitrary payload, subscribed function receives an event in above schema, where request payload is passed as `data` field @@ -306,7 +325,12 @@ Response: function response ### Invoking a Registered Function (Sync Function Invocation) -`POST /` with `Event` header set to `invoke` and `Function-ID` set to function ID. +`POST /` + +Request headers: + +- `Event` - `string` - `"invoke"` +- `Function-ID` - `string` - ID of a function to call Request: arbitrary payload, invoked function receives an event in above schema, where request payload is passed as `data` field @@ -427,6 +451,23 @@ Dummy endpoint (always returning 200 status code) for checking if the event gate `GET /v1/status` +## Client Libraries + +- [FDK for Node.js](https://github.com/serverless/fdk) + +## FAQ + +### Event Gateway vs FaaS Providers + +The Event Gateway is NOT a FaaS platform. It integrates with existing FaaS providers (AWS Lambda, Google Cloud Functions, +OpenWhisk Actions). The Event Gateway enables building large serverless architectures in a unified way across different +providers. + +### What The Event Gateway is NOT + +- it's not a replacement for message queues (no message ordering, currently weak durability guarantees only) +- it's not a replacement for streaming platforms (no processing capability and consumers group) +- it's not a replacement for existing service discovery solutions from the microservices world ## Architecture @@ -469,19 +510,14 @@ GCloud us-c│ntral1───┐ │ │ λ ├┐ │ └────────────────────┘ └────────────────────┘ ``` -The Event Gateway instances use a strongly consistent, subscribable DB (initially etcd, with support for Consul, Zookeeper, and Dynamo planned) to store and broadcast configuration. The instances locally cache configuration used to drive low-latency event routing. - -## What The Event Gateway is NOT - -- it's not a replacement for message queues (no message ordering, currently weak durability guarantees only) -- it's not a replacement for streaming platforms (no processing capability and consumers group) -- it's not a replacement for existing service discovery solutions from the microservices world +The Event Gateway instances use a strongly consistent, subscribable DB (initially etcd, with support for Consul, +Zookeeper, and Dynamo planned) to store and broadcast configuration. The instances locally cache configuration used to +drive low-latency event routing. ## Background -### SOA challenges - -SOA came along with a new set of challenges. In monolithic architectures, it was simple to call a built-in library or rarely-changing external service. In SOA it involves much more network communication which [is not reliable](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing). The main problems to solve include: +SOA came along with a new set of challenges. In monolithic architectures, it was simple to call a built-in library or +rarely-changing external service. In SOA it involves much more network communication which [is not reliable](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing). The main problems to solve include: 1. Where is the service deployed? How many instances are there? Which instance is the closest to me? (service discovery) 2. Requests to the service should be balanced between all service instances (load balancing) @@ -499,7 +535,7 @@ The following systems are solutions those problems: The main goal of those tools is to manage the inconveniences of network communication. -### Microservices challenges & FaaS +### Microservices Challenges & FaaS The greatest benefit of serverless/FaaS is that it solves almost all of above problems: @@ -512,36 +548,16 @@ The greatest benefit of serverless/FaaS is that it solves almost all of above pr Tools like Envoy/Linkerd solve different domain of technical problems that doesn't occur in serverless space. They have a lot of features that are unnecessary in the context of serverless computing. -### Service discovery in FaaS = Function discovery +### Service Discovery in FaaS = Function Discovery Service discovery problems may be relevant to serverless architectures, especially when we have a multi-cloud setup or we want to call a serverless function from a legacy system (microservices, etc...). There is a need for some proxy that will know where the function is actually deployed and have retry logic built-in. Mapping from function name to serverless function calling metadata is a different problem from tracking the availability of a changing number of service instances. That's why there is a room for new tools that solves **function discovery** problem rather than the service discovery problem. Those problems are fundamentally different. -## Comparison - -### Event Gateway vs FaaS providers - -The Event Gateway is NOT a FaaS platform. It integrates with existing FaaS providers (AWS Lambda, Google Cloud Functions, OpenWhisk Actions). The Event Gateway enables building large serverless architectures in a unified way across different providers. - -### Gateway vs OpenWhisk - -Apache OpenWhisk is an integrated serverless platform. OpenWhisk is built around three concepts: - -- actions -- triggers -- rules - -OpenWhisk, as mentioned above, is a FaaS platform. Triggers & Rules enable building event-driven systems. Those two concepts are similar to the Event Gateway's Pub/Sub system. However, there are few differences: - -- OpenWhisk Rules don't integrate with other FaaS providers -- OpenWhisk doesn't provide a fine-grained access control system -- OpenWhisk doesn't enable exporting events outside OpenWhisk - ## Community -* [Email Updates](http://eepurl.com/b8dv4P) -* [Serverless Forum](http://forum.serverless.com) -* [Gitter Chatroom](https://gitter.im/serverless/serverless) -* [Serverless Meetups](http://www.meetup.com/serverless/) -* [Facebook](https://www.facebook.com/serverless) -* [Twitter](https://twitter.com/goserverless) -* [Contact Us](mailto:hello@serverless.com) +- [Email Updates](http://eepurl.com/b8dv4P) +- [Serverless Forum](http://forum.serverless.com) +- [Gitter Chatroom](https://gitter.im/serverless/serverless) +- [Serverless Meetups](http://www.meetup.com/serverless/) +- [Facebook](https://www.facebook.com/serverless) +- [Twitter](https://twitter.com/goserverless) +- [Contact Us](mailto:hello@serverless.com) \ No newline at end of file diff --git a/install.sh b/install.sh new file mode 100644 index 0000000..643b313 --- /dev/null +++ b/install.sh @@ -0,0 +1,309 @@ +#!/bin/sh +set -e +# Code generated by godownloader. DO NOT EDIT. +# + +usage() { + this=$1 + cat </dev/null +} +uname_os() { + os=$(uname -s | tr '[:upper:]' '[:lower:]') + echo "$os" +} +uname_arch() { + arch=$(uname -m) + case $arch in + x86_64) arch="amd64" ;; + x86) arch="386" ;; + i686) arch="386" ;; + i386) arch="386" ;; + aarch64) arch="arm64" ;; + armv5*) arch="arm5" ;; + armv6*) arch="arm6" ;; + armv7*) arch="arm7" ;; + esac + echo ${arch} +} +uname_os_check() { + os=$(uname_os) + case "$os" in + darwin) return 0 ;; + dragonfly) return 0 ;; + freebsd) return 0 ;; + linux) return 0 ;; + android) return 0 ;; + nacl) return 0 ;; + netbsd) return 0 ;; + openbsd) return 0 ;; + plan9) return 0 ;; + solaris) return 0 ;; + windows) return 0 ;; + esac + echo "$0: uname_os_check: internal error '$(uname -s)' got converted to '$os' which is not a GOOS value. Please file bug at https://github.com/client9/shlib" + return 1 +} +uname_arch_check() { + arch=$(uname_arch) + case "$arch" in + 386) return 0 ;; + amd64) return 0 ;; + arm64) return 0 ;; + armv5) return 0 ;; + armv6) return 0 ;; + armv7) return 0 ;; + ppc64) return 0 ;; + ppc64le) return 0 ;; + mips) return 0 ;; + mipsle) return 0 ;; + mips64) return 0 ;; + mips64le) return 0 ;; + s390x) return 0 ;; + amd64p32) return 0 ;; + esac + echo "$0: uname_arch_check: internal error '$(uname -m)' got converted to '$arch' which is not a GOARCH value. Please file bug report at https://github.com/client9/shlib" + return 1 +} +untar() { + tarball=$1 + case "${tarball}" in + *.tar.gz | *.tgz) tar -xzf "${tarball}" ;; + *.tar) tar -xf "${tarball}" ;; + *.zip) unzip "${tarball}" ;; + *) + echo "Unknown archive format for ${tarball}" + return 1 + ;; + esac +} +mktmpdir() { + test -z "$TMPDIR" && TMPDIR="$(mktemp -d)" + mkdir -p "${TMPDIR}" + echo "${TMPDIR}" +} +http_download() { + local_file=$1 + source_url=$2 + header=$3 + headerflag='' + destflag='' + if is_command curl; then + cmd='curl --fail -sSL' + destflag='-o' + headerflag='-H' + elif is_command wget; then + cmd='wget -q' + destflag='-O' + headerflag='--header' + else + echo "http_download: unable to find wget or curl" + return 1 + fi + if [ -z "$header" ]; then + $cmd $destflag "$local_file" "$source_url" + else + $cmd $headerflag "$header" $destflag "$local_file" "$source_url" + fi +} +github_api() { + local_file=$1 + source_url=$2 + header="" + case "$source_url" in + https://api.github.com*) + test -z "$GITHUB_TOKEN" || header="Authorization: token $GITHUB_TOKEN" + ;; + esac + http_download "$local_file" "$source_url" "$header" +} +github_last_release() { + owner_repo=$1 + giturl="https://api.github.com/repos/${owner_repo}/releases/latest" + html=$(github_api - "$giturl") + version=$(echo "$html" | grep -m 1 "\"tag_name\":" | cut -f4 -d'"') + test -z "$version" && return 1 + echo "$version" +} +hash_sha256() { + TARGET=${1:-/dev/stdin} + if is_command gsha256sum; then + hash=$(gsha256sum "$TARGET") || return 1 + echo "$hash" | cut -d ' ' -f 1 + elif is_command sha256sum; then + hash=$(sha256sum "$TARGET") || return 1 + echo "$hash" | cut -d ' ' -f 1 + elif is_command shasum; then + hash=$(shasum -a 256 "$TARGET" 2>/dev/null) || return 1 + echo "$hash" | cut -d ' ' -f 1 + elif is_command openssl; then + hash=$(openssl -dst openssl dgst -sha256 "$TARGET") || return 1 + echo "$hash" | cut -d ' ' -f a + else + echo "hash_sha256: unable to find command to compute sha-256 hash" + return 1 + fi +} +hash_sha256_verify() { + TARGET=$1 + checksums=$2 + if [ -z "$checksums" ]; then + echo "hash_sha256_verify: checksum file not specified in arg2" + return 1 + fi + BASENAME=${TARGET##*/} + want=$(grep "${BASENAME}" "${checksums}" 2>/dev/null | tr '\t' ' ' | cut -d ' ' -f 1) + if [ -z "$want" ]; then + echo "hash_sha256_verify: unable to find checksum for '${TARGET}' in '${checksums}'" + return 1 + fi + got=$(hash_sha256 "$TARGET") + if [ "$want" != "$got" ]; then + echo "hash_sha256_verify: checksum for '$TARGET' did not verify ${want} vs $got" + return 1 + fi +} +cat /dev/null < Date: Wed, 16 Aug 2017 09:52:45 +0200 Subject: [PATCH 2/2] get back comparision section --- README.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index d1edf9f..348950b 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ The Event Gateway is a layer-7 proxy and realtime dataflow engine, intended for 1. [Events API](#events-api) 1. [Configuration API](#configuration-api) 1. [Client Libraries](#client-libraries) -1. [FAQ](#faq) +1. [Comparison](#comparison) 1. [Architecture](#architecture) 1. [Background](#background) @@ -455,13 +455,7 @@ Dummy endpoint (always returning 200 status code) for checking if the event gate - [FDK for Node.js](https://github.com/serverless/fdk) -## FAQ - -### Event Gateway vs FaaS Providers - -The Event Gateway is NOT a FaaS platform. It integrates with existing FaaS providers (AWS Lambda, Google Cloud Functions, -OpenWhisk Actions). The Event Gateway enables building large serverless architectures in a unified way across different -providers. +## Comparison ### What The Event Gateway is NOT @@ -469,6 +463,14 @@ providers. - it's not a replacement for streaming platforms (no processing capability and consumers group) - it's not a replacement for existing service discovery solutions from the microservices world + +### Event Gateway vs FaaS Providers + +The Event Gateway is NOT a FaaS platform. It integrates with existing FaaS providers (AWS Lambda, Google Cloud Functions, +Azure Functions, OpenWhisk Actions). The Event Gateway enables building large serverless architectures in a unified way +across different providers. + + ## Architecture ```