diff --git a/docs/README.md b/docs/README.md index 609ae07610..4a33ce2e92 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,80 +1,12 @@ # Knowledge Base -Below lies a list of resources that may be helpful to those looking to understand the go-control-plane API. The aim of these artifacts is to provide enough knowledge and understanding to newcomers and users who wish to use this API within their own codebases to implement an xDS compliant control-plane. +Below lies a list of resources that may be helpful to those looking to understand the go-control-plane API. -## Snapshot Cache +## Implementations The following guides may be helpful on how to use go-control-plane's Snapshot Cache: -- [Snapshot.md](cache/Snapshot.md) -- [Server.md](cache/Server.md) +- [cache.md](snapshot.md) +- [server.md](server.md) ## Getting Started -Below is an example of a simple xDS ready server utilizing the provided Snapshot Cache and gRPC server logic. -```go -import ( - "context" - "net" - "time" - - "google.golang.org/grpc" - "google.golang.org/grpc/keepalive" - - api "github.com/envoyproxy/go-control-plane/envoy/api/v2" - discovery "github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2" - "github.com/envoyproxy/go-control-plane/pkg/cache/v2" - xds "github.com/envoyproxy/go-control-plane/pkg/server/v2" -) - -const ( - grpcKeepaliveTime = 30 * time.Second - grpcKeepaliveTimeout = 5 * time.Second - grpcKeepaliveMinTime = 30 * time.Second - grpcMaxConcurrentStreams = 1000000 -) - -func main() { - snapshotCache := cache.NewSnapshotCache(false, cache.IDHash{}, nil) - server := xds.NewServer(context.Background(), snapshotCache, nil) - var grpcOptions []grpc.ServerOption - grpcOptions = append(grpcOptions, - grpc.MaxConcurrentStreams(grpcMaxConcurrentStreams), - grpc.KeepaliveParams(keepalive.ServerParameters{ - Time: grpcKeepaliveTime, - Timeout: grpcKeepaliveTimeout, - }), - grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{ - MinTime: grpcKeepaliveMinTime, - PermitWithoutStream: true, - }), - ) - grpcServer := grpc.NewServer(grpcOptions...) - lis, _ := net.Listen("tcp", ":8080") - - discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) - api.RegisterEndpointDiscoveryServiceServer(grpcServer, server) - api.RegisterClusterDiscoveryServiceServer(grpcServer, server) - api.RegisterRouteDiscoveryServiceServer(grpcServer, server) - api.RegisterListenerDiscoveryServiceServer(grpcServer, server) - go func() { - if err := grpcServer.Serve(lis); err != nil { - // error handling - } - }() -} -``` - -As mentioned in the README's [Scope](https://github.com/envoyproxy/go-control-plane/blob/master/README.md#scope), you need to cache Envoy configurations. -Generate the key for the corresponding snapshot based on the node information provided from an Envoy node, then cache the configurations. - -```go -import ( - "github.com/envoyproxy/go-control-plane/pkg/cache/v2" - "github.com/envoyproxy/go-control-plane/pkg/cache/types" -) - -var clusters, endpoints, routes, listeners, runtimes []types.Resource - -snapshotCache := cache.NewSnapshotCache(false, cache.IDHash{}, nil) -snapshot := cache.NewSnapshot("1.0", endpoints, clusters, routes, listeners, runtimes) -_ = snapshotCache.SetSnapshot("node1", snapshot) -``` +A fully functional server/cache example is provided inside `internal/example`. Head there to see how a basic control-plane is initialized with a go-control-plane backed cache and server. \ No newline at end of file diff --git a/docs/cache.md b/docs/cache.md new file mode 100644 index 0000000000..649c3b8ca5 --- /dev/null +++ b/docs/cache.md @@ -0,0 +1,65 @@ +# SnapshotCache + +[SnapshotCache](https://github.com/envoyproxy/go-control-plane/blob/main/pkg/cache/v3/simple.go#L40) is a snapshot-based caching mechanism that maintains a versioned config snapshot per node. See the original README scope section for more detailed explanations on the individual cache systems. + +> *NOTE*: SnapshotCache can operate as a REST or bi-di streaming xDS server + +## Create a Snapshot Cache + +To create a snapshot cache, we simply call the provided constructor: +```go +// Create a cache +cache := cache.NewSnapshotCache(false, cache.IDHash{}, nil) +``` +This `cache` object holds a fully compliant [SnapshotCache](https://github.com/envoyproxy/go-control-plane/blob/main/pkg/cache/v3/simple.go#L40) with the necessary methods to manage a configuration cache lifecycle. + +> *NOTE*: The cache needs a high access level inside the management server as it's the source of truth for xDS resources. + +## Snapshots + +Snapshots are groupings of resources at a given point in time for a node cluster. In other words Envoy's and consuming xDS clients registering as node `abc` all share a snapshot of config. This snapshot is the singular source of truth in the cache that represents config for any of those consumers. + +> *NOTE*: Snapshots can be partial, e.g., only including RDS or EDS resources. + +```go +snap, err := cache.NewSnapshot("v0", map[resource.Type][]types.Resource{ + resource.EndpointType: endpoints, + resource.ClusterType: clusters, + resource.RouteType: routes, + resource.ScopedRouteType: scopedRoutes, + resource.ListenerType: listeners, + resource.RuntimeType: runtimes, + resource.SecretType: secrets, + resource.ExtensionConfigType: extensions, +}) +``` + +For a more in-depth example of how to genereate a snapshot, explore our example found [here](https://github.com/envoyproxy/go-control-plane/blob/main/internal/example/resource.go#L168). + +We recommend verifying that your new `snapshot` is consistent within itself meaning that the dependent resources are exactly listed in the snapshot: + +```go +if err := snapshot.Consistent(); err != nil { + l.Errorf("snapshot inconsistency: %+v\n%+v", snapshot, err) + os.Exit(1) +} +``` + +- all EDS resources are listed by name in CDS resources +- all SRDS/RDS resources are listed by name in LDS resources +- all RDS resources are listed by name in SRDS resources + +> *NOTE*: Clusters and Listeners are requested without name references, so Envoy will accept the snapshot list of clusters as-is even if it does not match all references found in xDS. + +Setting a snapshot is as simple as: +```go +// Add the snapshot to the cache +if err := cache.SetSnapshot("envoy-node-id", snapshot); err != nil { + l.Errorf("snapshot error %q for %+v", err, snapshot) + os.Exit(1) +} +``` + +This will trigger all open watches internal to the caching [config watchers](https://github.com/envoyproxy/go-control-plane/blob/main/pkg/cache/v3/cache.go#L45) and anything listening for changes will received updates and responses from the new snapshot. + +*Note*: that a node ID must be provided along with the snapshot object. Internally a mapping of the two is kept so each node can receive the latest version of its configuration. diff --git a/docs/cache/Server.md b/docs/cache/Server.md deleted file mode 100644 index 4e2afc54c2..0000000000 --- a/docs/cache/Server.md +++ /dev/null @@ -1,100 +0,0 @@ -# xDS Server Implementation - -go-control-plane ships with a full streaming implementation of the xDS protocol. We provide an interface as well as helper functions to instantiate a fully xDS compliant gRPC server which utilizes the [SnapshotCache](Snapshot.md) logic behind the scenes. - -## Getting Started - -Below is an example of a functional gRPC server utilizing the go-control-plane interfaces and helper functions. - -```go -// Create a cache -cache := cachev3.NewSnapshotCache(false, cachev3.IDHash{}, example.Logger{}) - -// Create the snapshot that we'll serve to Envoy -snapshot := example.GenerateSnapshot() -if err := snapshot.Consistent(); err != nil { - l.Errorf("snapshot inconsistency: %+v\n%+v", snapshot, err) - os.Exit(1) -} -l.Debugf("will serve snapshot %+v", snapshot) - -// Add the snapshot to the cache -if err := cache.SetSnapshot("envoy-node-id", snapshot); err != nil { - l.Errorf("snapshot error %q for %+v", err, snapshot) - os.Exit(1) -} - -// Run the xDS server -ctx := context.Background() -cb := &testv3.Callbacks{} -srv := serverv3.NewServer(ctx, cache, cb) -example.RunServer(ctx, srv, port) -``` - -For a full in-depth example of this code, please view the integration test located in `pkg/test/main` and the example code in `internal/example`. - -### Callbacks - -All go-control-plane xDS servers require `Callback` methods to run properly. These callbacks are executed throughout certain steps of the xDS lifecycle. The Callback interface which must be implemented can be found [here](https://godoc.org/github.com/envoyproxy/go-control-plane/pkg/server/v2#Callbacks). - -An example implemention of the Callback interface can be found below: -```go -type Callbacks struct { - Signal chan struct{} - Debug bool - Fetches int - Requests int - mu sync.Mutex -} - -func (cb *Callbacks) Report() { - cb.mu.Lock() - defer cb.mu.Unlock() - log.Printf("server callbacks fetches=%d requests=%d\n", cb.Fetches, cb.Requests) -} - -func (cb *Callbacks) OnStreamOpen(_ context.Context, id int64, typ string) error { - if cb.Debug { - log.Printf("stream %d open for %s\n", id, typ) - } - return nil -} - -func (cb *Callbacks) OnStreamClosed(id int64) { - if cb.Debug { - log.Printf("stream %d closed\n", id) - } -} - -func (cb *Callbacks) OnStreamRequest(int64, *discovery.DiscoveryRequest) error { - cb.mu.Lock() - defer cb.mu.Unlock() - cb.Requests++ - if cb.Signal != nil { - close(cb.Signal) - cb.Signal = nil - } - return nil -} - -func (cb *Callbacks) OnStreamResponse(int64, *discovery.DiscoveryRequest, *discovery.DiscoveryResponse) { -} - -func (cb *Callbacks) OnFetchRequest(_ context.Context, req *discovery.DiscoveryRequest) error { - cb.mu.Lock() - defer cb.mu.Unlock() - cb.Fetches++ - if cb.Signal != nil { - close(cb.Signal) - cb.Signal = nil - } - return nil -} -func (cb *Callbacks) OnFetchResponse(*discovery.DiscoveryRequest, *discovery.DiscoveryResponse) {} -``` - -## Info - -The provided gRPC server will take care of creating new watches when new Envoy nodes register with the management server. - -> *NOTE*: The server supports REST/JSON as well as gRPC streaming diff --git a/docs/cache/Snapshot.md b/docs/cache/Snapshot.md deleted file mode 100644 index e07f8e04ff..0000000000 --- a/docs/cache/Snapshot.md +++ /dev/null @@ -1,67 +0,0 @@ -# SnapshotCache - -SnapshotCache is a snapshot-based cache that maintains a single versioned snapshot of responses per node. SnapshotCache consistently replies with the latest snapshot. For the protocol to work correctly in ADS mode, EDS/RDS requests are responded to only when all resources in the snapshot xDS response are named as part of the request. It is expected that the CDS response names all EDS clusters, and the LDS response names all RDS routes in a snapshot, to ensure that Envoy makes the request for all EDS clusters or RDS routes eventually. - -SnapshotCache can operate as a REST or regular xDS backend. The snapshot can be partial, e.g., only including RDS or EDS resources. - -## Create a Snapshot Cache - -To create a snapshot cache, we simply need to execute the provided constructor which will return an implemented `cache.SnapshotCache` interface. - -```go -// Create a cache -cache := cachev3.NewSnapshotCache(false, cachev3.IDHash{}, logger) -``` - -The `IDHash{}` struct is requested by the cache to use when mapping snapshots to nodes. If no hashing function is present, it will simply use the Envoy node ID as the key within the cache map. - -We have defined a `logger` interface which enables us to log at different levels within the source. Please refer to the [godoc](https://godoc.org/github.com/envoyproxy/go-control-plane/pkg/log) on what -functions to implement when creating the logger. - -> *NOTE*: This cache will need high level access within your management server as it is the source of truth for all configuration. - -## Setting a Snapshot - -To create a new snapshot with resources that must be mapped to an Envoy node, we provide the following helper method: - -```go -snapshot := cache.NewSnapshot( - "v1", - endpoints, - clusters, - routes, - listeners, - runtimes, - secrets, -) -``` - -For a more in-depth example on how to genereate a snapshot programmatically, explore how we create a generated snapshot in the integration test found [here](https://github.com/envoyproxy/go-control-plane/blob/master/pkg/test/resource/v2/resource.go#L317). - -> *NOTE*: Not all resources are required to generate a configuration snapshot as it can be a partial. - -We recommend verifying that your new `snapshot` is consistent within itself meaning that the dependent resources are exactly listed in the snapshot: - -- all EDS resources are listed by name in CDS resources -- all RDS resources are listed by name in LDS resources. - -> *NOTE*: Clusters and listeners are requested without name references, so Envoy will accept the snapshot list of clusters as-is even if it does not match all references found in xDS. - -```go -if err := snapshot.Consistent(); err != nil { - l.Errorf("snapshot inconsistency: %+v\n%+v", snapshot, err) - os.Exit(1) -} -``` - -When ready to set a snapshot to an Envoy node, execute the below logic: - -```go -// Add the snapshot to the cache -if err := cache.SetSnapshot("envoy-node-id", snapshot); err != nil { - l.Errorf("snapshot error %q for %+v", err, snapshot) - os.Exit(1) -} -``` - -Note that an Envoy node ID must be provided along with the snapshot object. Internally we map the two together so each Envoy node can receive the latest version of its configuration snapshot. When this is executed, go-control-plane will kick off new watches on the new resources and watch for changes throughout the server. diff --git a/docs/server.md b/docs/server.md new file mode 100644 index 0000000000..d50fc23e48 --- /dev/null +++ b/docs/server.md @@ -0,0 +1,110 @@ +# xDS Server Implementation + +go-control-plane ships with a full [streaming implementation](https://github.com/envoyproxy/go-control-plane/blob/main/pkg/server/v3/server.go#L175) of the xDS protocol. Current support for the servers lists as follows: +- REST HTTP/1.1 *(This will soon be deprecated)* +- gRPC Bi-Di + - State of the World + - Incremental + +## Getting Started + +For a fully functional gRPC server, check out the provided example for what that looks like: +- https://github.com/envoyproxy/go-control-plane/blob/main/internal/example/server.go + +### Callbacks + +All go-control-plane xDS server implementations require `Callback` methods. Callbacks are executed at certain steps of the management server lifecycle. The interface to be implemented can be found [here](https://godoc.org/github.com/envoyproxy/go-control-plane/pkg/server/v2#Callbacks). + +An example implemention of the Callback interface can be found below: +```go +import ( + "context" + "log" + "sync" + + discovery "github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3" +) + +type Callbacks struct { + Signal chan struct{} + Debug bool + Fetches int + Requests int + DeltaRequests int + DeltaResponses int + mu sync.Mutex +} + +func (cb *Callbacks) Report() { + cb.mu.Lock() + defer cb.mu.Unlock() + log.Printf("server callbacks fetches=%d requests=%d\n", cb.Fetches, cb.Requests) +} +func (cb *Callbacks) OnStreamOpen(_ context.Context, id int64, typ string) error { + if cb.Debug { + log.Printf("stream %d open for %s\n", id, typ) + } + return nil +} +func (cb *Callbacks) OnStreamClosed(id int64) { + if cb.Debug { + log.Printf("stream %d closed\n", id) + } +} +func (cb *Callbacks) OnDeltaStreamOpen(_ context.Context, id int64, typ string) error { + if cb.Debug { + log.Printf("delta stream %d open for %s\n", id, typ) + } + return nil +} +func (cb *Callbacks) OnDeltaStreamClosed(id int64) { + if cb.Debug { + log.Printf("delta stream %d closed\n", id) + } +} +func (cb *Callbacks) OnStreamRequest(int64, *discovery.DiscoveryRequest) error { + cb.mu.Lock() + defer cb.mu.Unlock() + cb.Requests++ + if cb.Signal != nil { + close(cb.Signal) + cb.Signal = nil + } + return nil +} +func (cb *Callbacks) OnStreamResponse(context.Context, int64, *discovery.DiscoveryRequest, *discovery.DiscoveryResponse) { +} +func (cb *Callbacks) OnStreamDeltaResponse(id int64, req *discovery.DeltaDiscoveryRequest, res *discovery.DeltaDiscoveryResponse) { + cb.mu.Lock() + defer cb.mu.Unlock() + cb.DeltaResponses++ +} +func (cb *Callbacks) OnStreamDeltaRequest(id int64, req *discovery.DeltaDiscoveryRequest) error { + cb.mu.Lock() + defer cb.mu.Unlock() + cb.DeltaRequests++ + if cb.Signal != nil { + close(cb.Signal) + cb.Signal = nil + } + + return nil +} +func (cb *Callbacks) OnFetchRequest(_ context.Context, req *discovery.DiscoveryRequest) error { + cb.mu.Lock() + defer cb.mu.Unlock() + cb.Fetches++ + if cb.Signal != nil { + close(cb.Signal) + cb.Signal = nil + } + return nil +} +func (cb *Callbacks) OnFetchResponse(*discovery.DiscoveryRequest, *discovery.DiscoveryResponse) {} +``` + +## Info + +The internal go-control-plane gRPC server implementations take care of managing watches with the [Config Watcher](https://github.com/envoyproxy/go-control-plane/blob/main/pkg/cache/v3/cache.go#L45) when new xDS clients register themselves. + +> *NOTE*: The server supports REST/JSON as well as gRPC bi-di streaming diff --git a/internal/example/README.md b/internal/example/README.md index 179bc86aac..19ececa068 100644 --- a/internal/example/README.md +++ b/internal/example/README.md @@ -3,7 +3,7 @@ This is an example of a trivial xDS V3 control plane server. It serves an Envoy configuration that's roughly equivalent to the one used by the Envoy ["Quick Start"](https://www.envoyproxy.io/docs/envoy/latest/start/start#quick-start-to-run-simple-example) docs: a simple http proxy. You can run the example using the project top-level Makefile, e.g.: ``` -go-control-plane$ make example +$ make example ``` The Makefile builds the example server and then runs `build/example.sh` which runs both Envoy and the example server. The example server serves a configuration defined in `internal/example/resource.go`. If everything works correctly, you should be able to open a browser to [http://localhost:10000](http://localhost:10000) and see Envoy's website.