config: full delta xDS (including ADS) support #7293

fredlas · 2019-06-15T22:25:02Z

Delta ADS/xDS support. Added NewGrpcMuxImpl, a GrpcMux implementation that supports delta (and is intended to soon take over for state-of-the-world). DeltaSubscriptionImpl (which can soon be renamed to/replace GrpcSubscriptionImpl) now uses that new class. NewGrpcMuxImpl is the first user of the recently added WatchMap.

Risk Level: medium (huge new thing that people aren't using yet)
Testing: added delta param variant to ads_integration_test
Docs Changes: added source/common/config/README.md and accompanying diagram, added to root/configuration/overview.rst and root/intro/arch_overview/operations/dynamic_configuration.rst.
Release Notes: added support for delta xDS (including ADS) delivery
Fixes #4991

Signed-off-by: Fred Douglas <fredlas@google.com>

fredlas · 2019-09-27T18:34:43Z

Thanks, done.

lizan

Almost good to go, comments around stats / test coverage.

lizan · 2019-09-27T18:59:25Z

source/common/config/delta_subscription_impl.cc

+    const std::string& version_info) {
+  callbacks_.onConfigUpdate(resources, version_info);
+  stats_.update_success_.inc();
+  stats_.update_attempt_.inc();


Shouldn't this increment be before calling callbacks_.onConfigUpdate(resources, version_info);, which may throw exception? (same for other similar place)

Good question. This exact ordering is ultimately carried over from the old code... and for a second I thought you had found a bug in it!

What actually happens is the throw that can happen within that onConfigUpdate will get caught by the caller of this onConfigUpdate, who will then switch over into onConfigUpdateFailed, which has its own update_attempt.inc().

I think it's safe to call this unacceptably subtle behavior. It's not possible to make it as completely clean as I would like given the current logic, but I have at least changed it so that we're no longer relying on a throw->catch->otherFunction to update the stat. However, looking more closely, I actually don't think the current logic gives the most accurate value for this stat, so I left TODOs.

lizan · 2019-09-27T19:03:57Z

source/common/config/new_grpc_mux_impl.cc

+  updateWatch(type_url, watch, {});
+  auto entry = subscriptions_.find(type_url);
+  if (entry == subscriptions_.end()) {
+    ENVOY_LOG(error, "removeWatch() called for non-existent subscription {}.", type_url);


No test coverage here, how could this happen? Shouldn't be just an ASSERT?

True, it should not happen, and keeping it from never happening only needs the logic within this class to be right. So, I have made this and most other instances in this file into asserts.

lizan · 2019-09-27T19:04:12Z

source/common/config/new_grpc_mux_impl.cc

+            message->system_version_info());
+  auto sub = subscriptions_.find(message->type_url());
+  if (sub == subscriptions_.end()) {
+    ENVOY_LOG(warn,


No coverage here either.

Here is the one instance I will leave as not an assert, since it's about the server's behavior.

Added a test, in new_grpc_mux_impl_test (temporary, will be merged back into grpc_mux_impl_test upon unification).

lizan · 2019-09-27T19:05:32Z

source/common/config/new_grpc_mux_impl.cc

@@ -0,0 +1,238 @@
+#include "common/config/new_grpc_mux_impl.h"


this file doesn't have test coverage in many error cases: https://279265-65214191-gh.circle-artifacts.com/0/coverage/coverage/proc/self/cwd/source/common/config/new_grpc_mux_impl.cc.html

Done. The ratelimit-related ones are handled in grpc_mux_impl_test, so the unification PR should take care of those.

…ent sub test Signed-off-by: Fred Douglas <fredlas@google.com>

Signed-off-by: Fred Douglas <fredlas@google.com>

itayd

please ignore, testing

Signed-off-by: Fred Douglas <fredlas@google.com>

fredlas · 2019-09-30T18:56:02Z

Well, the tests all passed, but clang_tidy had a couple of issues; should be fixed now. Actually, it had many issues, all in server.cc, but IIUC the issues that are pre-existing in parts of the code I'm not modifying won't be blockers.

lizan

LGTM, Thanks! Any further comment @lambdai @htuch?

Signed-off-by: Fred Douglas <fredlas@google.com>

lambdai · 2019-10-01T19:01:12Z

LGTM!

test/integration/integration.h

test/integration/rtds_integration_test.cc

lizan

Thanks! Changes in api/ is one line doc only, so I'm going to merge this. We can iterate further from there.

fredlas · 2019-10-01T21:30:44Z

Thanks very much! It is great to have it in. I will try to get the delta+SotW unification in swiftly, so we will have nice clean, modular gRPC xDS code, more amenable to future changes and additions.

Delta ADS/xDS support. Added NewGrpcMuxImpl, a GrpcMux implementation that supports delta (and is intended to soon take over for state-of-the-world). DeltaSubscriptionImpl (which can soon be renamed to/replace GrpcSubscriptionImpl) now uses that new class. NewGrpcMuxImpl is the first user of the recently added WatchMap. Risk Level: medium (huge new thing that people aren't using yet) Testing: added delta param variant to ads_integration_test Docs Changes: added source/common/config/README.md and accompanying diagram, added to root/configuration/overview.rst and root/intro/arch_overview/operations/dynamic_configuration.rst. Release Notes: added support for delta xDS (including ADS) delivery Fixes envoyproxy#4991 Signed-off-by: Fred Douglas <fredlas@google.com>

hzxuzhonghu · 2019-10-09T03:48:42Z

@fredlas Does the api doc need update?

DELTA_GRPC is not yet entirely implemented! Initially, only CDS is available. Do not use for other xDSes. TODO(fredlas) update/remove this warning when appropriate.

https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/core/config_source.proto#enum-core-apiconfigsource-apitype

fredlas · 2019-10-09T15:01:06Z

@fredlas Does the api doc need update?
DELTA_GRPC is not yet entirely implemented! Initially, only CDS is available. Do not use for other xDSes. TODO(fredlas) update/remove this warning when appropriate.
https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/core/config_source.proto#enum-core-apiconfigsource-apitype

Looks like it, thanks for pointing it out! I'll add that to #8478

wbpcode · 2019-10-12T10:00:52Z

I have read the relevant documentation and code, but I still don't know how the lazy loading and other functions are implemented. I didn't find the code to update the 'watch' at runtime. Or is this just an infrastructure, and specific features such as lazy loading are not implemented? @fredlas

fredlas · 2019-10-14T14:48:53Z

Correct, lazy loading is for later. That was actually the motivation behind renaming all of this to "delta", to emphasize the aspect it actually includes, while leaving "incremental" to describe the eventual inclusion of lazy loading. (Which will also need to reach way into Envoy, through the various xDS users, all the way to the data plane, and generally be an even bigger change than this one, I think).

fredlas added 30 commits March 26, 2019 18:10

WIP snapshot for motivating the various refactor PRs it entails

6702453

Signed-off-by: Fred Douglas <fredlas@google.com>

resolve merge conflict

87d4004

Signed-off-by: Fred Douglas <fredlas@google.com>

quick snapshot so i can merge the detemplatization PR

f290ae7

Signed-off-by: Fred Douglas <fredlas@google.com>

resolve merge conflict

30926f8

Signed-off-by: Fred Douglas <fredlas@google.com>

working for non-agg delta, agg delta should in theory work too

d483936

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

860c03c

Signed-off-by: Fred Douglas <fredlas@google.com>

mostly working snapshot

1df7db5

Signed-off-by: Fred Douglas <fredlas@google.com>

aahhhhhh have to merge to get bazel working again

33b0103

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

916a03d

Signed-off-by: Fred Douglas <fredlas@google.com>

CDS integration test working again

bc7e515

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

164ac09

Signed-off-by: Fred Douglas <fredlas@google.com>

CDS integration passes with 6729 merged

0f7bcdb

Signed-off-by: Fred Douglas <fredlas@google.com>

lol polymorphism

62f42dc

Signed-off-by: Fred Douglas <fredlas@google.com>

fix unit tests

0806842

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

279355b

Signed-off-by: Fred Douglas <fredlas@google.com>

small XdsGrpcContext comment change

fd8b1be

Signed-off-by: Fred Douglas <fredlas@google.com>

quick commit of beginnings of subclass stuff before merge

d7d5b14

Signed-off-by: Fred Douglas <fredlas@google.com>

resolve merge conflict

6768c5c

Signed-off-by: Fred Douglas <fredlas@google.com>

tests partially passing

67ccc26

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

638984e

Signed-off-by: Fred Douglas <fredlas@google.com>

revert some test changes

d24744c

Signed-off-by: Fred Douglas <fredlas@google.com>

revert establishNewStream idempotency

804459c

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

c77d6b9

Signed-off-by: Fred Douglas <fredlas@google.com>

DISALLOW_COPY_AND_ASSIGN saves lives

83dfc89

Signed-off-by: Fred Douglas <fredlas@google.com>

resolve conflict

714d744

Signed-off-by: Fred Douglas <fredlas@google.com>

snapshot

0b96320

Signed-off-by: Fred Douglas <fredlas@google.com>

documentation

0f8a8f1

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

399f815

Signed-off-by: Fred Douglas <fredlas@google.com>

resolve conflict

cd5134e

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

d04b5da

Signed-off-by: Fred Douglas <fredlas@google.com>

fredlas requested review from alyssawilk and mattklein123 as code owners September 27, 2019 18:34

lizan reviewed Sep 27, 2019

View reviewed changes

fredlas added 2 commits September 27, 2019 18:28

rearrange update_attempt inc, assert on nonexistent sub, add nonexist…

dcf287c

…ent sub test Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

d32b4e0

Signed-off-by: Fred Douglas <fredlas@google.com>

itayd reviewed Sep 28, 2019

View reviewed changes

fredlas added 2 commits September 30, 2019 14:54

a couple of clang_tidy things

590e800

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

9ab78de

Signed-off-by: Fred Douglas <fredlas@google.com>

lizan previously approved these changes Sep 30, 2019

View reviewed changes

fredlas added 2 commits October 1, 2019 12:47

correct and simplify pausable_ack_queue RELEASE_ASSERTs

6be011e

Signed-off-by: Fred Douglas <fredlas@google.com>

Merge remote-tracking branch 'upstream/master' into ADS_incremental

aace5dc

Signed-off-by: Fred Douglas <fredlas@google.com>

fredlas dismissed lizan’s stale review via aace5dc October 1, 2019 16:48

kyessenov reviewed Oct 1, 2019

View reviewed changes

test/integration/integration.h Show resolved Hide resolved

kyessenov reviewed Oct 1, 2019

View reviewed changes

test/integration/rtds_integration_test.cc Show resolved Hide resolved

lizan approved these changes Oct 1, 2019

View reviewed changes

lizan merged commit abb14cd into envoyproxy:master Oct 1, 2019

fredlas mentioned this pull request Oct 3, 2019

config: unification of delta and SotW gRPC xDS #8478

Merged

fredlas mentioned this pull request Oct 11, 2019

config: improve temporary delta xDS choosing mechanism #8587

Merged

fredlas mentioned this pull request Nov 11, 2019

config: reinstate #8478 (unification of delta and SotW xDS), reverted by #8939 #8974

Merged

tbarrella mentioned this pull request Dec 7, 2020

integration tests: re-enable set_node_on_first_message_only #14270

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config: full delta xDS (including ADS) support #7293

config: full delta xDS (including ADS) support #7293

fredlas commented Jun 15, 2019 •

edited

Loading

fredlas commented Sep 27, 2019

lizan left a comment

lizan Sep 27, 2019

fredlas Sep 27, 2019

lizan Sep 27, 2019

fredlas Sep 27, 2019

lizan Sep 27, 2019

fredlas Sep 27, 2019

lizan Sep 27, 2019

fredlas Sep 27, 2019

itayd left a comment

fredlas commented Sep 30, 2019

lizan left a comment

lambdai commented Oct 1, 2019

lizan left a comment •

edited

Loading

fredlas commented Oct 1, 2019

hzxuzhonghu commented Oct 9, 2019

fredlas commented Oct 9, 2019

wbpcode commented Oct 12, 2019 •

edited

Loading

fredlas commented Oct 14, 2019

		@@ -0,0 +1,238 @@
		#include "common/config/new_grpc_mux_impl.h"

config: full delta xDS (including ADS) support #7293

config: full delta xDS (including ADS) support #7293

Conversation

fredlas commented Jun 15, 2019 • edited Loading

fredlas commented Sep 27, 2019

lizan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itayd left a comment

Choose a reason for hiding this comment

fredlas commented Sep 30, 2019

lizan left a comment

Choose a reason for hiding this comment

lambdai commented Oct 1, 2019

lizan left a comment • edited Loading

Choose a reason for hiding this comment

fredlas commented Oct 1, 2019

hzxuzhonghu commented Oct 9, 2019

fredlas commented Oct 9, 2019

wbpcode commented Oct 12, 2019 • edited Loading

fredlas commented Oct 14, 2019

fredlas commented Jun 15, 2019 •

edited

Loading

lizan left a comment •

edited

Loading

wbpcode commented Oct 12, 2019 •

edited

Loading