config: adding deprecation utilities (envoyproxy#5840)

Adding hooks for phase 2 of deprecation: fail-by-default features, and runtime overrides.

_Risk Level_: Low (minimal refactors to core code)
_Testing_: new unit tests TODO more tests
_Docs Changes_: TODO docs on deprecation process
_Release Notes_: n/a - will add a release note when we add existing config to deprecated
Part of envoyproxy#5559 and envoyproxy#5693

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Signed-off-by: Fred Douglas <fredlas@google.com>

CONTRIBUTING.md

@@ -33,6 +33,17 @@ maximize the chances of your PR being merged.
   deprecations between 1.3.0 and 1.4.0 will be deleted soon AFTER 1.5.0 is tagged and released
   (at the beginning of the 1.6.0 release cycle). This results in a three to six month window for
   migrating from deprecated code paths to new code paths.
+* Unless the community and Envoy maintainer team agrees on an exception, during the
+  first release cycle after a feature has been deprecated, use of that feature
+  will cause a logged warning, and incrementing the
+  [runtime](https://www.envoyproxy.io/docs/envoy/latest/configuration/runtime#config-runtime)
+  runtime.deprecated_feature_use stat.
+  During the second release cycle, use of the deprecated configuration will
+  cause a configuration load failure, unless the feature in question is
+  explicitly overridden in
+  [runtime](https://www.envoyproxy.io/docs/envoy/latest/configuration/runtime#config-runtime)
+  config. Finally during the third release cycle the code and configuration will be removed
+  entirely.
 * This policy means that organizations deploying master should have some time to get ready for
   breaking changes, but we make no guarantees about the length of time.
 * The breaking change policy also applies to source level extensions (e.g., filters). Code that

docs/root/configuration/runtime.rst

@@ -79,6 +79,32 @@ old tree to the new runtime tree, using the equivalent of the following command:
 
 It's beyond the scope of this document how the file system data is deployed, garbage collected, etc.
 
+Using runtime overrides for deprecated features
+-----------------------------------------------
+
+The Envoy runtime is also a part of the Envoy feature deprecation process.
+
+As described in the Envoy :repo:`breaking change policy <CONTRIBUTING.md#breaking-change-policy>`,
+feature deprecation in Envoy is in 3 phases: warn-by-default, fail-by-default, and code removal.
+
+In the first phase, Envoy logs a warning to the warning log that the feature is deprecated and
+increments the :ref:`deprecated_feature_use <runtime_stats>` runtime stat.
+Users are encouraged to go to :repo:`DEPRECATED.md <DEPRECATED.md>` to see how to
+migrate to the new code path and make sure it is suitable for their use case.
+
+In the second phase the message and filename will be added to
+:repo:`runtime_features.h <source/common/runtime/runtime_features.h>`
+and use of that configuration field will cause the config to be rejected by default. 
+This fail-by-default mode can be overridden in runtime configuration by setting
+envoy.deprecated_features.filename.proto:fieldname to true. For example, for a deprecated field
+``Foo.Bar.Eep`` in ``baz.proto`` set ``envoy.deprecated_features.baz.proto:Eep`` to
+``true``. Use of this override is **strongly discouraged**.
+Fatal-by-default configuration indicates that the removal of the old code paths is imminent. It is
+far better for both Envoy users and for Envoy contributors if any bugs or feature gaps with the new
+code paths are flushed out ahead of time, rather than after the code is removed!
+
+.. _runtime_stats:
+
 Statistics
 ----------
 
@@ -92,4 +118,5 @@ The file system runtime provider emits some statistics in the *runtime.* namespa
   override_dir_not_exists, Counter, Total number of loads that did not use an override directory
   override_dir_exists, Counter, Total number of loads that did use an override directory
   load_success, Counter, Total number of load attempts that were successful
+  deprecated_feature_use, Counter, Total number of times deprecated features were used.
   num_keys, Gauge, Number of keys currently loaded

include/envoy/runtime/BUILD

@@ -13,6 +13,8 @@ envoy_cc_library(
     hdrs = ["runtime.h"],
     external_deps = ["abseil_optional"],
     deps = [
+        "//source/common/common:assert_lib",
+        "//source/common/singleton:threadsafe_singleton",
         "@envoy_api//envoy/type:percent_cc",
     ],
 )

include/envoy/runtime/runtime.h

@@ -9,6 +9,9 @@
 #include "envoy/common/pure.h"
 #include "envoy/type/percent.pb.h"
 
+#include "common/common/assert.h"
+#include "common/singleton/threadsafe_singleton.h"
+
 #include "absl/types/optional.h"
 
 namespace Envoy {
@@ -46,6 +49,7 @@ class Snapshot {
     std::string raw_string_value_;
     absl::optional<uint64_t> uint_value_;
     absl::optional<envoy::type::FractionalPercent> fractional_percent_value_;
+    absl::optional<bool> bool_value_;
   };
 
   typedef std::unordered_map<std::string, Entry> EntryMap;
@@ -71,6 +75,14 @@ class Snapshot {
 
   typedef std::unique_ptr<const OverrideLayer> OverrideLayerConstPtr;
 
+  // Returns true if a deprecated feature is allowed.
+  //
+  // Fundamentally, deprecated features are boolean values.
+  // They are allowed by default or with explicit configuration to "true" via runtime configuration.
+  // They can be disallowed either by inclusion in the hard-coded disallowed_features[] list, or by
+  // configuration of "false" in runtime config.
+  virtual bool deprecatedFeatureEnabled(const std::string& key) const PURE;
+
   /**
    * Test if a feature is enabled using the built in random generator. This is done by generating
    * a random number in the range 0-99 and seeing if this number is < the value stored in the
@@ -201,7 +213,17 @@ class Loader {
   virtual void mergeValues(const std::unordered_map<std::string, std::string>& values) PURE;
 };
 
-typedef std::unique_ptr<Loader> LoaderPtr;
+using LoaderPtr = std::unique_ptr<Loader>;
+
+// To make the runtime generally accessible, we make use of the dreaded
+// singleton class. For Envoy, the runtime will be created and cleaned up by the
+// Server::InstanceImpl initialize() and destructor, respectively.
+//
+// This makes it possible for call sites to easily make use of runtime values to
+// determine if a given feature is on or off, as well as various deprecated configuration
+// protos being enabled or disabled by default.
+using LoaderSingleton = InjectableSingleton<Loader>;
+using ScopedLoaderSingleton = ScopedInjectableLoader<Loader>;
 
 } // namespace Runtime
 } // namespace Envoy

source/common/protobuf/BUILD

@@ -41,6 +41,7 @@ envoy_cc_library(
     deps = [
         ":protobuf",
         "//include/envoy/api:api_interface",
+        "//include/envoy/runtime:runtime_interface",
         "//source/common/common:assert_lib",
         "//source/common/common:hash_lib",
         "//source/common/common:utility_lib",

source/common/protobuf/utility.cc

@@ -8,6 +8,18 @@
 #include "absl/strings/match.h"
 
 namespace Envoy {
+namespace {
+
+absl::string_view filenameFromPath(absl::string_view full_path) {
+  size_t index = full_path.rfind("/");
+  if (index == std::string::npos || index == full_path.size()) {
+    return full_path;
+  }
+  return full_path.substr(index + 1, full_path.size());
+}
+
+} // namespace
+
 namespace ProtobufPercentHelper {
 
 uint64_t checkAndReturnDefault(uint64_t default_value, uint64_t max_value) {
@@ -106,7 +118,7 @@ void MessageUtil::loadFromFile(const std::string& path, Protobuf::Message& messa
   }
 }
 
-void MessageUtil::checkForDeprecation(const Protobuf::Message& message, bool warn_only) {
+void MessageUtil::checkForDeprecation(const Protobuf::Message& message, Runtime::Loader* runtime) {
   const Protobuf::Descriptor* descriptor = message.GetDescriptor();
   const Protobuf::Reflection* reflection = message.GetReflection();
   for (int i = 0; i < descriptor->field_count(); ++i) {
@@ -118,17 +130,32 @@ void MessageUtil::checkForDeprecation(const Protobuf::Message& message, bool war
       continue;
     }
 
+    bool warn_only = true;
+    absl::string_view filename = filenameFromPath(field->file()->name());
+    // Allow runtime to be null both to not crash if this is called before server initialization,
+    // and so proto validation works in context where runtime singleton is not set up (e.g.
+    // standalone config validation utilities)
+    if (runtime && !runtime->snapshot().deprecatedFeatureEnabled(
+                       absl::StrCat("envoy.deprecated_features.", filename, ":", field->name()))) {
+      warn_only = false;
+    }
+
     // If this field is deprecated, warn or throw an error.
     if (field->options().deprecated()) {
       std::string err = fmt::format(
-          "Using deprecated option '{}'. This configuration will be removed from Envoy soon. "
-          "Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for "
-          "details.",
-          field->full_name());
+          "Using deprecated option '{}' from file {}. This configuration will be removed from "
+          "Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md "
+          "for details.",
+          field->full_name(), filename);
       if (warn_only) {
         ENVOY_LOG_MISC(warn, "{}", err);
       } else {
-        throw ProtoValidationException(err, message);
+        const char fatal_error[] =
+            " If continued use of this field is absolutely necessary, see "
+            "https://www.envoyproxy.io/docs/envoy/latest/configuration/runtime"
+            "#using-runtime-overrides-for-deprecated-features for how to apply a temporary and"
+            "highly discouraged override.";
+        throw ProtoValidationException(err + fatal_error, message);
       }
     }
 
@@ -137,10 +164,10 @@ void MessageUtil::checkForDeprecation(const Protobuf::Message& message, bool war
       if (field->is_repeated()) {
         const int size = reflection->FieldSize(message, field);
         for (int j = 0; j < size; ++j) {
-          checkForDeprecation(reflection->GetRepeatedMessage(message, field, j), warn_only);
+          checkForDeprecation(reflection->GetRepeatedMessage(message, field, j), runtime);
         }
       } else {
-        checkForDeprecation(reflection->GetMessage(message, field), warn_only);
+        checkForDeprecation(reflection->GetMessage(message, field), runtime);
       }
     }
   }

source/common/protobuf/utility.h

@@ -5,6 +5,7 @@
 #include "envoy/api/api.h"
 #include "envoy/common/exception.h"
 #include "envoy/json/json_object.h"
+#include "envoy/runtime/runtime.h"
 #include "envoy/type/percent.pb.h"
 
 #include "common/common/hash.h"
@@ -192,11 +193,13 @@ class MessageUtil {
   /**
    * Checks for use of deprecated fields in message and all sub-messages.
    * @param message message to validate.
-   * @param warn_only if true, logs a warning rather than throwing an exception if deprecated fields
-   *   are in use.
-   * @throw ProtoValidationException if deprecated fields are used and warn_only is false.
+   * @param loader optional a pointer to the runtime loader for live deprecation status.
+   * @throw ProtoValidationException if deprecated fields are used and listed
+   *   Runtime::DisallowedFeatures
    */
-  static void checkForDeprecation(const Protobuf::Message& message, bool warn_only);
+  static void
+  checkForDeprecation(const Protobuf::Message& message,
+                      Runtime::Loader* loader = Runtime::LoaderSingleton::getExisting());
 
   /**
    * Validate protoc-gen-validate constraints on a given protobuf.
@@ -206,8 +209,8 @@ class MessageUtil {
    * @throw ProtoValidationException if the message does not satisfy its type constraints.
    */
   template <class MessageType> static void validate(const MessageType& message) {
-    // Log warnings if deprecated fields are in use.
-    checkForDeprecation(message, true);
+    // Log warnings or throw errors if deprecated fields are in use.
+    checkForDeprecation(message);
 
     std::string err;
     if (!Validate(message, &err)) {

source/common/runtime/BUILD

@@ -11,7 +11,10 @@ envoy_package()
 envoy_cc_library(
     name = "runtime_lib",
     srcs = ["runtime_impl.cc"],
-    hdrs = ["runtime_impl.h"],
+    hdrs = [
+        "runtime_features.h",
+        "runtime_impl.h",
+    ],
     external_deps = ["ssl"],
     deps = [
         "//include/envoy/event:dispatcher_interface",

source/common/runtime/runtime_features.h

@@ -0,0 +1,37 @@
+#pragma once
+
+#include <string>
+
+#include "common/singleton/const_singleton.h"
+
+#include "absl/container/flat_hash_set.h"
+
+namespace Envoy {
+namespace Runtime {
+
+const char* disallowed_features[] = {
+    // Acts as both a test entry for deprecated.proto and a marker for the Envoy
+    // deprecation scripts.
+    "envoy.deprecated_features.deprecated.proto:is_deprecated_fatal",
+};
+
+class DisallowedFeatures {
+public:
+  DisallowedFeatures() {
+    for (auto& feature : disallowed_features) {
+      disallowed_features_.insert(feature);
+    }
+  }
+
+  bool disallowedByDefault(const std::string& feature) const {
+    return disallowed_features_.find(feature) != disallowed_features_.end();
+  }
+
+private:
+  absl::flat_hash_set<std::string> disallowed_features_;
+};
+
+using DisallowedFeaturesDefaults = ConstSingleton<DisallowedFeatures>;
+
+} // namespace Runtime
+} // namespace Envoy

source/common/runtime/runtime_impl.cc

@@ -14,7 +14,9 @@
 #include "common/common/utility.h"
 #include "common/filesystem/directory.h"
 #include "common/protobuf/utility.h"
+#include "common/runtime/runtime_features.h"
 
+#include "absl/strings/match.h"
 #include "openssl/rand.h"
 
 namespace Envoy {
@@ -144,6 +146,25 @@ std::string RandomGeneratorImpl::uuid() {
   return std::string(uuid, UUID_LENGTH);
 }
 
+bool SnapshotImpl::deprecatedFeatureEnabled(const std::string& key) const {
+  bool allowed = false;
+  // See if this value is explicitly set as a runtime boolean.
+  bool stored = getBoolean(key, allowed);
+  // If not, the default value is based on disallowedByDefault.
+  if (!stored) {
+    allowed = !DisallowedFeaturesDefaults::get().disallowedByDefault(key);
+  }
+
+  if (!allowed) {
+    // If either disallowed by default or configured off, the feature is not enabled.
+    return false;
+  }
+  // The feature is allowed. It is assumed this check is called when the feature
+  // is about to be used, so increment the feature use stat.
+  stats_.deprecated_feature_use_.inc();
+  return true;
+}
+
 bool SnapshotImpl::featureEnabled(const std::string& key, uint64_t default_value,
                                   uint64_t random_value, uint64_t num_buckets) const {
   return random_value % num_buckets < std::min(getInteger(key, default_value), num_buckets);
@@ -212,13 +233,22 @@ uint64_t SnapshotImpl::getInteger(const std::string& key, uint64_t default_value
   }
 }
 
+bool SnapshotImpl::getBoolean(const std::string& key, bool& value) const {
+  auto entry = values_.find(key);
+  if (entry != values_.end() && entry->second.bool_value_.has_value()) {
+    value = entry->second.bool_value_.value();
+    return true;
+  }
+  return false;
+}
+
 const std::vector<Snapshot::OverrideLayerConstPtr>& SnapshotImpl::getLayers() const {
   return layers_;
 }
 
 SnapshotImpl::SnapshotImpl(RandomGenerator& generator, RuntimeStats& stats,
                            std::vector<OverrideLayerConstPtr>&& layers)
-    : layers_{std::move(layers)}, generator_{generator} {
+    : layers_{std::move(layers)}, generator_{generator}, stats_{stats} {
   for (const auto& layer : layers_) {
     for (const auto& kv : layer->values()) {
       values_.erase(kv.first);
@@ -239,6 +269,20 @@ SnapshotImpl::Entry SnapshotImpl::createEntry(const std::string& value) {
   return entry;
 }
 
+bool SnapshotImpl::parseEntryBooleanValue(Entry& entry) {
+  absl::string_view stripped = entry.raw_string_value_;
+  stripped = absl::StripAsciiWhitespace(stripped);
+
+  if (absl::EqualsIgnoreCase(stripped, "true")) {
+    entry.bool_value_ = true;
+    return true;
+  } else if (absl::EqualsIgnoreCase(stripped, "false")) {
+    entry.bool_value_ = false;
+    return true;
+  }
+  return false;
+}
+
 bool SnapshotImpl::parseEntryUintValue(Entry& entry) {
   uint64_t converted_uint64;
   if (StringUtil::atoull(entry.raw_string_value_.c_str(), converted_uint64)) {