Merge remote-tracking branch 'upstream/master' into poc

Signed-off-by: Asra Ali <asraa@google.com>

.azure-pipelines/bazel.yml

@@ -11,6 +11,9 @@ steps:
       path: $(Build.StagingDirectory)/repository_cache
     continueOnError: true
 
+  - bash: .azure-pipelines/cleanup.sh
+    displayName: "Removing tools from agent"
+
   - bash: |
       echo "disk space at beginning of build:"
       df -h

.azure-pipelines/cleanup.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -e
+
+# Temporary script to remove tools from Azure pipelines agent to create more disk space room.
+
+sudo apt-get purge -y 'ghc-*' 'zulu-*-azure-jdk' 'libllvm*' 'mysql-*' 'dotnet-*' 'cpp-*'
+
+dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -rn

CONTRIBUTING.md

@@ -52,7 +52,7 @@ versioning guidelines:
   cause a configuration load failure, unless the feature in question is
   explicitly overridden in
   [runtime](https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features)
-  config ([example](configs/using_deprecated_config.v2.yaml)). Finally, following the deprecation 
+  config ([example](configs/using_deprecated_config.v2.yaml)). Finally, following the deprecation
   of the API major version where the field was first
   marked deprecated, the entire implementation code will be removed from the Envoy implementation.
 * This policy means that organizations deploying master should have some time to get ready for
@@ -63,7 +63,7 @@ versioning guidelines:
   deprecation window. Within this window, a warning of deprecation should be carefully logged (some
   features might need rate limiting for logging this). We make no guarantees about code or deployments
   that rely on undocumented behavior.
-* All deprecations/breaking changes will be clearly listed in the [deprecated log](docs/root/intro/deprecated.rst).
+* All deprecations/breaking changes will be clearly listed in the [version history](docs/root/version_history/).
 * High risk deprecations/breaking changes may be announced to the
   [envoy-announce](https://groups.google.com/forum/#!forum/envoy-announce) email list but by default
   it is expected the multi-phase warn-by-default/fail-by-default is sufficient to warn users to move
@@ -109,8 +109,8 @@ versioning guidelines:
   changes for 7 days. Obviously PRs that are closed due to lack of activity can be reopened later.
   Closing stale PRs helps us to keep on top of all of the work currently in flight.
 * If a commit deprecates a feature, the commit message must mention what has been deprecated.
-  Additionally, the [deprecated log](docs/root/intro/deprecated.rst) must be updated with relevant
-  RST links for fields and messages as part of the commit.
+  Additionally, the [version history](docs/root/version_history/current.rst) must be updated with
+  relevant RST links for fields and messages as part of the commit.
 * Please consider joining the [envoy-dev](https://groups.google.com/forum/#!forum/envoy-dev)
   mailing list.
 * If your PR involves any changes to
@@ -167,7 +167,7 @@ There are four suggested options for testing new runtime features:
 3. Set up integration tests with custom runtime defaults as documented in the
    [integration test README](https://github.com/envoyproxy/envoy/blob/master/test/integration/README.md)
 4. Run a given unit test with the new runtime value explicitly set true as done
-   for [runtime_flag_override_test](https://github.com/envoyproxy/envoy/blob/master/test/common/runtime/BUILD) 
+   for [runtime_flag_override_test](https://github.com/envoyproxy/envoy/blob/master/test/common/runtime/BUILD)
 
 Runtime code is held to the same standard as regular Envoy code, so both the old
 path and the new should have 100% coverage both with the feature defaulting true

PULL_REQUESTS.md

@@ -54,7 +54,7 @@ N/A if there were no documentation changes.
 ### <a name="relnotes"></a>Release notes
 
 If this change is user impacting OR extension developer impacting (filter API, etc.) you **must**
-add a release note to the [version history](docs/root/version_history/current.rst) for the 
+add a release note to the [version history](docs/root/version_history/current.rst) for the
 current version. Please include any relevant links. Each release note should be prefixed with the
 relevant subsystem in **alphabetical order** (see existing examples as a guide) and include links
 to relevant parts of the documentation. Thank you! Please write in N/A if there are no release notes.
@@ -73,11 +73,10 @@ you may instead just tag the PR with the issue:
 
 ### <a name="deprecated"></a>Deprecated
 
-If this PR deprecates existing Envoy APIs or code, it should include
-an update to the [deprecated file](docs/root/intro/deprecated.rst) and a one line note in the PR
-description.
+If this PR deprecates existing Envoy APIs or code, it should include an update to the deprecated
+section of the [version history](docs/root/version_history/current.rst) and a one line note in the
+PR description.
 
 If you mark existing APIs or code as deprecated, when the next release is cut, the
 deprecation script will create and assign an issue to you for
 cleaning up the deprecated code path.
-

bazel/README.md

@@ -6,15 +6,18 @@ It is recommended to use [Bazelisk](https://github.com/bazelbuild/bazelisk) inst
 On Linux, run the following commands:
 
 ```
-sudo wget -O /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/download/v0.0.8/bazelisk-linux-amd64
+sudo wget -O /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-amd64
 sudo chmod +x /usr/local/bin/bazel
 ```
 
-On macOS, run the follwing command:
+On macOS, run the following command:
 ```
 brew install bazelbuild/tap/bazelisk
 ```
 
+On Windows, download [bazelisk-windows-amd64.exe](https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-windows-amd64.exe)
+and save this binary in a directory on the PATH as `bazel.exe`.
+
 If you're building from an revision of Envoy prior to August 2019, which doesn't contains a `.bazelversion` file, run `ci/run_envoy_docker.sh "bazel version"`
 to find the right version of Bazel and set the version to `USE_BAZEL_VERSION` environment variable to build.
 
@@ -94,6 +97,37 @@ for how to update or override dependencies.
     version of `ar` on the PATH, so if you run into issues building third party code like luajit
     consider uninstalling binutils.
 
+    On Windows, additional dependencies are required:
+
+    Install the [MSYS2 shell](https://msys2.github.io/) and install the `diffutils`, `patch`,
+    `unzip`, and `zip` packages using `pacman`. Set the `BAZEL_SH` environment variable to the path
+    of the installed MSYS2 `bash.exe` executable. Setting the `MSYS2_ARG_CONV_EXCL` environment
+    variable to a value of `*` is often advisable to ensure argument parsing in the MSYS2 shell
+    behaves as expected.
+
+    `Git` is required. The version installable via MSYS2 is sufficient.
+
+    Install the Windows-native [python3](https://www.python.org/downloads/), the POSIX flavor
+    available via MSYS2 will not work.
+
+    For building with MSVC (the `msvc-cl` config option), you must install at least the VC++
+    workload from the
+    [Build Tools for Visual Studio 2019](https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019). 
+    You may also download Visual Studio 2019 and use the Build Tools packaged with that
+    installation. Earlier versions of VC++ Build Tools/Visual Studio are not recommended at this
+    time. If installed in a non-standard filesystem location, be sure to set the `BAZEL_VC`
+    environment variable to the path of the VC++ package to allow Bazel to find your installation
+    of VC++. Use caution to ensure the `link.exe` that resolves on your PATH is from VC++ Build Tools and
+    not MSYS2.
+
+    Ensure `CMake` and `ninja` binaries are on the PATH. The versions packaged with VC++ Build
+    Tools are sufficient.
+
+    In addition, because of the behavior of the `rules_foreign_cc` component of Bazel, set the
+    `TMPDIR` environment variable to a path usable as a temporary directory (e.g.
+    `C:\Windows\TEMP`). This variable is used frequently by `mktemp` from MSYS2 in the Envoy Bazel
+    build and can cause problems if not set to a value outside the MSYS2 filesystem.
+
 1. Install Golang on your machine. This is required as part of building [BoringSSL](https://boringssl.googlesource.com/boringssl/+/HEAD/BUILDING.md)
    and also for [Buildifer](https://github.com/bazelbuild/buildtools) which is used for formatting bazel BUILD files.
 1. `go get -u github.com/bazelbuild/buildtools/buildifier` to install buildifier. You may need to set `BUILDIFIER_BIN` to `$GOPATH/bin/buildifier`

docs/requirements.txt

@@ -1,4 +1,4 @@
-GitPython==3.0.0
+GitPython==3.1.1
 Jinja2==2.10.3
 MarkupSafe==1.1.1
 Pygments==2.4.2

docs/root/configuration/http/http_filters/fault_filter.rst

@@ -36,19 +36,58 @@ The fault filter has the capability to allow fault configuration to be specified
 This is useful in certain scenarios in which it is desired to allow the client to specify its own
 fault configuration. The currently supported header controls are:
 
-* Request abort configuration via the *x-envoy-fault-abort-request* header. The header value
-  should be an integer that specifies the HTTP status code to return in response to a request
-  and must be in the range [200, 600). In order for the header to work, :ref:`header_abort
+x-envoy-fault-abort-request
+  HTTP status code to abort a request with. The header value should be an integer that specifies
+  the HTTP status code to return in response to a request and must be in the range [200, 600). 
+  In order for the header to work, :ref:`header_abort
   <envoy_api_field_config.filter.http.fault.v2.FaultAbort.header_abort>` needs to be set.
-* Request delay configuration via the *x-envoy-fault-delay-request* header. The header value
-  should be an integer that specifies the number of milliseconds to throttle the latency for.
-  In order for the header to work, :ref:`header_delay
+
+x-envoy-fault-abort-request-percentage
+  The percentage of requests that should be failed with a status code that's defined
+  by the value of *x-envoy-fault-abort-request* HTTP header. The header value should be an integer
+  that specifies the numerator of the percentage of request to apply aborts to and must be greater
+  or equal to 0 and its maximum value is capped by the value of the numerator of
+  :ref:`percentage <envoy_api_field_config.filter.http.fault.v2.FaultAbort.percentage>` field.
+  Percentage's denominator is equal to default percentage's denominator
+  :ref:`percentage <envoy_api_field_config.filter.http.fault.v2.FaultAbort.percentage>` field.
+  In order for the header to work, :ref:`header_abort
+  <envoy_api_field_config.filter.http.fault.v2.FaultAbort.header_abort>` needs to be set and
+  *x-envoy-fault-abort-request* HTTP header needs to be a part of a request.
+
+x-envoy-fault-delay-request
+  The duration to delay a request by. The header value should be an integer that specifies the number
+  of milliseconds to throttle the latency for. In order for the header to work, :ref:`header_delay
   <envoy_api_field_config.filter.fault.v2.FaultDelay.header_delay>` needs to be set.
-* Response rate limit configuration via the *x-envoy-fault-throughput-response* header. The
-  header value should be an integer that specifies the limit in KiB/s and must be > 0. In order
-  for the header to work, :ref:`header_limit
+
+x-envoy-fault-delay-request-percentage
+  The percentage of requests that should be delayed by a duration that's defined by the value of
+  *x-envoy-fault-delay-request* HTTP header. The header value should be an integer that
+  specifies the percentage of request to apply delays to and must be greater
+  or equal to 0 and its maximum value is capped by the value of the numerator of
+  :ref:`percentage <envoy_api_field_config.filter.fault.v2.FaultDelay.percentage>` field.
+  Percentage's denominator is equal to default percentage's denominator
+  :ref:`percentage <envoy_api_field_config.filter.fault.v2.FaultDelay.percentage>` field.
+  In order for the header to work, :ref:`header_delay
+  <envoy_api_field_config.filter.fault.v2.FaultDelay.header_delay>` needs to be set and
+  *x-envoy-fault-delay-request* HTTP header needs to be a part of a request.
+
+x-envoy-fault-throughput-response
+  The rate limit to use when a response to a caller is sent. The header value should be an integer
+  that specifies the limit in KiB/s and must be > 0. In order for the header to work, :ref:`header_limit
   <envoy_api_field_config.filter.fault.v2.FaultRateLimit.header_limit>` needs to be set.
 
+x-envoy-fault-throughput-response-percentage
+  The percentage of requests whose response rate should be limited to the value of
+  *x-envoy-fault-throughput-response* HTTP header. The header value should be an integer that
+  specifies the percentage of request to apply delays to and must be greater
+  or equal to 0 and its maximum value is capped by the value of the numerator of
+  :ref:`percentage <envoy_api_field_config.filter.fault.v2.FaultRateLimit.percentage>` field.
+  Percentage's denominator is equal to default percentage's denominator
+  :ref:`percentage <envoy_api_field_config.filter.fault.v2.FaultRateLimit.percentage>` field.
+  In order for the header to work, :ref:`header_limit
+  <envoy_api_field_config.filter.fault.v2.FaultRateLimit.header_limit>` needs to be set and
+  *x-envoy-fault-delay-request* HTTP header needs to be a part of a request.
+
 .. attention::
 
   Allowing header control is inherently dangerous if exposed to untrusted clients. In this case,
@@ -98,7 +137,7 @@ fault.http.abort.abort_percent
   <envoy_api_field_config.filter.http.fault.v2.HTTPFault.abort>`.
 
 fault.http.abort.http_status
-  HTTP status code that will be used as the  of requests that will be
+  HTTP status code that will be used as the response status code of requests that will be
   aborted if the headers match. Defaults to the HTTP status code specified
   in the config. If the config does not contain an *abort* block, then
   *http_status* defaults to 0. For historic reasons, this runtime key is

docs/root/configuration/http/http_filters/rate_limit_filter.rst

@@ -101,7 +101,7 @@ ratelimit.http_filter_enabled
   % of requests that will call the rate limit service. Defaults to 100.
 
 ratelimit.http_filter_enforcing
-  % of requests that will call the rate limit service and enforce the decision. Defaults to 100.
+  % of requests that that will have the rate limit service decision enforced. Defaults to 100.
   This can be used to test what would happen before fully enforcing the outcome.
 
 ratelimit.<route_key>.http_filter_enabled

docs/root/configuration/observability/access_log.rst

@@ -351,6 +351,9 @@ The following command operators are supported:
 %DOWNSTREAM_LOCAL_ADDRESS_WITHOUT_PORT%
     Same as **%DOWNSTREAM_LOCAL_ADDRESS%** excluding port if the address is an IP address.
 
+%GRPC_STATUS%
+  gRPC status code which is easy to interpret with text message corresponding with number.
+
 %DOWNSTREAM_LOCAL_PORT%
     Similar to **%DOWNSTREAM_LOCAL_ADDRESS_WITHOUT_PORT%**, but only extracts the port portion of the **%DOWNSTREAM_LOCAL_ADDRESS%**
 

docs/root/version_history/current.rst

@@ -4,6 +4,9 @@
 Changes
 -------
 
+* access loggers: added GRPC_STATUS operator on logging format.
+* fault: added support for controlling the percentage of requests that abort, delay and response rate limits faults 
+  are applied to using :ref:`HTTP headers <config_http_filters_fault_injection_http_header>` to the HTTP fault filter.
 * http: fixed a bug where the upgrade header was not cleared on responses to non-upgrade requests.
   Can be reverted temporarily by setting runtime feature `envoy.reloadable_features.fix_upgrade_response` to false.
 * tracing: tracing configuration has been made fully dynamic and every HTTP connection manager

source/common/access_log/BUILD

@@ -47,6 +47,7 @@ envoy_cc_library(
         "//source/common/common:assert_lib",
         "//source/common/common:utility_lib",
         "//source/common/config:metadata_lib",
+        "//source/common/grpc:common_lib",
         "//source/common/http:utility_lib",
         "//source/common/protobuf:message_validator_lib",
         "//source/common/stream_info:utility_lib",

source/common/access_log/access_log_formatter.cc

@@ -14,6 +14,8 @@
 #include "common/common/fmt.h"
 #include "common/common/utility.h"
 #include "common/config/metadata.h"
+#include "common/grpc/common.h"
+#include "common/grpc/status.h"
 #include "common/http/utility.h"
 #include "common/protobuf/message_validator_impl.h"
 #include "common/protobuf/utility.h"
@@ -309,6 +311,9 @@ std::vector<FormatterProviderPtr> AccessLogFormatParser::parse(const std::string
           throw EnvoyException("Invalid header configuration. Format string contains newline.");
         }
         formatters.emplace_back(FormatterProviderPtr{new StartTimeFormatter(args)});
+      } else if (absl::StartsWith(token, "GRPC_STATUS")) {
+        formatters.emplace_back(FormatterProviderPtr{
+            new GrpcStatusFormatter("grpc-status", "", absl::optional<size_t>())});
       } else {
         formatters.emplace_back(FormatterProviderPtr{new StreamInfoFormatter(token)});
       }
@@ -866,6 +871,42 @@ ResponseTrailerFormatter::formatValue(const Http::RequestHeaderMap&, const Http:
   return HeaderFormatter::formatValue(response_trailers);
 }
 
+GrpcStatusFormatter::GrpcStatusFormatter(const std::string& main_header,
+                                         const std::string& alternative_header,
+                                         absl::optional<size_t> max_length)
+    : HeaderFormatter(main_header, alternative_header, max_length) {}
+
+std::string GrpcStatusFormatter::format(const Http::RequestHeaderMap&,
+                                        const Http::ResponseHeaderMap& response_headers,
+                                        const Http::ResponseTrailerMap& response_trailers,
+                                        const StreamInfo::StreamInfo& info) const {
+  const auto grpc_status =
+      Grpc::Common::getGrpcStatus(response_trailers, response_headers, info, true);
+  if (!grpc_status.has_value()) {
+    return UnspecifiedValueString;
+  }
+  const auto grpc_status_message = Grpc::Utility::grpcStatusToString(grpc_status.value());
+  if (grpc_status_message == EMPTY_STRING || grpc_status_message == "InvalidCode") {
+    return std::to_string(grpc_status.value());
+  }
+  return grpc_status_message;
+}
+
+ProtobufWkt::Value GrpcStatusFormatter::formatValue(
+    const Http::RequestHeaderMap&, const Http::ResponseHeaderMap& response_headers,
+    const Http::ResponseTrailerMap& response_trailers, const StreamInfo::StreamInfo& info) const {
+  const auto grpc_status =
+      Grpc::Common::getGrpcStatus(response_trailers, response_headers, info, true);
+  if (!grpc_status.has_value()) {
+    return unspecifiedValue();
+  }
+  const auto grpc_status_message = Grpc::Utility::grpcStatusToString(grpc_status.value());
+  if (grpc_status_message == EMPTY_STRING || grpc_status_message == "InvalidCode") {
+    return ValueUtil::stringValue(std::to_string(grpc_status.value()));
+  }
+  return ValueUtil::stringValue(grpc_status_message);
+}
+
 MetadataFormatter::MetadataFormatter(const std::string& filter_namespace,
                                      const std::vector<std::string>& path,
                                      absl::optional<size_t> max_length)

source/common/access_log/access_log_formatter.h

@@ -208,6 +208,23 @@ class ResponseTrailerFormatter : public FormatterProvider, HeaderFormatter {
                                  const StreamInfo::StreamInfo&) const override;
 };
 
+/**
+ * FormatterProvider for grpc-status
+ */
+class GrpcStatusFormatter : public FormatterProvider, HeaderFormatter {
+public:
+  GrpcStatusFormatter(const std::string& main_header, const std::string& alternative_header,
+                      absl::optional<size_t> max_length);
+
+  // FormatterProvider
+  std::string format(const Http::RequestHeaderMap&, const Http::ResponseHeaderMap& response_headers,
+                     const Http::ResponseTrailerMap& response_trailers,
+                     const StreamInfo::StreamInfo&) const override;
+  ProtobufWkt::Value formatValue(const Http::RequestHeaderMap&, const Http::ResponseHeaderMap&,
+                                 const Http::ResponseTrailerMap&,
+                                 const StreamInfo::StreamInfo&) const override;
+};
+
 /**
  * FormatterProvider based on StreamInfo fields.
  */