From 27c861b333213c1f4ce52bb8aa89bdb2f8443ee9 Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Tue, 19 Mar 2024 08:05:55 -0600 Subject: [PATCH] Reconfig test results for 1.2.0 (#1718) Add reconfig test results for the 1.2.0 release --- tests/reconfig/results/1.2.0/1.2.0.md | 106 ++++++++++++++++++++++++ tests/reconfig/scripts/cafe-routes.yaml | 2 +- 2 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 tests/reconfig/results/1.2.0/1.2.0.md diff --git a/tests/reconfig/results/1.2.0/1.2.0.md b/tests/reconfig/results/1.2.0/1.2.0.md new file mode 100644 index 0000000000..543c40e6ab --- /dev/null +++ b/tests/reconfig/results/1.2.0/1.2.0.md @@ -0,0 +1,106 @@ +# Reconfiguration testing Results + + +- [Reconfiguration testing Results](#reconfiguration-testing-results) + - [Summary](#summary) + - [Test environment](#test-environment) + - [Results Tables](#results-tables) + - [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready) + - [Event Batch Processing](#event-batch-processing) + - [NumResources to Total Resources](#numresources-to-total-resources) + - [Observations](#observations) + - [Future Improvements](#future-improvements) + + +## Summary + +- Time to ready stayed consistent, if not slightly faster. +- Reload time has slightly increased in some instances. +- Number of batch events has reduced, subsequently increasing the average time of each batch. + +## Test environment + +GKE cluster: + +- Node count: 3 +- Instance Type: e2-medium +- k8s version: 1.27.8-gke.1067004 +- Zone: us-west2-a +- Total vCPUs: 6 +- Total RAM: 12GB +- Max pods per node: 110 + +NGF deployment: + +- NGF version: edge - git commit 96a44240d317875406a8aef8fd1e424f2fb906eb +- NGINX OSS Version: 1.25.4 +- NGINX Plus Version: R31 + +## Results Tables + +> Note: After fixing the `cafe-tls-redirect` to point to the proper Gateway, tests that created 450 HTTPRoutes failed due to https://github.com/nginxinc/nginx-gateway-fabric/issues/1107. Therefore, those tests were re-run after reverting the `cafe-tls-redirect` issue to maintain consistency with the previous release tests. Going forward, results should look different once the above bug is fixed. Added N+ tests, but without testing 150 since it has the bug mentioned above. + +### NGINX Reloads and Time to Ready + +#### OSS + +| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms | +|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------| +| 1 | 30 | 2 | <1 | 2 | 189.5 | 100% | 100% | +| 1 | 150 | 2 | <1 | 2 | 389 | 100% | 100% | +| 2 | 30 | 30 | <1 | 94 | 161 | 100% | 100% | +| 2 | 150 | 154 | <1 | 387 | 267.48 | 100% | 100% | +| 3 | 30 | <1 | <1 | 94 | 127.91 | 100% | 100% | +| 3 | 150 | <1 | <1 | 454 | 128 | 100% | 100% | + +#### Plus + +| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms | +|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------| +| 1 | 30 | 1 | <1 | 2 | 151.5 | 100% | 100% | +| 2 | 30 | 30 | <1 | 94 | 157 | 100% | 100% | +| 3 | 30 | <1 | <1 | 94 | 128 | 100% | 100% | + +### Event Batch Processing + +#### OSS + +| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms | +|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------| +| 1 | 30 | 5 | 733.6 | 80% | 80% | 100% | 100% | 100% | +| 1 | 150 | 5 | 2967 | 40% | 40% | 40% | 40% | 40% | +| 2 | 30 | 371 | 57.32 | 100% | 100% | 100% | 100% | 100% | +| 2 | 150 | 1743 | 75.87 | 98.45% | 100% | 100% | 100% | 100% | +| 3 | 30 | 370 | 37.48 | 99.73% | 99.73% | 100% | 100% | 100% | +| 3 | 150 | 1808 | 40.18 | 99.94% | 99.94% | 99.94% | 99.94% | 100% | + +#### Plus + +| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms | +|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------| +| 1 | 30 | 3 | 1170 | 66% | 66% | 100% | 100% | 100% | +| 2 | 30 | 370 | 58.79 | 100% | 100% | 100% | 100% | 100% | +| 3 | 30 | 370 | 41.32 | 99.73% | 99.73% | 100% | 100% | 100% | + +## NumResources to Total Resources + +| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Attached HTTPRoutes | Total Resources | +|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|---------------------|-----------------| +| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | 2x | | +| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 60 | 244 | +| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 300 | 1204 | + +> Note: Only 2x HTTPRoutes attach to the Gateway because the parentRef name in the `cafe-tls-redirect` HTTPRoute is incorrect. This has been fixed, but until https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 is fixed we can't actually run the test successfully. + +## Observations + +1. Reload time seems to have a increased slightly in a few instances, though time to ready is consistent if not faster. + +2. Processing fewer batches overall due to improvements in resource event tracking. Overall processing time didn't change much, so the average increased due to fewer batches. + +3. No errors in the logs. + + +## Future Improvements + +Fix https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 to allow for 150 resource tests to properly run. diff --git a/tests/reconfig/scripts/cafe-routes.yaml b/tests/reconfig/scripts/cafe-routes.yaml index f4d9823da9..006a8eba92 100644 --- a/tests/reconfig/scripts/cafe-routes.yaml +++ b/tests/reconfig/scripts/cafe-routes.yaml @@ -4,7 +4,7 @@ metadata: name: cafe-tls-redirect spec: parentRefs: - - name: gateway.networking.k8s.io/v1 + - name: gateway namespace: default sectionName: http hostnames: