Skip to content

Commit

Permalink
Reconfig test results for 1.2.0 (#1718)
Browse files Browse the repository at this point in the history 
Add reconfig test results for the 1.2.0 release
  • Loading branch information
@sjberman
sjberman authored Mar 19, 2024
1 parent e1d6ebb commit 27c861b
Show file tree
Hide file tree
Showing 2 changed files with 107 additions and 1 deletion.
106 changes: 106 additions & 0 deletions tests/reconfig/results/1.2.0/1.2.0.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Reconfiguration testing Results

<!-- TOC -->
- [Reconfiguration testing Results](#reconfiguration-testing-results)
- [Summary](#summary)
- [Test environment](#test-environment)
- [Results Tables](#results-tables)
- [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready)
- [Event Batch Processing](#event-batch-processing)
- [NumResources to Total Resources](#numresources-to-total-resources)
- [Observations](#observations)
- [Future Improvements](#future-improvements)
<!-- TOC -->

## Summary

- Time to ready stayed consistent, if not slightly faster.
- Reload time has slightly increased in some instances.
- Number of batch events has reduced, subsequently increasing the average time of each batch.

## Test environment

GKE cluster:

- Node count: 3
- Instance Type: e2-medium
- k8s version: 1.27.8-gke.1067004
- Zone: us-west2-a
- Total vCPUs: 6
- Total RAM: 12GB
- Max pods per node: 110

NGF deployment:

- NGF version: edge - git commit 96a44240d317875406a8aef8fd1e424f2fb906eb
- NGINX OSS Version: 1.25.4
- NGINX Plus Version: R31

## Results Tables

> Note: After fixing the `cafe-tls-redirect` to point to the proper Gateway, tests that created 450 HTTPRoutes failed due to https://github.com/nginxinc/nginx-gateway-fabric/issues/1107. Therefore, those tests were re-run after reverting the `cafe-tls-redirect` issue to maintain consistency with the previous release tests. Going forward, results should look different once the above bug is fixed. Added N+ tests, but without testing 150 since it has the bug mentioned above.
### NGINX Reloads and Time to Ready

#### OSS

| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
| 1 | 30 | 2 | <1 | 2 | 189.5 | 100% | 100% |
| 1 | 150 | 2 | <1 | 2 | 389 | 100% | 100% |
| 2 | 30 | 30 | <1 | 94 | 161 | 100% | 100% |
| 2 | 150 | 154 | <1 | 387 | 267.48 | 100% | 100% |
| 3 | 30 | <1 | <1 | 94 | 127.91 | 100% | 100% |
| 3 | 150 | <1 | <1 | 454 | 128 | 100% | 100% |

#### Plus

| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
| 1 | 30 | 1 | <1 | 2 | 151.5 | 100% | 100% |
| 2 | 30 | 30 | <1 | 94 | 157 | 100% | 100% |
| 3 | 30 | <1 | <1 | 94 | 128 | 100% | 100% |

### Event Batch Processing

#### OSS

| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
| 1 | 30 | 5 | 733.6 | 80% | 80% | 100% | 100% | 100% |
| 1 | 150 | 5 | 2967 | 40% | 40% | 40% | 40% | 40% |
| 2 | 30 | 371 | 57.32 | 100% | 100% | 100% | 100% | 100% |
| 2 | 150 | 1743 | 75.87 | 98.45% | 100% | 100% | 100% | 100% |
| 3 | 30 | 370 | 37.48 | 99.73% | 99.73% | 100% | 100% | 100% |
| 3 | 150 | 1808 | 40.18 | 99.94% | 99.94% | 99.94% | 99.94% | 100% |

#### Plus

| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
| 1 | 30 | 3 | 1170 | 66% | 66% | 100% | 100% | 100% |
| 2 | 30 | 370 | 58.79 | 100% | 100% | 100% | 100% | 100% |
| 3 | 30 | 370 | 41.32 | 99.73% | 99.73% | 100% | 100% | 100% |

## NumResources to Total Resources

| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Attached HTTPRoutes | Total Resources |
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|---------------------|-----------------|
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | 2x | <total> |
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 60 | 244 |
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 300 | 1204 |

> Note: Only 2x HTTPRoutes attach to the Gateway because the parentRef name in the `cafe-tls-redirect` HTTPRoute is incorrect. This has been fixed, but until https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 is fixed we can't actually run the test successfully.
## Observations

1. Reload time seems to have a increased slightly in a few instances, though time to ready is consistent if not faster.

2. Processing fewer batches overall due to improvements in resource event tracking. Overall processing time didn't change much, so the average increased due to fewer batches.

3. No errors in the logs.


## Future Improvements

Fix https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 to allow for 150 resource tests to properly run.
2 changes: 1 addition & 1 deletion tests/reconfig/scripts/cafe-routes.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ metadata:
name: cafe-tls-redirect
spec:
parentRefs:
- name: gateway.networking.k8s.io/v1
- name: gateway
namespace: default
sectionName: http
hostnames:
Expand Down

0 comments on commit 27c861b

Please sign in to comment.