Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconfig test results for 1.2.0 #1718

Merged
merged 4 commits into from
Mar 19, 2024
Merged

Reconfig test results for 1.2.0 #1718

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
41e090a
Reconfig test results for 1.2.0
sjberman Mar 18, 2024
ac0a18b
Clear up summary
sjberman Mar 18, 2024
5dcc9a1
Fix bug phrasing
sjberman Mar 18, 2024
c98da1e
Add plus results
sjberman Mar 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions tests/reconfig/results/1.2.0/1.2.0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Reconfiguration testing Results

<!-- TOC -->
- [Reconfiguration testing Results](#reconfiguration-testing-results)
- [Summary](#summary)
- [Test environment](#test-environment)
- [Results Tables](#results-tables)
- [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready)
- [Event Batch Processing](#event-batch-processing)
- [NumResources to Total Resources](#numresources-to-total-resources)
- [Observations](#observations)
- [Future Improvements](#future-improvements)
<!-- TOC -->

## Summary

- Time to ready stayed consistent, if not slightly faster.
- Reload time has slightly increased in some instances.
- Number of batch events has reduced, subsequently increasing the average time of each batch.

## Test environment

GKE cluster:

- Node count: 3
- Instance Type: e2-medium
- k8s version: 1.27.8-gke.1067004
- Zone: us-west2-a
- Total vCPUs: 6
- Total RAM: 12GB
- Max pods per node: 110

NGF deployment:

- NGF version: edge - git commit 96a44240d317875406a8aef8fd1e424f2fb906eb
- NGINX OSS Version: 1.25.4
- NGINX Plus Version: R31

## Results Tables

> Note: After fixing the `cafe-tls-redirect` to point to the proper Gateway, tests that created 450 HTTPRoutes failed due to https://github.com/nginxinc/nginx-gateway-fabric/issues/1107. Therefore, those tests were re-run after reverting the `cafe-tls-redirect` issue to maintain consistency with the previous release tests. Going forward, results should look different once the above bug is fixed. Added N+ tests, but without testing 150 since it has the bug mentioned above.

### NGINX Reloads and Time to Ready

#### OSS

| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
| 1 | 30 | 2 | <1 | 2 | 189.5 | 100% | 100% |
| 1 | 150 | 2 | <1 | 2 | 389 | 100% | 100% |
| 2 | 30 | 30 | <1 | 94 | 161 | 100% | 100% |
| 2 | 150 | 154 | <1 | 387 | 267.48 | 100% | 100% |
| 3 | 30 | <1 | <1 | 94 | 127.91 | 100% | 100% |
| 3 | 150 | <1 | <1 | 454 | 128 | 100% | 100% |

#### Plus

| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
| 1 | 30 | 1 | <1 | 2 | 151.5 | 100% | 100% |
| 2 | 30 | 30 | <1 | 94 | 157 | 100% | 100% |
| 3 | 30 | <1 | <1 | 94 | 128 | 100% | 100% |

### Event Batch Processing

#### OSS

| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
| 1 | 30 | 5 | 733.6 | 80% | 80% | 100% | 100% | 100% |
| 1 | 150 | 5 | 2967 | 40% | 40% | 40% | 40% | 40% |
| 2 | 30 | 371 | 57.32 | 100% | 100% | 100% | 100% | 100% |
| 2 | 150 | 1743 | 75.87 | 98.45% | 100% | 100% | 100% | 100% |
| 3 | 30 | 370 | 37.48 | 99.73% | 99.73% | 100% | 100% | 100% |
| 3 | 150 | 1808 | 40.18 | 99.94% | 99.94% | 99.94% | 99.94% | 100% |

#### Plus

| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
| 1 | 30 | 3 | 1170 | 66% | 66% | 100% | 100% | 100% |
| 2 | 30 | 370 | 58.79 | 100% | 100% | 100% | 100% | 100% |
| 3 | 30 | 370 | 41.32 | 99.73% | 99.73% | 100% | 100% | 100% |

## NumResources to Total Resources

| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Attached HTTPRoutes | Total Resources |
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|---------------------|-----------------|
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | 2x | <total> |
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 60 | 244 |
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 300 | 1204 |

> Note: Only 2x HTTPRoutes attach to the Gateway because the parentRef name in the `cafe-tls-redirect` HTTPRoute is incorrect. This has been fixed, but until https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 is fixed we can't actually run the test successfully.

## Observations

1. Reload time seems to have a increased slightly in a few instances, though time to ready is consistent if not faster.

2. Processing fewer batches overall due to improvements in resource event tracking. Overall processing time didn't change much, so the average increased due to fewer batches.

3. No errors in the logs.


## Future Improvements

Fix https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 to allow for 150 resource tests to properly run.
2 changes: 1 addition & 1 deletion tests/reconfig/scripts/cafe-routes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ metadata:
name: cafe-tls-redirect
spec:
parentRefs:
- name: gateway.networking.k8s.io/v1
- name: gateway
sjberman marked this conversation as resolved.
Show resolved Hide resolved
namespace: default
sectionName: http
hostnames:
Expand Down
Loading