Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconfig test results for 1.2.0 #1718

Merged
merged 4 commits into from
Mar 19, 2024
Merged

Reconfig test results for 1.2.0 #1718

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
41e090a
Reconfig test results for 1.2.0
sjberman Mar 18, 2024
ac0a18b
Clear up summary
sjberman Mar 18, 2024
5dcc9a1
Fix bug phrasing
sjberman Mar 18, 2024
c98da1e
Add plus results
sjberman Mar 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions tests/reconfig/results/1.2.0/1.2.0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Reconfiguration testing Results

<!-- TOC -->
- [Reconfiguration testing Results](#reconfiguration-testing-results)
- [Summary](#summary)
- [Test environment](#test-environment)
- [Results Tables](#results-tables)
- [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready)
- [Event Batch Processing](#event-batch-processing)
- [NumResources to Total Resources](#numresources-to-total-resources)
- [Observations](#observations)
- [Future Improvements](#future-improvements)
<!-- TOC -->

## Summary

- Time to ready stayed consistent, if not slightly faster.
- Reload time has slightly increased in some instances.
- Number of batch events has reduced, subsequently increasing the size and time of each batch.

## Test environment

GKE cluster:

- Node count: 3
- Instance Type: e2-medium
- k8s version: 1.27.8-gke.1067004
- Zone: us-west2-a
- Total vCPUs: 6
- Total RAM: 12GB
- Max pods per node: 110

NGF deployment:

- NGF version: edge - git commit 96a44240d317875406a8aef8fd1e424f2fb906eb
- NGINX Version: 1.25.4
sjberman marked this conversation as resolved.
Show resolved Hide resolved

## Results Tables

> Note: After fixing the `cafe-tls-redirect` to point to the proper Gateway, tests that created 150 namespaces failed due to https://github.com/nginxinc/nginx-gateway-fabric/issues/1107. Therefore, those tests were re-run after reverting the `cafe-tls-redirect` issue to maintain consistency with the previous release tests. Going forward, results should look different once the above bug is fixed.
sjberman marked this conversation as resolved.
Show resolved Hide resolved

### NGINX Reloads and Time to Ready

| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
| 1 | 30 | 2 | <1 | 2 | 189.5 | 100% | 100% |
| 1 | 150 | 2 | <1 | 2 | 389 | 100% | 100% |
| 2 | 30 | 30 | <1 | 94 | 161 | 100% | 100% |
| 2 | 150 | 154 | <1 | 387 | 267.48 | 100% | 100% |
| 3 | 30 | <1 | <1 | 94 | 127.91 | 100% | 100% |
| 3 | 150 | <1 | <1 | 454 | 128 | 100% | 100% |

### Event Batch Processing

| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
| 1 | 30 | 5 | 733.6 | 80% | 80% | 100% | 100% | 100% |
| 1 | 150 | 5 | 2967 | 40% | 40% | 40% | 40% | 40% |
| 2 | 30 | 371 | 57.32 | 100% | 100% | 100% | 100% | 100% |
| 2 | 150 | 1743 | 75.87 | 98.45% | 100% | 100% | 100% | 100% |
| 3 | 30 | 370 | 37.48 | 99.73% | 99.73% | 100% | 100% | 100% |
| 3 | 150 | 1808 | 40.18 | 99.94% | 99.94% | 99.94% | 99.94% | 100% |

## NumResources to Total Resources

| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Attached HTTPRoutes | Total Resources |
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|---------------------|-----------------|
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | 2x | <total> |
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 60 | 244 |
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 300 | 1204 |

> Note: Only 2x HTTPRoutes attach to the Gateway because the parentRef name in the `cafe-tls-redirect` HTTPRoute is incorrect. This has been fixed, but until https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 is fixed we can't actually run the test successfully.

## Observations

1. Reload time seems to have a increased slightly in a few instances, though time to ready is consistent if not faster.

2. We appear to be processing fewer batches, especially for Test 1 where the resources exist before NGF. This subsequently increased the time to process each
batch since each batch was much larger. I don't think we changed any of this logic recently, so not really sure why it's happening. Test 1 number seems very different
sjberman marked this conversation as resolved.
Show resolved Hide resolved
from past results, while the other tests are only slightly different from past results.

3. No errors in the logs.


## Future Improvements

Fix https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 to allow for 150 resource tests to properly run.
2 changes: 1 addition & 1 deletion tests/reconfig/scripts/cafe-routes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ metadata:
name: cafe-tls-redirect
spec:
parentRefs:
- name: gateway.networking.k8s.io/v1
- name: gateway
sjberman marked this conversation as resolved.
Show resolved Hide resolved
namespace: default
sectionName: http
hostnames:
Expand Down
Loading