feat: Add n8n-benchmark cli (no-changelog) #10410

tomi · 2024-08-14T14:13:15Z

Summary

Adds a new cli tool for executing benchmark tests against any n8n instance. The tool consists of following parts:

Test scenarios
- Consists of test data (workflow(s)) and k6 script to run the benchmark
- In the testScenarios folder
n8n-benchmark cli
- In the src folder
- Based on oclif
- Two commands:
  - list : lists all available test scenarios
  - run : runs all available test scenarios (or specified ones in the future)

Test scenario execution works as follows:

Setup: setup owner if needed, login
For each test scenario:
- Import the specified test data using n8n internal API
- Run the benchmark and print out results

This is the first step in creating an automated benchmark test setup. Next step is to
add the test environment setup and orchestration to run the benchmarks there.
The feasibility of the approach has been validated in CAT-26
See the linear project for more details

Example outputs:

List:

➜ ./bin/n8n-benchmark list
Available test scenarios:

         CodeNode : Workflow with a single code node that is triggered using a webhook
         SingleWebhook : A single webhook trigger that responds with a 200 status code

Run:

./bin/n8n-benchmark run
Waiting for n8n http://localhost:5678 to become online
Setting up owner
Owner already set up
Running scenario: CodeNode
Loading and importing test data
Executing test script

     ✓ is status 200

     checks.........................: 100.00% ✓ 9381       ✗ 0   
     data_received..................: 4.4 MB  73 kB/s
     data_sent......................: 910 kB  15 kB/s
     http_req_blocked...............: avg=4.12µs  min=0s     med=3µs     max=1.38ms   p(90)=5µs     p(95)=6µs    
     http_req_connecting............: avg=133ns   min=0s     med=0s      max=467µs    p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=15.84ms min=5.05ms med=15.12ms max=170.48ms p(90)=24.74ms p(95)=27.99ms
       { expected_response:true }...: avg=15.84ms min=5.05ms med=15.12ms max=170.48ms p(90)=24.74ms p(95)=27.99ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 9381
     http_req_receiving.............: avg=67.92µs min=7µs    med=49µs    max=11.34ms  p(90)=98µs    p(95)=125µs  
     http_req_sending...............: avg=17.49µs min=2µs    med=14µs    max=3.61ms   p(90)=26µs    p(95)=31µs   
     http_req_tls_handshaking.......: avg=0s      min=0s     med=0s      max=0s       p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=15.75ms min=4.96ms med=15.04ms max=170.43ms p(90)=24.65ms p(95)=27.89ms
     http_reqs......................: 9381    156.290412/s
     iteration_duration.............: avg=15.98ms min=5.2ms  med=15.23ms max=170.59ms p(90)=24.9ms  p(95)=28.2ms 
     iterations.....................: 9381    156.290412/s
     vus............................: 4       min=1        max=4 
     vus_max........................: 5       min=5        max=5 


Running scenario: SingleWebhook
Loading and importing test data
Executing test script

     ✓ is status 200

     checks.........................: 100.00% ✓ 15725      ✗ 0    
     data_received..................: 4.2 MB  71 kB/s
     data_sent......................: 1.6 MB  27 kB/s
     http_req_blocked...............: avg=5.29µs  min=0s     med=3µs    max=4.83ms   p(90)=5µs     p(95)=6µs    
     http_req_connecting............: avg=75ns    min=0s     med=0s     max=410µs    p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=9.36ms  min=1.45ms med=9.21ms max=164.77ms p(90)=14.56ms p(95)=15.53ms
       { expected_response:true }...: avg=9.36ms  min=1.45ms med=9.21ms max=164.77ms p(90)=14.56ms p(95)=15.53ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 15725
     http_req_receiving.............: avg=67.73µs min=7µs    med=52µs   max=4.43ms   p(90)=95µs    p(95)=129µs  
     http_req_sending...............: avg=19.92µs min=1µs    med=15µs   max=5.73ms   p(90)=25µs    p(95)=31µs   
     http_req_tls_handshaking.......: avg=0s      min=0s     med=0s     max=0s       p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=9.27ms  min=1.38ms med=9.12ms max=164.68ms p(90)=14.48ms p(95)=15.45ms
     http_reqs......................: 15725   262.022199/s
     iteration_duration.............: avg=9.52ms  min=1.55ms med=9.38ms max=164.85ms p(90)=14.68ms p(95)=15.66ms
     iterations.....................: 15725   262.022199/s
     vus............................: 4       min=1        max=4  
     vus_max........................: 5       min=5        max=5

Related Linear tickets, Github issues, and Community forum posts

CAT-25

Review / Merge checklist

PR title and summary are descriptive. (conventions)
Docs updated or follow-up ticket created.
Tests included.
PR Labeled with release/backport (if the PR is an urgent fix that needs to be backported)

Adds a new cli tool for executing benchmark tests against any n8n instance. The idea is to treat n8n as a black box and use webhook triggers to initiate workflow executions. The tool consists of following parts: - Test scenarios - Consists of test data (workflow(s)) and k6 script to run the benchmark - In the `testScenarios` folder - n8n-benchmark cli - In the `src` folder - Based on oclif - Two commands: - list : lists all available test scenarios - run : runs all available test scenarios (or specified ones in the future) Test scenario execution works as follows: - Setup: setup owner if needed, login - For each test scenario: - Import the specified test data using n8n internal API - Run the benchmark and print out results This is the first step in creating an automated benchmark test setup. Next step is to add the test environment setup and orchestration to run the benchmarks there. The feasibility of the approach has been validated in [CAT-26](https://linear.app/n8n/issue/CAT-26/spike-select-infra-to-run-the-benchmarks) See the [linear project](https://linear.app/n8n/project/n8n-benchmarking-suite-3e7679d176b4/overview) for more details

Continuation of #10410

packages/benchmark/src/n8nApiClient/n8nApiClient.ts

packages/benchmark/src/commands/list.ts

packages/benchmark/src/n8nApiClient/workflowsApiClient.ts

packages/benchmark/testScenarios/singleWebhook/singleWebhook.script.ts

packages/benchmark/Dockerfile

packages/benchmark/README.md

packages/benchmark/testScenarios/codeNode/codeNode.manifest.json

packages/benchmark/src/n8nApiClient/n8nApiClient.ts

packages/benchmark/src/n8nApiClient/n8nApiClient.types.ts

packages/benchmark/src/n8nApiClient/workflowsApiClient.ts

packages/benchmark/tsconfig.json

ivov · 2024-08-20T08:40:32Z

Forgot to mention, the PR title shouldn't contain ticket ID. This is validated by the PR title GH action but needs updating so it wasn't caught.

packages/benchmark/package.json

RicardoE105 · 2024-08-20T11:40:12Z

packages/benchmark/package.json

+  "main": "dist/index",
+  "scripts": {
+    "build": "tsc -p tsconfig.build.json && tsc-alias -p tsconfig.build.json",
+    "start": "./bin/n8n-benchmark",


Are we adding prettier and the .vscode folder to make sure prettier is used to format and to format on save by default?

Not sure I follow. Do we have some prettier config in .vscode? We have the .prettierrc.js in the repo root, which AFAIK should apply to all packages?

I meant the formatter configuration in vscode as in here -> https://github.com/n8n-io/ai-assistant-service/blob/main/.vscode/settings.json

Think we have those already. I think it's a better pattern to provide "default" settings for the repository, but allow devs to customize them however they want. Should probably do it the same way in the AI Assistant service

tomi · 2024-08-20T11:51:53Z

@ivov Thank you for a great review 🙏 Addressed the issues you raised. LMK if I missed something

tomi · 2024-08-20T18:21:10Z

Can we use the new watch flag that comes with node out of the box?

@RicardoE105 the nodemon was actually a leftover which is not needed anymore. tsc and tsc-alias both provide watch out of the box

ivov

Nice work! Final thoughts, no blockers, so approving.

When the path to the k6 executable is not found, it'd be nice to fall back to /opt/homebrew/bin/k6. Most of the team is on mac and will be using homebrew when running the cli locally.
Now that we have the @n8n/config package I tend to think all monorepo config should go in there, even if as in this case it's a CLI tool rather than the app itself, but that's only my opinion. The purpose of @n8n/config is to have full type safety, better inline docs for DX, and to allow us to build tooling for all env vars for the docs team.
In future I'd love to see some documentation on (1) best practices to create a benchmark scenario, and (2) how best to make sense of the results reported.

K6_PATH=/opt/homebrew/bin/k6 N8N_USER_EMAIL=ivan@n8n.io N8N_USER_PASSWORD=test1234A ./bin/n8n-benchmark run
Waiting for n8n http://localhost:5678 to become online
Setting up owner
Owner already set up
Running scenario: SingleWebhook
Loading and importing data
Executing scenario script

     ✓ is status 200

     checks.........................: 100.00% ✓ 9161       ✗ 0
     data_received..................: 2.5 MB  41 kB/s
     data_sent......................: 934 kB  16 kB/s
     http_req_blocked...............: avg=1.63µs  min=0s     med=1µs     max=452µs    p(90)=2µs     p(95)=3µs
     http_req_connecting............: avg=82ns    min=0s     med=0s      max=208µs    p(90)=0s      p(95)=0s
     http_req_duration..............: avg=16.31ms min=4.66ms med=15.17ms max=332.68ms p(90)=23.01ms p(95)=26.76ms
       { expected_response:true }...: avg=16.31ms min=4.66ms med=15.17ms max=332.68ms p(90)=23.01ms p(95)=26.76ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 9161
     http_req_receiving.............: avg=26.8µs  min=7µs    med=27µs    max=244µs    p(90)=41µs    p(95)=48µs
     http_req_sending...............: avg=6.99µs  min=1µs    med=7µs     max=230µs    p(90)=11µs    p(95)=12µs
     http_req_tls_handshaking.......: avg=0s      min=0s     med=0s      max=0s       p(90)=0s      p(95)=0s
     http_req_waiting...............: avg=16.27ms min=4.62ms med=15.15ms max=332.62ms p(90)=22.98ms p(95)=26.72ms
     http_reqs......................: 9161    152.648771/s
     iteration_duration.............: avg=16.37ms min=4.73ms med=15.22ms max=332.81ms p(90)=23.04ms p(95)=26.82ms
     iterations.....................: 9161    152.648771/s
     vus............................: 4       min=1        max=4
     vus_max........................: 5       min=5        max=5

cypress · 2024-08-22T08:08:08Z

Test summary

414 • 0 • 0 • 0 • 0

Run details


Project	n8n
Status	Passed
Commit	`3823de6`
Started	Aug 22, 2024 8:03 AM
Ended	Aug 22, 2024 8:08 AM
Duration	04:46 💡
OS	Linux Debian -
Browser	Electron 118

View run in Cypress Cloud ➡️

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Cloud

github-actions · 2024-08-22T08:08:09Z

✅ All Cypress E2E specs passed

tomi · 2024-08-22T08:33:32Z

@ivov thank you for the review 🙏 I'll address the comments in follow-up PRs

* master: ci: Fix benchmark cli path (no-changelog) (#10506) refactor(core): Standardize filenames in `cli` (no-changelog) (#10484) fix(AI Agent Node): Allow AWS Bedrock Chat to be used with conversational agent (#10489) feat(AI Agent Node): Add tutorial link to agent node (#10493) feat: Add n8n-benchmark cli (no-changelog) (#10410) feat(core): Logout should invalidate the auth token (no-changelog) (#10335) refactor(editor): Add types to importCurlEventBus (no-changelog) (#10497) refactor(editor): Add types to htmlEditorEventBus (no-changelog) (#10498) refactor(editor): Add types for dataPinningEventBus (no-changelog) (#10499) refactor(editor): Add types to codeNodeEditorEventBus (no-changelog) (#10501)

janober · 2024-08-28T11:48:46Z

Got released with n8n@1.57.0

tomi force-pushed the benchmark-suite-spike branch from 2af83e3 to cdb9308 Compare August 14, 2024 14:16

tomi changed the title ~~feat: Add n8n-benchmark cli~~ feat: Add n8n-benchmark cli (CAT-25) (no-changelog) Aug 14, 2024

n8n-assistant bot added the n8n team Authored by the n8n team label Aug 14, 2024

tomi added a commit that referenced this pull request Aug 15, 2024

ci: Add workflow to build benchmark cli

3db871b

Continuation of #10410

tomi added a commit that referenced this pull request Aug 15, 2024

ci: Add workflow to build benchmark cli

00bfe9e

Continuation of #10410

tomi mentioned this pull request Aug 15, 2024

ci: Add workflow to build benchmark cli (no-changelog) #10441

Merged

4 tasks

ivov self-requested a review August 19, 2024 08:58

RicardoE105 reviewed Aug 19, 2024

View reviewed changes

packages/benchmark/src/n8nApiClient/n8nApiClient.ts Outdated Show resolved Hide resolved

RicardoE105 reviewed Aug 19, 2024

View reviewed changes

packages/benchmark/src/commands/list.ts Outdated Show resolved Hide resolved

packages/benchmark/src/n8nApiClient/workflowsApiClient.ts Outdated Show resolved Hide resolved

packages/benchmark/testScenarios/singleWebhook/singleWebhook.script.ts Outdated Show resolved Hide resolved

tomi added 2 commits August 20, 2024 10:55

ci: Add workflow to build benchmark cli (no-changelog) (#10441)

df231f3

docs(benchmark): Update readme

5de4646

tomi mentioned this pull request Aug 20, 2024

feat: Benchmark env with run scripts (no-changelog) #10477

Merged

4 tasks

ivov reviewed Aug 20, 2024

View reviewed changes

tomi added 12 commits August 20, 2024 13:19

refactor: Specify dockerfile version

e72fa26

docs: Update readme

efb4ba1

chore: Remove nodemon config

d47b8d8

chore: Update package metadata

ffd1005

refactor: Rename test scenarios to benchmark scenarios

98381c4

refactor: Rename method

cbe6869

fix: Remove payload from delete

35e5b33

refactor: Code style changes

30433a6

fix: Fix scenario manifest after renaming

cf08002

chore: Remove code node scenario

0ec1af1

refactor: Repeat api endpoint

e5b629e

chore: Remove unused ts compiler options

664bfb4

RicardoE105 reviewed Aug 20, 2024

View reviewed changes

tomi added 2 commits August 20, 2024 14:44

refactor: Move benchmark under @n8n

3568e4e

revert: Revert n8n-workflow package usage

90db188

tomi changed the title ~~feat: Add n8n-benchmark cli (CAT-25) (no-changelog)~~ feat: Add n8n-benchmark cli (no-changelog) Aug 20, 2024

tomi requested review from ivov and RicardoE105 August 20, 2024 18:21

tomi added 2 commits August 21, 2024 09:02

Merge branch 'master' into benchmark-suite-spike

edcf542

Merge branch 'master' into benchmark-suite-spike

3823de6

ivov approved these changes Aug 22, 2024

View reviewed changes

tomi merged commit ea6ca04 into master Aug 22, 2024
33 checks passed

tomi deleted the benchmark-suite-spike branch August 22, 2024 08:33

janober added the Released label Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add n8n-benchmark cli (no-changelog) #10410

feat: Add n8n-benchmark cli (no-changelog) #10410

tomi commented Aug 14, 2024

ivov commented Aug 20, 2024

RicardoE105 Aug 20, 2024

tomi Aug 20, 2024

RicardoE105 Aug 20, 2024

tomi Aug 22, 2024 •

edited

Loading

tomi commented Aug 20, 2024

tomi commented Aug 20, 2024

ivov left a comment

cypress bot commented Aug 22, 2024

github-actions bot commented Aug 22, 2024

tomi commented Aug 22, 2024

janober commented Aug 28, 2024

feat: Add n8n-benchmark cli (no-changelog) #10410

feat: Add n8n-benchmark cli (no-changelog) #10410

Conversation

tomi commented Aug 14, 2024

Summary

Related Linear tickets, Github issues, and Community forum posts

Review / Merge checklist

ivov commented Aug 20, 2024

RicardoE105 Aug 20, 2024

Choose a reason for hiding this comment

tomi Aug 20, 2024

Choose a reason for hiding this comment

RicardoE105 Aug 20, 2024

Choose a reason for hiding this comment

tomi Aug 22, 2024 • edited Loading

Choose a reason for hiding this comment

tomi commented Aug 20, 2024

tomi commented Aug 20, 2024

ivov left a comment

Choose a reason for hiding this comment

cypress bot commented Aug 22, 2024

Test summary

Run details

github-actions bot commented Aug 22, 2024

tomi commented Aug 22, 2024

janober commented Aug 28, 2024

tomi Aug 22, 2024 •

edited

Loading