Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(bin/bench): rename 1-conn/10_000-1b-seq-resp to parallel #1796

Merged
merged 1 commit into from
Apr 8, 2024

Conversation

mxinden
Copy link
Collaborator

@mxinden mxinden commented Apr 7, 2024

The benchmark is using the default concurrency factor. By default neqo-client runs up to 100 requests in parallel.

#[arg(name = "concurrency", long, default_value = "100")]
/// The maximum number of requests to have outstanding at one time.
concurrency: usize,

Thus the benchmark name is wrong, i.e. the requests are run in parallel and not sequentially.


Sorry for the all noise for such a minor change.

The benchmark is using the default concurrency factor. By default `neqo-client`
runs up to `100` requests in parallel.

https://github.com/mozilla/neqo/blob/5dfe106669ccb695187511305c21b8e8a8775e91/neqo-bin/src/client/mod.rs#L151-L153

Thus the benchmark name is wrong, i.e. the requests are run in parallel and not
sequentially.
Copy link

codecov bot commented Apr 7, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.12%. Comparing base (5dfe106) to head (d9ca8e7).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1796   +/-   ##
=======================================
  Coverage   93.12%   93.12%           
=======================================
  Files         116      116           
  Lines       36097    36097           
=======================================
  Hits        33614    33614           
  Misses       2483     2483           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Apr 7, 2024

Benchmark results

Performance differences relative to 5dfe106.

  • coalesce_acked_from_zero 1+1 entries
    time: [197.27 ns 197.77 ns 198.30 ns]
    change: [-1.3207% -1.0148% -0.7051%] (p = 0.00 < 0.05)
    Change within noise threshold.

  • coalesce_acked_from_zero 3+1 entries
    time: [239.12 ns 239.75 ns 240.40 ns]
    change: [-0.6022% -0.2226% +0.1550%] (p = 0.25 > 0.05)
    No change in performance detected.

  • coalesce_acked_from_zero 10+1 entries
    time: [238.32 ns 239.27 ns 240.36 ns]
    change: [-1.1276% -0.5516% +0.0197%] (p = 0.06 > 0.05)
    No change in performance detected.

  • coalesce_acked_from_zero 1000+1 entries
    time: [219.62 ns 219.76 ns 219.93 ns]
    change: [-6.0924% -1.8384% +0.8039%] (p = 0.53 > 0.05)
    No change in performance detected.

  • RxStreamOrderer::inbound_frame()
    time: [119.22 ms 119.29 ms 119.38 ms]
    change: [-0.0292% +0.0670% +0.1669%] (p = 0.19 > 0.05)
    No change in performance detected.

  • transfer/Run multiple transfers with varying seeds
    time: [122.29 ms 122.56 ms 122.83 ms]
    thrpt: [32.566 MiB/s 32.636 MiB/s 32.708 MiB/s]
    change:
    time: [+2.1468% +2.4881% +2.8279%] (p = 0.00 < 0.05)
    thrpt: [-2.7501% -2.4277% -2.1017%]
    Change within noise threshold.

  • transfer/Run multiple transfers with the same seed
    time: [122.97 ms 123.13 ms 123.30 ms]
    thrpt: [32.442 MiB/s 32.486 MiB/s 32.530 MiB/s]
    change:
    time: [+1.8354% +2.0286% +2.2289%] (p = 0.00 < 0.05)
    thrpt: [-2.1803% -1.9883% -1.8023%]
    Change within noise threshold.

  • 1-conn/1-100mb-resp (aka. Download)/client
    time: [1.0990 s 1.1272 s 1.1595 s]
    thrpt: [86.247 MiB/s 88.717 MiB/s 90.989 MiB/s]
    change:
    time: [-0.7498% +3.2780% +7.2938%] (p = 0.14 > 0.05)
    thrpt: [-6.7979% -3.1740% +0.7554%]
    No change in performance detected.

  • 1-conn/10_000-parallel-1b-resp (aka. RPS)/client
    time: [383.28 ms 385.89 ms 388.48 ms]
    thrpt: [25.742 Kelem/s 25.914 Kelem/s 26.090 Kelem/s]

  • 100-seq-conn/1-1b-resp (aka. HPS)/client
    time: [3.3856 s 3.3884 s 3.3912 s]
    thrpt: [29.488 elem/s 29.512 elem/s 29.537 elem/s]
    change:
    time: [+0.1370% +0.2608% +0.3823%] (p = 0.00 < 0.05)
    thrpt: [-0.3809% -0.2601% -0.1368%]
    Change within noise threshold.

Client/server transfer results

Transfer of 134217728 bytes over loopback.

Client Server CC Pacing Mean [ms] Min [ms] Max [ms] Relative
msquic msquic 691.2 ± 232.3 445.7 1108.0 1.00
neqo msquic reno on 2083.2 ± 208.1 1913.8 2466.5 1.00
neqo msquic reno 2092.5 ± 335.5 1877.0 2851.7 1.00
neqo msquic cubic on 1965.2 ± 172.7 1777.3 2395.8 1.00
neqo msquic cubic 1955.7 ± 132.3 1785.1 2251.5 1.00
msquic neqo reno on 3400.4 ± 201.2 3162.8 3869.8 1.00
msquic neqo reno 3412.6 ± 293.8 3112.3 3914.1 1.00
msquic neqo cubic on 3321.2 ± 94.6 3232.5 3547.0 1.00
msquic neqo cubic 3334.9 ± 211.9 3188.3 3928.9 1.00
neqo neqo reno on 3048.8 ± 187.8 2842.8 3525.3 1.00
neqo neqo reno 3257.1 ± 214.4 3081.4 3653.3 1.00
neqo neqo cubic on 3298.1 ± 222.1 3053.0 3707.5 1.00
neqo neqo cubic 3250.3 ± 178.0 3087.8 3740.1 1.00

⬇️ Download logs

@larseggert larseggert added this pull request to the merge queue Apr 8, 2024
Merged via the queue into mozilla:main with commit aca1352 Apr 8, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants