Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: Prepare to show failure differences of QNS run relative to main #1856

Merged
merged 24 commits into from
May 2, 2024
Merged

ci: Prepare to show failure differences of QNS run relative to main #1856

merged 24 commits into from
May 2, 2024

Conversation

larseggert
Copy link
Collaborator

No description provided.

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 25, 2024 •
edited
Loading

Failed Interop Tests

QUIC Interop Runner, client vs. server

All results

QUIC Interop Runner, client vs. server

Succeeded Interop Tests

Unsupported Interop Tests

@codecov Codecov
Copy link

codecov bot commented Apr 25, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.25%. Comparing base (d4b3e4c) to head (673f7c8).

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #1856   +/-   ##
=======================================
  Coverage   93.25%   93.25%           
=======================================
  Files         110      110           
  Lines       35793    35793           
=======================================
  Hits        33377    33377           
  Misses       2416     2416

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 25, 2024 •
edited
Loading

Benchmark results

Performance differences relative to dcc88e3.

  • drain a timer quickly time: [311.58 ns 320.10 ns 328.07 ns]
    change: [-2.1662% +0.1458% +2.4025%] (p = 0.90 > 0.05)
    No change in performance detected.

  • coalesce_acked_from_zero 1+1 entries
    time: [194.15 ns 194.59 ns 195.06 ns]
    change: [-0.1565% +0.1264% +0.3851%] (p = 0.36 > 0.05)
    No change in performance detected.

  • coalesce_acked_from_zero 3+1 entries
    time: [237.73 ns 238.48 ns 239.26 ns]
    change: [-0.0584% +0.3481% +0.8186%] (p = 0.12 > 0.05)
    No change in performance detected.

  • coalesce_acked_from_zero 10+1 entries
    time: [235.86 ns 236.67 ns 237.64 ns]
    change: [-0.5607% -0.1960% +0.1607%] (p = 0.31 > 0.05)
    No change in performance detected.

  • coalesce_acked_from_zero 1000+1 entries
    time: [217.14 ns 222.38 ns 234.42 ns]
    change: [-1.0432% +2.5425% +9.1241%] (p = 0.63 > 0.05)
    No change in performance detected.

  • RxStreamOrderer::inbound_frame()
    time: [119.55 ms 119.60 ms 119.66 ms]
    change: [+1.0469% +1.1350% +1.2188%] (p = 0.00 < 0.05)
    💔 Performance has regressed.

  • transfer/Run multiple transfers with varying seeds
    time: [117.07 ms 117.33 ms 117.59 ms]
    thrpt: [34.017 MiB/s 34.091 MiB/s 34.167 MiB/s]
    change:
    time: [-1.1415% -0.8503% -0.5338%] (p = 0.00 < 0.05)
    thrpt: [+0.5367% +0.8576% +1.1546%]
    Change within noise threshold.

  • transfer/Run multiple transfers with the same seed
    time: [117.04 ms 117.24 ms 117.44 ms]
    thrpt: [34.061 MiB/s 34.118 MiB/s 34.177 MiB/s]
    change:
    time: [-1.1465% -0.8993% -0.6493%] (p = 0.00 < 0.05)
    thrpt: [+0.6535% +0.9074% +1.1598%]
    Change within noise threshold.

  • 1-conn/1-100mb-resp (aka. Download)/client
    time: [1.1129 s 1.1207 s 1.1290 s]
    thrpt: [88.572 MiB/s 89.233 MiB/s 89.857 MiB/s]
    change:
    time: [+0.9052% +1.9674% +3.0576%] (p = 0.00 < 0.05)
    thrpt: [-2.9669% -1.9294% -0.8971%]
    Change within noise threshold.

  • 1-conn/10_000-parallel-1b-resp (aka. RPS)/client
    time: [424.12 ms 426.37 ms 428.60 ms]
    thrpt: [23.332 Kelem/s 23.454 Kelem/s 23.578 Kelem/s]
    change:
    time: [-1.0615% -0.3529% +0.4017%] (p = 0.36 > 0.05)
    thrpt: [-0.4001% +0.3542% +1.0729%]
    No change in performance detected.

  • 1-conn/1-1b-resp (aka. HPS)/client
    time: [48.476 ms 48.927 ms 49.366 ms]
    thrpt: [20.257 elem/s 20.439 elem/s 20.629 elem/s]
    change:
    time: [-2.0246% -0.8994% +0.2042%] (p = 0.11 > 0.05)
    thrpt: [-0.2038% +0.9076% +2.0664%]
    No change in performance detected.

Client/server transfer results

Transfer of 134217728 bytes over loopback.

Client Server CC Pacing Mean [ms] Min [ms] Max [ms] Relative
msquic msquic 683.5 ± 327.5 378.1 1285.5 1.00
neqo msquic reno on 965.7 ± 376.3 757.6 1789.3 1.00
neqo msquic reno 877.4 ± 254.9 742.3 1587.6 1.00
neqo msquic cubic on 926.2 ± 235.3 732.7 1369.6 1.00
neqo msquic cubic 858.5 ± 176.9 746.0 1352.8 1.00
msquic neqo reno on 4428.6 ± 205.2 4129.4 4794.9 1.00
msquic neqo reno 4311.7 ± 224.3 4039.4 4770.7 1.00
msquic neqo cubic on 4476.9 ± 147.8 4257.5 4741.1 1.00
msquic neqo cubic 4400.7 ± 251.0 4049.1 4858.9 1.00
neqo neqo reno on 3273.4 ± 193.1 2991.2 3665.4 1.00
neqo neqo reno 3521.5 ± 178.9 3324.9 3883.3 1.00
neqo neqo cubic on 4337.9 ± 295.1 3958.9 4979.8 1.00
neqo neqo cubic 3894.1 ± 452.7 2796.5 4454.7 1.00

⬇️ Download logs

@larseggert larseggert mentioned this pull request Apr 29, 2024
Merged
@larseggert larseggert changed the title ci: Show failure differences of QNS run relative to main ci: Prepare to show failure differences of QNS run relative to main Apr 30, 2024
@larseggert larseggert marked this pull request as ready for review April 30, 2024 10:09
@larseggert
Copy link
Collaborator Author

larseggert commented Apr 30, 2024 •
edited
Loading

Need to commit this, so the QNS run for the main branch generates data for diffing. Will need a follow-on PR to actually generate and show the diff afterwards.

martinthomson
martinthomson reviewed May 1, 2024
View reviewed changes
Copy link
Member

@martinthomson martinthomson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The jq stuff makes me cringe, so I'm not reviewing that.

.github/workflows/qns.yml Show resolved Hide resolved
.github/workflows/qns.yml Outdated Show resolved Hide resolved
.github/workflows/qns.yml Show resolved Hide resolved
@larseggert larseggert added this pull request to the merge queue May 2, 2024
Merged via the queue into mozilla:main with commit 7070e7b May 2, 2024
48 checks passed
@larseggert larseggert deleted the ci-qns-diff branch May 2, 2024 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@martinthomson martinthomson martinthomson approved these changes

@KershawChang KershawChang Awaiting requested review from KershawChang KershawChang is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@larseggert @martinthomson