Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build: Add benchmark task #22161

Merged
merged 57 commits into from
Jun 23, 2023
Merged

Build: Add benchmark task #22161

merged 57 commits into from
Jun 23, 2023

Conversation

shilman
Copy link
Member

@shilman shilman commented Apr 19, 2023

Closes #21511

What I did

I replaced our script for bench-marking storybook.
The @storybook/addon-bench is now deprecated, no longer used.
I've archived that repository.

I'm now tracking a bunch of more meaningful data about storybook:

export interface BenchResults {
branch: string;
commit: string;
timestamp: string;
label: string;
/** The time it takes to create the base sandbox without storybook */
createTime: number;
/** The time it takes to install the base sandbox after it has been initialized */
generateTime: number;
/** The time it takes to run `sb init` on the base sandbox */
initTime: number;
/** Size of base sandbox node_modules without storybook pre-install */
createSize: number;
/** Size of base sandbox node_modules without storybook post-install */
generateSize: number;
/** Size of the sandbox node_modules post `sb init` */
initSize: number;
/** Difference bewtween `initSize` and `generateSize` */
diffSize: number;
/** Full `sb build` time */
buildTime: number;
/** Size of the storybook-static directory in total */
buildSize: number;
/** Size of the storybook-static/sb-addons in total */
buildSbAddonsSize: number;
/** Size of the storybook-static/sb-common-assets */
buildSbCommonSize: number;
/** Size of the storybook-static/sb-manager */
buildSbManagerSize: number;
/** Size of storybook-static/sb-preview */
buildSbPreviewSize: number;
/** Size of the `static` directory if it exists */
buildStaticSize: number;
/** Total size of `sb-x` above */
buildPrebuildSize: number;
/** Total size of everything else (user's stories & components & CSS & assets etc.) */
buildPreviewSize: number;
/** Time to wait-on iframe.html */
devPreviewResponsive: number;
/** Time to wait-on index.html */
devManagerResponsive: number;
/** Time to browse to index.html and view the SB logo */
devManagerHeaderVisible: number;
/** Time to browse to index.html and load the story index */
devManagerIndexVisible: number;
/** Time to browse to index.html and load iframe content and the story is rendered */
devStoryVisible: number;
/** Time to browse to index.html and load iframe content and the docs page is rendered */
devDocsVisible: number;
/** Time to browse to index.html and view the SB logo */
buildManagerHeaderVisible: number;
/** Time to browse to index.html and load the story index */
buildManagerIndexVisible: number;
/** Time to browse to index.html and load iframe content and the story is rendered */
buildStoryVisible: number;
/** Time to browse to index.html and load iframe content and the docs page is rendered */
buildDocsVisible: number;
}

This should make it easier for us to detect regressions in performance and size, and also make it easier to verify if what we're doing has impact we think it should have.

How to test

When running sandboxes, a bench.json file should be created when tasks run.
Each task is responsible for tracking it's own metrics. This is to ensure we can expand this system over time.

I did make 1 bench task that collects data via playwright.

Checklist

  • Make sure your changes are tested (stories and/or unit, integration, or end-to-end tests)
  • Make sure to add/update documentation regarding your changes
  • If you are deprecating/removing a feature, make sure to update
    MIGRATION.MD

Maintainers

  • If this PR should be tested against many or all sandboxes,
    make sure to add the ci:merged or ci:daily GH label to it.
  • Make sure this PR contains one of the labels below.

["cleanup", "BREAKING CHANGE", "feature request", "bug", "documentation", "maintenance", "dependencies", "other"]

@ndelangen ndelangen added the build Internal-facing build tooling & test updates label Apr 19, 2023
@shilman shilman added the ci:pr label Apr 24, 2023
@socket-security
Copy link

socket-security bot commented Apr 24, 2023

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

Next steps

Take a deeper look at the dependency

Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support [AT] socket [DOT] dev.

Remove the package

If you happen to install a dependency that Socket reports as Known Malware you should immediately remove it and select a different dependency. For other alert types, you may may wish to investigate alternative packages or consider if there are other ways to mitigate the specific risk posed by the dependency.

Mark a package as acceptable risk

To ignore an alert, reply with a comment starting with @SocketSecurity ignore followed by a space separated list of package-name@version specifiers. e.g. @SocketSecurity ignore foo@1.0.0 bar@* or ignore all packages with @SocketSecurity ignore-all

@socket-security
Copy link

socket-security bot commented Jun 19, 2023

New and updated dependency changes detected. Learn more about Socket for GitHub ↗︎

Packages Version New capabilities Transitives1 Size Publisher
@types/wait-on 🆕 5.3.1 None +0 6.08 kB types
@types/http-server 🆕 0.12.1 None +0 5.06 kB types
pretty-ms 🆕 8.0.0 None +1 15.3 kB sindresorhus
pretty-bytes 🆕 6.1.0 None +0 11.3 kB sindresorhus
@google-cloud/bigquery 🆕 6.2.0 filesystem +12 1.72 MB google-wombot

Footnotes

  1. https://docs.socket.dev

@ndelangen ndelangen changed the title New benchmark task WIP Testing: New benchmark task Jun 19, 2023
@ndelangen
Copy link
Member

ndelangen commented Jun 19, 2023

@shilman It's green!

Let's discuss next step.

  • Shall I remove the previous bench tasks?
  • Shall I archive the @storybook/bench repo?
  • Is data going into bigQuery correctly?
  • Is this data not conflicting with existing data during the transition period (a lot of open PR's won't have this change)

@ndelangen
Copy link
Member

locator.innerText: Timeout 30000ms exceeded.

😢

@shilman
Copy link
Member Author

shilman commented Jun 23, 2023

@ndelangen there are still a bunch of values that are not getting filled in, but otherwise it is starting to look OK

image

@shilman shilman merged commit 162ba43 into next Jun 23, 2023
@shilman shilman deleted the shilman/bench-task branch June 23, 2023 14:38
@shilman shilman added the patch:yes Bugfix & documentation PR that need to be picked to main branch label Jun 23, 2023
@ndelangen
Copy link
Member

This is absolutely awesome! 🎉

shilman added a commit that referenced this pull request Jun 27, 2023
Build: Add benchmark task
(cherry picked from commit 162ba43)
@JReinhold JReinhold added the patch:done Patch/release PRs already cherry-picked to main/release branch label Jul 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Internal-facing build tooling & test updates patch:done Patch/release PRs already cherry-picked to main/release branch patch:yes Bugfix & documentation PR that need to be picked to main branch
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[Bug]: Composition - open storybook link contains /index.html
4 participants