Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Sharding #633

Merged
merged 3 commits into from
Feb 2, 2023
Merged

feat: Sharding #633

merged 3 commits into from
Feb 2, 2023

Conversation

Mike-Dax
Copy link
Contributor

We have a growing list of stories to screenshot and we've had trouble scaling vertically on the one computer. Splitting the work up via multiple storybook servers has become hard to handle at scale. Instead we've added a feature to storycap to allow for 'sharding' on the screenshot end.

By adding --shard <shardNumber>/<totalShards> and running storycap on multiple computers, we can horizontally scale the screenshot process.

Stories are sorted by their IDs then distributed across shards in a round-robin style. We have some stories that rely on WebGL which are much slower than other simple components, since they're named similarly, this method reasonably distributes them across our workers.

For example, to run across two computers:

storycap --shard 1/2
storycap --shard 2/2

Merging the resultant screenshot artifacts is left to user, our CI solution makes that pretty easy and it seems out of the scope of storycap.

I hope this is helpful!

@Quramy Quramy merged commit a2e346b into reg-viz:master Feb 2, 2023
@Mike-Dax Mike-Dax deleted the sharding branch July 24, 2023 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants