Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build ordering and mem requirements for tiles via a --tile-specifier-file optional param #71

Open
wants to merge 23 commits into
base: master
Choose a base branch
from

Conversation

tgrigsby-sc
Copy link
Contributor

@tgrigsby-sc tgrigsby-sc commented Apr 20, 2021

Still a WIP This works - verified manually with last tile build. I want to try one full build before I call this done which I will do next week.

  • Added --tile-specifier-file param to make_meta_tiles.py that points to an ordering file
  • Added TileSpecifier that uses the tile specifier file to dictate ordering and memory requirements of a list of tiles
  • Updated enqueue_tiles to make 1 call per job to tilequeue_batch_enqueue instead of a single call.
  • Updated enqueue_tiles to override every job with it's requested memory number
  • Improved logging and handled TileSpecifier creation exceptions by falling back to a no-op default

@tgrigsby-sc tgrigsby-sc requested a review from iandees April 20, 2021 19:29
@tgrigsby-sc tgrigsby-sc force-pushed the travisg/20210415-add-build-ordering branch from 255344d to 10b4769 Compare April 20, 2021 19:34
@tgrigsby-sc tgrigsby-sc force-pushed the travisg/20210415-add-build-ordering branch from 10b4769 to a4901c0 Compare April 20, 2021 19:35
@iandees
Copy link
Member

iandees commented Apr 20, 2021

Looks good so far.

@tgrigsby-sc tgrigsby-sc force-pushed the travisg/20210415-add-build-ordering branch from 3cd0e4d to 87c4ba3 Compare April 23, 2021 00:05
@tgrigsby-sc tgrigsby-sc changed the title WIP: Add build ordering and mem requirements for tiles via a --tile-specifier-file optional param Add build ordering and mem requirements for tiles via a --tile-specifier-file optional param Apr 23, 2021
@tgrigsby-sc tgrigsby-sc force-pushed the travisg/20210415-add-build-ordering branch from 0818757 to ffd4b9e Compare April 26, 2021 21:57
@tgrigsby-sc tgrigsby-sc force-pushed the travisg/20210415-add-build-ordering branch from 9149231 to 6d99e95 Compare May 7, 2021 21:47
@tgrigsby-sc tgrigsby-sc force-pushed the travisg/20210415-add-build-ordering branch from 6d99e95 to 5b7e092 Compare May 7, 2021 21:54
@tgrigsby-sc tgrigsby-sc force-pushed the travisg/20210415-add-build-ordering branch from 8ce9f99 to 1a6469d Compare May 18, 2021 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants