Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add basic support for multiple storage backends. Closes #1245 #1255

Merged
merged 25 commits into from
Oct 11, 2019
Merged

Add basic support for multiple storage backends. Closes #1245 #1255

merged 25 commits into from
Oct 11, 2019

Conversation

brollb
Copy link
Contributor

@brollb brollb commented Oct 11, 2019

This includes basic support for configurable storage backends. This includes updates to:

  • Exporting branches (include the correct GME hashes)
  • Update ImportArtifact (transfer to storage location via GME blob)
  • Update to pipelines seed (and migration for existing data references)
  • Update to worker script (fetch and upload data on the workers)
  • Updates to visualizers (ArtifactIndex, etc)

@brollb
WIP #1245 add storage abstraction for different backends
a276421
@brollb
WIP Update pipeline seed to use userAsset attribute type
df00059
@brollb
WIP Update ImportArtifact to use Storage adapter
b903936
@brollb
WIP Add migration for data nodes
290fde5
@brollb
WIP Add getMetadata, getDownloadURL
b7c2869
@brollb
WIP Add storage location to ImportArtifact plugin
b970205
@brollb
WIP Updated createDataInfo to new method sig
01f7508
@brollb
Merge branch 'master' into 1245-custom-storage-backends
6140afb
@brollb
WIP #1245 update GenerateJob and generated files
9014cee
@brollb
WIP #1245 Update ArtifactIndex
80872f4
@brollb
WIP Upload data to storage location from the worker
4154330
@brollb
WIP #1245 Updated worker cache (restoring useBlob fn-ality)
b6c5a25
@brollb
WIP #1245 Only create results.json on successful completion
4924756
@brollb
WIP #1245 Update Export plugin
c9aedec
@brollb
Merge branch 'master' into 1245-custom-storage-backends
934d987
@brollb
WIP switched arg order for Storage
f08a9e8
@brollb
WIP #1245 update export branch functionality to collect correct assets
9929ff7
@brollb
WIP Set userAsset hashes correctly
18cd483
@brollb brollb merged commit 6482cb8 into master Oct 11, 2019
@brollb brollb deleted the 1245-custom-storage-backends branch October 11, 2019 21:46
@brollb brollb mentioned this pull request Oct 16, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement major
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@brollb