Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage blob copy start-batch needs "prefix" arg. Not pattern. #10924

Closed
worldspawn opened this issue Oct 21, 2019 · 2 comments
Closed

storage blob copy start-batch needs "prefix" arg. Not pattern. #10924

worldspawn opened this issue Oct 21, 2019 · 2 comments
Assignees
Labels
Storage az storage
Milestone

Comments

@worldspawn
Copy link

worldspawn commented Oct 21, 2019

Is your feature request related to a problem? Please describe.
When trying to use start-batch I wanted to copy all files beginning with a prefix. Prefix in this case is how prefix is applied with this command:

az storage blob list --container-name $container --account-name testbullclip --prefix $_.name

This appears to do a performant query against blob storage. I get results in seconds. Start-batch doesn't this. It has --pattern, what --pattern does I couldn't say but it appears to be scanning every single blob name. My container has 1.34M blobs making the implementation of --pattern of no use at all.

Describe the solution you'd like
Use whatever magic the blob list --prefix arg is using with a new arg for start-batch called --prefix.

Describe alternatives you've considered
I've tried using azcopy but it has the same sort of implementation with --include-pattern.

@mmyyrroonn mmyyrroonn added the Storage az storage label Oct 21, 2019
@Juliehzl Juliehzl added this to the S161 milestone Oct 22, 2019
@Juliehzl
Copy link
Contributor

Hi @worldspawn, thanks for your feedback. Two things to do in next Sprint:

  1. Find a way to add --prefix for az storage blob copy start-batch
  2. Integrate Azcopy v10.3 and add --include/exclude related flags to az storage copy.
    We may do both of the above improvements or one of them, but anyway we will provide a way to copy with prefix.

@Juliehzl
Copy link
Contributor

Since Azcopy v10.3 will bring some breaking changes, we plan to add breaking changes information in this sprint in PR #11546 and will do the change in next sprint.

@Juliehzl Juliehzl modified the milestones: S162, S163 Dec 11, 2019
@yonzhan yonzhan modified the milestones: S163, S164 Dec 31, 2019
@yonzhan yonzhan modified the milestones: S164, S165 Feb 1, 2020
@yonzhan yonzhan modified the milestones: S165, S166 Feb 15, 2020
@yonzhan yonzhan modified the milestones: S166, S167 Mar 5, 2020
@yonzhan yonzhan modified the milestones: S167, S168 Mar 26, 2020
@Juliehzl Juliehzl closed this as completed Apr 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Storage az storage
Projects
None yet
Development

No branches or pull requests

5 participants