Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs.Glob skips directories while using .zip file system #816

Closed
mtojek opened this issue May 31, 2022 · 0 comments · Fixed by #817
Closed

fs.Glob skips directories while using .zip file system #816

mtojek opened this issue May 31, 2022 · 0 comments · Fixed by #817
Assignees
Labels
bug Something isn't working Team:Ecosystem Label for the Packages Ecosystem team

Comments

@mtojek
Copy link
Contributor

mtojek commented May 31, 2022

While I was working on using the Package Registry's model in the Package Storage Infra, I found a bug in handling fs.Glob. GetDataStreamPaths checks if there are any data stream directories in the .zip archive, but it may happen that there are no intermediate directories and the .zip is 100% correct.

$ unzip -l jobs/indexing/testdata/1password-0.1.1.zip
Archive:  jobs/indexing/testdata/1password-0.1.1.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      351  03-04-2022 14:43   1password-0.1.1/changelog.yml
     1344  03-04-2022 14:43   1password-0.1.1/data_stream/item_usages/agent/stream/httpjson.yml.hbs
     3560  03-04-2022 14:43   1password-0.1.1/data_stream/item_usages/elasticsearch/ingest_pipeline/default.yml
      592  03-04-2022 14:43   1password-0.1.1/data_stream/item_usages/fields/base-fields.yml
     3605  03-04-2022 14:43   1password-0.1.1/data_stream/item_usages/fields/ecs.yml
     1085  03-04-2022 14:43   1password-0.1.1/data_stream/item_usages/fields/fields.yml
     1619  03-04-2022 14:43   1password-0.1.1/data_stream/item_usages/manifest.yml
     2242  03-04-2022 14:43   1password-0.1.1/data_stream/item_usages/sample_event.json
     1348  03-04-2022 14:43   1password-0.1.1/data_stream/signin_attempts/agent/stream/httpjson.yml.hbs
     4081  03-04-2022 14:43   1password-0.1.1/data_stream/signin_attempts/elasticsearch/ingest_pipeline/default.yml
      596  03-04-2022 14:43   1password-0.1.1/data_stream/signin_attempts/fields/base-fields.yml
     4865  03-04-2022 14:43   1password-0.1.1/data_stream/signin_attempts/fields/ecs.yml
     1282  03-04-2022 14:43   1password-0.1.1/data_stream/signin_attempts/fields/fields.yml
     1628  03-04-2022 14:43   1password-0.1.1/data_stream/signin_attempts/manifest.yml
     2291  03-04-2022 14:43   1password-0.1.1/data_stream/signin_attempts/sample_event.json
     8657  03-04-2022 14:43   1password-0.1.1/docs/README.md
   450008  03-04-2022 14:43   1password-0.1.1/img/1password-itemusages-screenshot.png
     4441  03-04-2022 14:43   1password-0.1.1/img/1password-logo-light-bg.svg
   427378  03-04-2022 14:43   1password-0.1.1/img/1password-signinattempts-screenshot.png
     3814  03-04-2022 14:43   1password-0.1.1/kibana/dashboard/1password-item-usages-full-dashboard.json
     3872  03-04-2022 14:43   1password-0.1.1/kibana/dashboard/1password-signin-attempts-full-dashboard.json
     2214  03-04-2022 14:43   1password-0.1.1/kibana/map/1password-item-usages-source-IPs-map.json
     2227  03-04-2022 14:43   1password-0.1.1/kibana/map/1password-signin-attempts-source-IPs-map.json
     1065  03-04-2022 14:43   1password-0.1.1/kibana/search/1password-all-events.json
     1083  03-04-2022 14:43   1password-0.1.1/kibana/search/1password-item-usages.json
     1081  03-04-2022 14:43   1password-0.1.1/kibana/search/1password-signin-attempts.json
     1412  03-04-2022 14:43   1password-0.1.1/kibana/visualization/1password-item-usages-hot-items.json
     1715  03-04-2022 14:43   1password-0.1.1/kibana/visualization/1password-item-usages-hot-users.json
     1523  03-04-2022 14:43   1password-0.1.1/kibana/visualization/1password-item-usages-hot-vaults.json
     2333  03-04-2022 14:43   1password-0.1.1/kibana/visualization/1password-item-usages-over-time.json
     2859  03-04-2022 14:43   1password-0.1.1/kibana/visualization/1password-signin-attempts-categories-over-time.json
     2352  03-04-2022 14:43   1password-0.1.1/kibana/visualization/1password-signin-attempts-count-over-time.json
     1795  03-04-2022 14:43   1password-0.1.1/kibana/visualization/1password-signin-attempts-failed-gauge.json
     1735  03-04-2022 14:43   1password-0.1.1/kibana/visualization/1password-signin-attempts-hot-users.json
     2376  03-04-2022 14:43   1password-0.1.1/manifest.yml
---------                     -------

For the 1password-0.1.1.zip file, the logic assumes that there are no data streams, but you can unpack .zip file and all dirs will be put in right places.

@jsoriano I can try to adjust the logic, but would like to confirm if this is a bug.

@jlind23 jlind23 added the Team:Ecosystem Label for the Packages Ecosystem team label May 31, 2022
@jsoriano jsoriano self-assigned this May 31, 2022
@jsoriano jsoriano added the bug Something isn't working label May 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Team:Ecosystem Label for the Packages Ecosystem team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants