Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add source to download multiple files via HTTP #546

Merged
merged 1 commit into from
Mar 6, 2023
Merged

add source to download multiple files via HTTP #546

merged 1 commit into from
Mar 6, 2023

Conversation

jonas-kaufmann
Copy link
Contributor

@jonas-kaufmann jonas-kaufmann commented Mar 4, 2023

Hi, for some experiments in https://github.com/se-sic/VaRA-Tool-Suite, we need to download multiple files over HTTP. This is currently a bit cumbersome as it requires us to specify multiple HTTP sources. @vulder had the idea to implement a source class that allows you to specify the url and then the files you want to download, like so:

class HTTPMultiple(
    local="geo-maps",
    remote={
        "1.0":
            "https://github.com/simonepri/geo-maps/releases/"
            "download/v0.6.0/"
    }
    files = ["countries-land-1km.geo.json", ...]
)

Let me know, what you think.

@simbuerg
Copy link
Member

simbuerg commented Mar 6, 2023

Hi,

sounds like a good idea to me. Thanks for working on this!

Copy link
Member

@simbuerg simbuerg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An override of version is required, to make sure you have access to the additional files in the build directory of the project.

benchbuild/source/http.py Show resolved Hide resolved
@simbuerg
Copy link
Member

simbuerg commented Mar 6, 2023

Had a further look. I should read my own code more... :-)
This is fine, you return the cache path as a directory. HTTP will take care of the copy to build directory for you.

LGTM.

@jonas-kaufmann
Copy link
Contributor Author

Had a further look. I should read my own code more... :-) This is fine, you return the cache path as a directory. HTTP will take care of the copy to build directory for you.

LGTM.

No problem but yes, that's what I intended :)

@simbuerg simbuerg merged commit 5d3c977 into PolyJIT:master Mar 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants