Limit simultaneous downloads by ip address #39
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…urrent download state)
…istory to comply with privacy policy.
…es the download early.
…esn't continue reading all of file from Merritt on a download cancel by user
|
Thanks, Scott! This looks great, and works well in my testing. I corrected one minor rubocop issue. |
|
Not merging yet, since there is still a testing issue with counter_stat_spec. |
…vs %Y. It was producing 2020-52 instead of 2019-52 for the last week of last year.
|
Woot! Got tests to pass. It was some weird thing where it was calculating the year and week strangely (2020-week 52) instead of (2019-52). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See datadryad/dryad-product-roadmap#613 .
I've added testing where I can but this is a bit difficult to test because of concurrency.
This is a good dataset for testing against since it has a large file/dataset that takes a while to download https://dryad-dev.cdlib.org/stash/dataset/doi:10.7959/dryad.8cz8w9j1 .
Essentially this logs downloads to the stash_engine_download_histories table. Create a record when a download starts. state is downloading or finished. Change to finished when completed. The difference between created_at and updated_at times is the download length. Write ip_address and user_agent to table and I hope this will make it easier for us to see the download activity and see what is going on instead of searching through logs.
Also added rake task and the other config repo adds it to daily cron to cleanup older downloads than 60 days and fix suspicious states that never go to finished (which mostly shouldn't happen unless there is an ungraceful shutdown of a crashed server or something).
Added tests .