Add async friendly upload

[ivergara]

Still a work in progress, but should be working in the local package store.

the add_package was not directly tested, only the add_files, but the upload doesn't use the add_files. Thus I added a very minimal test for add_package (with a bonus refactoring) and then went to turn it into an asynchronous function. aiofiles was already part of the dependencies but was never used until now, which provides a facade of synchronicity for FS local operations.

We can discuss some decisions, but first I wanted to make it work. Instead of saving the uploaded file with a random name and then renaming it, I'm writing it directly with the correct name. I tried the other way but didn't manage. Could give it another try if it's really critical.

Anyhow, would appreciate early feedback.

Edit:

I did end up keeping the original add_package because it's used in other areas and don't want to pollute the codebase with async/await keywords. Thus, added a separate async version for add_package.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1204438786781707

[codecov-commenter]

Codecov Report

Patch coverage: 73.68% and project coverage change: -0.94 ⚠️

Comparison is base (1f29a9c) 82.28% compared to head (3612703) 81.34%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #626      +/-   ##
==========================================
- Coverage   82.28%   81.34%   -0.94%     
==========================================
  Files          78       78              
  Lines        6124     6139      +15     
==========================================
- Hits         5039     4994      -45     
- Misses       1085     1145      +60     

Impacted Files	Coverage Δ
quetz/main.py	86.25% <0.00%> (ø)
quetz/pkgstores.py	43.51% <77.77%> (-10.49%)	⬇️

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[janjagusch]

This is a great first attempt, and I like the general structure of the change. I would be very interested in seeing implementations and tests for the cloud blob stores, especially GCS.

[ivergara]

This is a great first attempt, and I like the general structure of the change.

Thanks! I've tried to be as noninvasive as possible. However, besides passing the tests, we don't know yet if it's solving the originating issue.

I would be very interested in seeing implementations and tests for the cloud blob stores, especially GCS.

I'm also curious about how it'd look like.

Should we add the changes to activate testing against a GCS blob store in a separate PR?

[janjagusch]

Should we add the changes to activate testing against a GCS blob store in a separate PR?

Yes, let's put that in a separate PR.

[janjagusch]

@ivergara, can you rebase with main? That should give you access to the GCS secrets.

[janjagusch]

@ivergara, can you take care of the lint? then this should be good to go. 👍

[ivergara]

quetz/tests/test_pkg_stores.py::test_remove_dirs[local_store] PASSED [ 23%]
quetz/tests/test_pkg_stores.py::test_remove_dirs[gcs_store] PASSED [ 23%]
quetz/tests/test_pkg_stores.py::test_store_add_list_files[gcs_store] PASSED [ 23%]
quetz/tests/test_pkg_stores.py::test_move_file[gcs_store] PASSED [ 24%]
quetz/tests/test_pkg_stores.py::test_create_channel_multiple_times[gcs_store] PASSED [ 25%]

The tests against GCS did pass 🚀! Thus, at least, behavior wise it's working. Will fix the linting issue and then we're ready to merge and do a manual stress test.

EDIT:

However, the issue in the linting showed me that perhaps it was a false positive. Hope it's not the case.

[janjagusch]

awesome!