Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"blob" materializer task #493

Merged
merged 9 commits into from
Aug 22, 2023
Merged

"blob" materializer task #493

merged 9 commits into from
Aug 22, 2023

Conversation

sandreae
Copy link
Member

@sandreae sandreae commented Aug 8, 2023

Introduces a new "blob" task to the materializer service which:

  • is issued once a dependency task finishes on a blob_v1 or blob_piece_v1 document
  • if the document in question is part of a completed blob (all dependencies met, this isn't always true when a blob_piece_v1 document triggered the "blob" task) then we persist the blob view to the filesystem at the expected path ../<blob_document_id>/<blob_view_id>
  • we also create a symlink from the expected path to the blob when served by it's document id ../documents/<blob_document_id>`.

Next steps

  • would be good to have a blob module which contains all io stuff

📋 Checklist

  • Add tests that cover your changes
  • Add this PR to the Unreleased section in CHANGELOG.md
  • Link this PR to any issues it closes
  • New files contain a SPDX license header

@sandreae sandreae changed the base branch from main to blob-store August 8, 2023 16:30
@codecov
Copy link

codecov bot commented Aug 8, 2023

Codecov Report

Patch coverage: 6.03% and project coverage change: +0.26% 🎉

Comparison is base (81cae95) 91.20% compared to head (1633382) 91.47%.

❗ Current head 1633382 differs from pull request most recent head c2a0a36. Consider uploading reports for the commit c2a0a36 to get more accurate results

Additional details and impacted files
@@               Coverage Diff               @@
##           development     #493      +/-   ##
===============================================
+ Coverage        91.20%   91.47%   +0.26%     
===============================================
  Files              101      102       +1     
  Lines            16393    16353      -40     
===============================================
+ Hits             14952    14959       +7     
+ Misses            1441     1394      -47     
Files Changed Coverage Δ
aquadoggo/src/config.rs 46.80% <0.00%> (+11.88%) ⬆️
aquadoggo/src/materializer/tasks/blob.rs 0.00% <0.00%> (ø)
aquadoggo/src/materializer/service.rs 94.76% <100.00%> (+0.01%) ⬆️
aquadoggo/src/materializer/tasks/dependency.rs 97.56% <100.00%> (+0.02%) ⬆️

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sandreae sandreae changed the base branch from blob-store to development August 9, 2023 09:37
@sandreae sandreae marked this pull request as ready for review August 11, 2023 09:44
@sandreae sandreae linked an issue Aug 11, 2023 that may be closed by this pull request
Copy link
Member

@adzialocha adzialocha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Looking all good.

I would consider adding our own routes and handlers for the /blobs paths. Both for MIME types and helping with the symlinking workaround. I'm fine to handle this in a separate PR though.

@sandreae sandreae merged commit ba98934 into development Aug 22, 2023
8 checks passed
sandreae added a commit that referenced this pull request Aug 31, 2023
* Introduce blob materializer task

* fmt

* Materialize blobs to filesystem in "blob" task

* Issue a "blob" task when dependencies met.

* Some errors are not Critical errors

* fmt

* Use tempfile for data dir during tests

* Test for blob filesystem materialization

* fmt
@sandreae sandreae deleted the blob-materializer-task branch August 31, 2023 18:04
adzialocha added a commit that referenced this pull request Sep 8, 2023
* development:
  Make sure `/tmp` directory does not run out of scope before application ends (#557)
  Integrate `Bytes` value (#554)
  Stream blob data in chunks to files to not occupy too much memory (#551)
  Blobs directory configuration (#549)
  Use correct MAX_BLOB_PIECE_LENGTH from p2panda_rs
  Build a byte buffer over paginated pieces when assembling blobs (#547)
  HTTP routes to serve files with correct content type and etag headers (#544)
  Task for automatic garbage collection of unused documents and views (#500)
  Refactor tmp blob dir creation after rebase
  Fix after rebase
  "blob" materializer task (#493)
  Add static file server to `http` service (#483)
  Enable deletion of dangling `document_views` and related `document_view_fields` from db  (#491)
  BlobStore for retrieving raw blob data from the db (#484)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants