Skip to content

Commit

Permalink
Blobs directory configuration (#549)
Browse files Browse the repository at this point in the history
* Move configuration of blobs base path into client

* Rename variable in all places

* Update READMEs and fix 'its' spelling

* Add entry to CHANGELOG.md

* Add warnings when database is persisted but blobs are not
  • Loading branch information
adzialocha authored Sep 6, 2023
1 parent 7cbf5f5 commit 3140f68
Show file tree
Hide file tree
Showing 39 changed files with 225 additions and 181 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Add method to store for pruning document views [#491](https://github.com/p2panda/aquadoggo/pull/491)
- Introduce `BlobStore` [#484](https://github.com/p2panda/aquadoggo/pull/484)
- Task for automatic garbage collection of unused documents and views [#500](https://github.com/p2panda/aquadoggo/pull/500)
- Blobs directory configuration [#549](https://github.com/p2panda/aquadoggo/pull/549)

### Changed

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ If you are not working with Rust you can create FFI bindings from the `aquadoggo

As an application developer the interface to `aquadoggo` you are likely to use the most is the GraphQL query API. For whichever schema your node supports a custom query api is generated, you use this to fetch data into your app. Results from a collection query can be paginated, filtered.

Fetch one "mushroom" by it's id, returning values for only the selected fields:
Fetch one "mushroom" by its id, returning values for only the selected fields:
```graphql
{
mushroom: mushroom_0020c3accb0b0c8822ecc0309190e23de5f7f6c82f660ce08023a1d74e055a3d7c4d(
Expand Down
2 changes: 1 addition & 1 deletion aquadoggo/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@ sqlx = { version = "0.6.1", features = [
"sqlite",
"runtime-tokio-rustls",
] }
tempfile = "3.7.0"
thiserror = "1.0.39"
tokio = { version = "1.28.2", features = [
"macros",
Expand Down Expand Up @@ -112,5 +111,6 @@ rstest = "0.15.0"
rstest_reuse = "0.3.0"
serde_bytes = "0.11.12"
serde_json = "1.0.85"
tempfile = "3.7.0"
tower = "0.4.13"
tower-service = "0.3.2"
14 changes: 7 additions & 7 deletions aquadoggo/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,6 @@ use p2panda_rs::schema::SchemaId;

use crate::network::NetworkConfiguration;

/// Blobs directory name.
pub const BLOBS_DIR_NAME: &str = "blobs";

/// Configuration object holding all important variables throughout the application.
#[derive(Debug, Clone)]
pub struct Configuration {
Expand All @@ -26,9 +23,6 @@ pub struct Configuration {
/// _not_ recommended for production settings.
pub allow_schema_ids: AllowList<SchemaId>,

/// Path to blobs directory.
pub blob_dir: Option<PathBuf>,

/// URL / connection string to PostgreSQL or SQLite database.
pub database_url: String,

Expand All @@ -44,6 +38,12 @@ pub struct Configuration {
/// 2020.
pub http_port: u16,

/// Path to folder where blobs (binary files) are kept and served from.
///
/// **Warning**: When set to a temporary directory, make sure that also the database itself is
/// not persisted, otherwise you will run into data inconsistencies.
pub blobs_base_path: PathBuf,

/// Number of concurrent workers which defines the maximum of materialization tasks which can
/// be worked on simultaneously.
///
Expand All @@ -59,10 +59,10 @@ impl Default for Configuration {
fn default() -> Self {
Self {
allow_schema_ids: AllowList::Wildcard,
blob_dir: None,
database_url: "sqlite::memory:".into(),
database_max_connections: 32,
http_port: 2020,
blobs_base_path: PathBuf::new(),
worker_pool_size: 16,
network: NetworkConfiguration::default(),
}
Expand Down
6 changes: 3 additions & 3 deletions aquadoggo/src/db/stores/blob.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ const BLOB_QUERY_PAGE_SIZE: u64 = 10;
pub type BlobData = Vec<u8>;

impl SqlStore {
/// Get the data for one blob from the store, identified by it's document id.
/// Get the data for one blob from the store, identified by its document id.
pub async fn get_blob(&self, id: &DocumentId) -> Result<Option<BlobData>, BlobStoreError> {
// Get the root blob document
let blob_document = match self.get_document(id).await? {
Expand All @@ -36,7 +36,7 @@ impl SqlStore {
document_to_blob_data(self, blob_document).await
}

/// Get the data for one blob from the store, identified by it's document view id.
/// Get the data for one blob from the store, identified by its document view id.
pub async fn get_blob_by_view_id(
&self,
view_id: &DocumentViewId,
Expand All @@ -62,7 +62,7 @@ impl SqlStore {

// If there are no documents referring to the blob then we continue with the purge.
if blob_reverse_relations.is_empty() {
// Collect the document view ids of all pieces this blob has ever referred to in it's
// Collect the document view ids of all pieces this blob has ever referred to in its
// `pieces`
let blob_piece_ids: Vec<String> = query_scalar(
"
Expand Down
Loading

0 comments on commit 3140f68

Please sign in to comment.