Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new validation spec #204

Merged
merged 127 commits into from
Aug 6, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
00e981f
Rename custom error type
sandreae Jul 10, 2022
9d167b6
Create validation module
sandreae Jul 10, 2022
a3f4993
Helper validation method used in nextArgs
sandreae Jul 12, 2022
45d7762
GQL scalar type for DocumentViewId
sandreae Jul 12, 2022
0e1d431
Include new modules
sandreae Jul 12, 2022
59c545b
Refactor next_entry_args
sandreae Jul 12, 2022
719aff0
WIP: refactor publish entry
sandreae Jul 13, 2022
fd42e32
WIP: refactor publish entry
sandreae Jul 14, 2022
4955f03
WIP: nearly there with publish_entry
sandreae Jul 14, 2022
2a5ac7b
Introduce domain module
sandreae Jul 14, 2022
2d5f33d
Make more fine grained methods for validation
sandreae Jul 14, 2022
9defbeb
Remove duplicate insertion of operation from replication service
sandreae Jul 14, 2022
cc8315f
Fix skiplink and backlink orsering mistake
sandreae Jul 14, 2022
9c09f40
Comment out test which hangs
sandreae Jul 14, 2022
650eca1
Introduce test methods to be used instead of publish() and next_args()
sandreae Jul 14, 2022
9b88df6
Use cbor macro for more readabe test operations
sandreae Jul 15, 2022
6181393
update expected test hashes
sandreae Jul 15, 2022
3a5aa27
Split test_utils in store into seperate files
sandreae Jul 15, 2022
affba03
Don't validate operation against schema for now
sandreae Jul 15, 2022
fb2472a
Make existing tests pass
sandreae Jul 15, 2022
efb93b1
Comment out operation schema validation for now
sandreae Jul 15, 2022
9fa17c5
Test for ensure_log_id
sandreae Jul 15, 2022
10492e8
Update test const names
sandreae Jul 15, 2022
f607dd8
Commit Cargo.lock
sandreae Jul 15, 2022
e48f655
Use cbor macro for more readabe test operations
sandreae Jul 15, 2022
93c8cb3
Remove modules which accidentaly came across in cherry-pick
sandreae Jul 15, 2022
177aa2a
Update tests for LogId's which now start from 0
sandreae Jul 15, 2022
c6f0450
Update CHANGELOG
sandreae Jul 15, 2022
33c653c
Specify p2panda-rs dep by commit hash
sandreae Jul 15, 2022
171b811
Merge branch 'bump-p2panda' into implement-new-validation-spec
sandreae Jul 15, 2022
b0f77f8
Fixes after p2panda bump
sandreae Jul 15, 2022
0f79f2a
Test for ensure_entry_contains_expected_log_id
sandreae Jul 15, 2022
086cfb4
Split up data and api validation tests for publish_entry
sandreae Jul 15, 2022
a898538
Refactor verify seq_num, log_id, skiplink and backlink methods
sandreae Jul 16, 2022
13a37f3
Propergate panic from test runner back to test runtime
sandreae Jul 16, 2022
b7872ba
Rework seq and log id validation
sandreae Jul 16, 2022
826cdef
Test for seq num validation
sandreae Jul 16, 2022
d9efd22
Refactor verify_log_id and add tests
sandreae Jul 16, 2022
9e17e39
Update test values
sandreae Jul 16, 2022
ace08e5
get_expected_skiplink doesn't return option plus tests
sandreae Jul 16, 2022
30c844c
Don't return option from get_expected_backlink plus tests
sandreae Jul 16, 2022
724502d
A few more e2e tests
sandreae Jul 17, 2022
9d1ef5b
Make clippy happy
sandreae Jul 17, 2022
8d8d4d0
Merge branch 'main' into implement-new-validation-spec
sandreae Jul 17, 2022
fb75a75
Fix after merge
sandreae Jul 17, 2022
8bf492f
Tidy test publish and next_args methods
sandreae Jul 18, 2022
7b9fc52
Don't depend on DocumentStore in ensure_document_not_deleted
sandreae Jul 18, 2022
ba2b3b0
Remove unverified test versions of publish and next_args (whoop!)
sandreae Jul 18, 2022
69fec10
Tests for ensure_document_not_deleted
sandreae Jul 18, 2022
b4918d5
Publish new entry on service bus in replication service
sandreae Jul 18, 2022
ae62214
Remove unused deps
sandreae Jul 18, 2022
b2ac6d1
Remove unused deps
sandreae Jul 18, 2022
b8ba2a1
Add comment about code which will be deprecated after this PR
sandreae Jul 18, 2022
2cddfab
Remove param validation from next_args
sandreae Jul 18, 2022
1f893a2
Fix comment
sandreae Jul 18, 2022
c75d485
Implement get_latest_log_id on SqlStorage and then safely increment w…
sandreae Jul 18, 2022
a0f546c
Helper methods for safely incrementing seq numbers
sandreae Jul 18, 2022
9b01fa9
Validate passed parameters in gql query method
sandreae Jul 18, 2022
4a820f9
Refactor verify_seq_num
sandreae Jul 18, 2022
3b940f7
Move validation of entry and operation out of
sandreae Jul 18, 2022
4cc2dce
Add doc strings for next_args
sandreae Jul 18, 2022
1185b79
Remove determine_document_id helper
sandreae Jul 18, 2022
c241592
Add doc strings for `publish`
sandreae Jul 18, 2022
c7a4251
Fix test error string
sandreae Jul 18, 2022
ecf7a7d
Better seperation for pre-publish validation logic
sandreae Jul 18, 2022
a78938e
Some nice refactoring in next_args
sandreae Jul 19, 2022
adc8c72
Move location of determine next args in publish
sandreae Jul 19, 2022
d092f68
Implement SqlStorage using new associated type pattern
sandreae Jul 19, 2022
b9fcf8e
Make store implementation generic
sandreae Jul 19, 2022
dacc583
Correct test names
sandreae Jul 19, 2022
6c44896
Test for publish with missing skiplink using MemoryStore
sandreae Jul 19, 2022
f487f5d
Commit Cargo.lock
sandreae Jul 19, 2022
9185caa
Clippy & fmt
sandreae Jul 19, 2022
846bdc2
Clippy...
sandreae Jul 19, 2022
4ed3e00
Refactor one test
sandreae Jul 20, 2022
de5814d
Test for errors when calling publish
sandreae Jul 20, 2022
fbd881e
Refactor seq_num and backlink getter to avoid unnecessary db call
sandreae Jul 20, 2022
795bc1a
More cases for missing entrytests
sandreae Jul 20, 2022
ed6dabb
WIP: tests for publishing operations
sandreae Jul 20, 2022
e703350
Helpers for domain tests
sandreae Jul 20, 2022
69ff9ef
Update duplicate entry test
sandreae Jul 20, 2022
b0855f7
Tests for next_args
sandreae Jul 20, 2022
4066645
Remove some TODOs
sandreae Jul 20, 2022
7f16f48
Fix one test
sandreae Jul 20, 2022
c57e487
Remove println
sandreae Jul 20, 2022
79d1603
Remove deprecated request and response structs
sandreae Jul 20, 2022
40264c5
More tests for next args
sandreae Jul 21, 2022
6ee0348
Change test order
sandreae Jul 21, 2022
0cdb3a1
Remove wrapper method around bemboo verification
sandreae Jul 21, 2022
e6f7b49
Test for DocumentViewId GQL scalar
sandreae Jul 21, 2022
6f6f76d
Revert to using DocumentId in next_args GQL query plus test
sandreae Jul 21, 2022
f380d04
Doc strings in verify module
sandreae Jul 21, 2022
e831445
Tests for publishing to valid and invalid logs
sandreae Jul 21, 2022
210ca56
Deleted document tests
sandreae Jul 21, 2022
bd6b18d
Test for missing next skiplink
sandreae Jul 21, 2022
97410b4
Increment seq and log tests
sandreae Jul 21, 2022
3617e17
Fix expected error strings
sandreae Jul 21, 2022
f1629c4
Test for latest_log_id
sandreae Jul 22, 2022
bbdf88a
Update CHANGELOG
sandreae Jul 22, 2022
525a852
Docstring and comment review
cafca Jul 27, 2022
4b9cd6a
Clearer error messages
cafca Jul 27, 2022
6f92ff1
Remove commited test logging file
sandreae Jul 28, 2022
2dc24bb
License header
sandreae Jul 28, 2022
3c7ce39
Comment about updating nex_entry_args API
sandreae Jul 28, 2022
f2033ac
Typo
sandreae Jul 28, 2022
7afde14
Remove println
sandreae Jul 28, 2022
a459c02
Better SQL query for latest_log_id
sandreae Jul 28, 2022
91d852c
Correct comment string
sandreae Jul 28, 2022
1133212
Don't use nested imports
sandreae Jul 28, 2022
253550d
Correct comment
sandreae Jul 28, 2022
ce960f3
Move error into match statement
sandreae Jul 28, 2022
f0e68e2
Revert get_expected_skiplink ensure behaviour and improve error message
sandreae Jul 29, 2022
c2cac49
Another test for get_expected_skiplink
sandreae Jul 29, 2022
1d03e99
Target p2panda-rs main branch
sandreae Aug 1, 2022
1837911
Fix CHANGELOG.md formatting
adzialocha Aug 1, 2022
1a43a9f
Use new string methods and `Human` trait for display in errors
sandreae Aug 1, 2022
06377c7
Merge branch 'implement-new-validation-spec' of github.com:p2panda/aq…
sandreae Aug 1, 2022
b96c0bc
Error in publish when next args have MAX_SEQ_NUM with tests
sandreae Aug 1, 2022
4bf9160
Comments in test
sandreae Aug 1, 2022
95b564d
Rename function with get_checked prefix
sandreae Aug 1, 2022
57293b8
Merge branch 'main' into implement-new-validation-spec
sandreae Aug 1, 2022
0eaf75c
A little more clippy happy
sandreae Aug 1, 2022
501c029
Refactor `get_expected_skiplink()` (#220)
cafca Aug 6, 2022
fcacdf9
Use rev for p2panda-rs version
sandreae Aug 6, 2022
cf1924f
Group imports
sandreae Aug 6, 2022
5d16d80
Refactor initialize_db method
sandreae Aug 6, 2022
fd8a302
Use only one thread for tarpaulin
cafca Aug 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -203,11 +203,13 @@ jobs:
- name: Run cargo-tarpaulin
uses: actions-rs/tarpaulin@v0.1
with:
version: '0.20.1'
# Force cleaning via `--force-clean` flag to prevent buggy code coverage
args: >-
--manifest-path ${{ env.cargo_manifest }}
--locked
--force-clean
-- --test-threads=1
env:
# Ensure debug output is also tested
RUST_LOG: debug
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- GraphQL replication service gets and verifies new entries and inserts them into the db [#137](https://github.com/p2panda/aquadoggo/pull/137)
- `validation` and `domain` modules used for publish and next args API [#204](https://github.com/p2panda/aquadoggo/pull/204)
- Add schema task and schema provider that update when new schema views are materialised [#166](https://github.com/p2panda/aquadoggo/pull/166)

### Changed
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions aquadoggo/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ gql_client = "^1.0.6"
lipmaa-link = "^0.2.2"
log = "^0.4.17"
openssl-probe = "^0.1.5"
p2panda-rs = { git = "https://github.com/p2panda/p2panda", rev = "e06fd08c45253d60fcd42778f59e946a9ed73f71" }
p2panda-rs = { git = "https://github.com/p2panda/p2panda", rev = "5d6508d5a9b4b766621c3bd14879cc568fbac02d" }
serde = { version = "^1.0.137", features = ["derive"] }
sqlx = { version = "^0.6.0", features = [
"any",
Expand Down Expand Up @@ -61,7 +61,7 @@ hex = "0.4.3"
http = "^0.2.8"
hyper = "^0.14.19"
once_cell = "^1.12.0"
p2panda-rs = { git = "https://github.com/p2panda/p2panda", rev = "e06fd08c45253d60fcd42778f59e946a9ed73f71", features = [
p2panda-rs = { git = "https://github.com/p2panda/p2panda", rev = "5d6508d5a9b4b766621c3bd14879cc568fbac02d", features = [
"testing",
] }
rand = "^0.8.5"
Expand Down
1 change: 0 additions & 1 deletion aquadoggo/src/db/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ use sqlx::migrate::MigrateDatabase;
pub mod errors;
pub mod models;
pub mod provider;
pub mod request;
pub mod stores;
pub mod traits;
pub mod utils;
Expand Down
33 changes: 18 additions & 15 deletions aquadoggo/src/db/provider.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,14 @@ use p2panda_rs::operation::VerifiedOperation;
use p2panda_rs::schema::SchemaId;
use p2panda_rs::storage_provider::errors::OperationStorageError;
use p2panda_rs::storage_provider::traits::StorageProvider;
use p2panda_rs::test_utils::db::{
EntryArgsRequest, EntryArgsResponse, PublishEntryRequest, PublishEntryResponse,
};
use sqlx::query_scalar;

use crate::db::request::{EntryArgsRequest, PublishEntryRequest};
use crate::db::stores::{StorageEntry, StorageLog};
use crate::db::Pool;
use crate::errors::StorageProviderResult;
use crate::graphql::client::NextEntryArguments;
use crate::errors::Result;

/// Sql based storage that implements `StorageProvider`.
#[derive(Clone, Debug)]
Expand All @@ -31,21 +32,21 @@ impl SqlStorage {
/// A `StorageProvider` implementation based on `sqlx` that supports SQLite and PostgreSQL
/// databases.
#[async_trait]
impl StorageProvider<StorageEntry, StorageLog, VerifiedOperation> for SqlStorage {
type EntryArgsResponse = NextEntryArguments;
impl StorageProvider for SqlStorage {
type EntryArgsRequest = EntryArgsRequest;
type PublishEntryResponse = NextEntryArguments;
type EntryArgsResponse = EntryArgsResponse;
type PublishEntryRequest = PublishEntryRequest;
type PublishEntryResponse = PublishEntryResponse;
type StorageLog = StorageLog;
type StorageEntry = StorageEntry;
type StorageOperation = VerifiedOperation;

/// Returns the related document for any entry.
///
/// Every entry is part of a document and, through that, associated with a specific log id used
/// by this document and author. This method returns that document id by looking up the log
/// that the entry was stored in.
async fn get_document_by_entry(
&self,
entry_hash: &Hash,
) -> StorageProviderResult<Option<DocumentId>> {
async fn get_document_by_entry(&self, entry_hash: &Hash) -> Result<Option<DocumentId>> {
let result: Option<String> = query_scalar(
"
SELECT
Expand Down Expand Up @@ -81,7 +82,7 @@ impl SqlStorage {
pub async fn get_schema_by_document_view(
&self,
view_id: &DocumentViewId,
) -> StorageProviderResult<Option<SchemaId>> {
) -> Result<Option<SchemaId>> {
let result: Option<String> = query_scalar(
"
SELECT
Expand All @@ -92,7 +93,7 @@ impl SqlStorage {
document_view_id = $1
",
)
.bind(view_id.as_str())
.bind(view_id.to_string())
.fetch_optional(&self.pool)
.await
.map_err(|e| OperationStorageError::FatalStorageError(e.to_string()))?;
Expand Down Expand Up @@ -120,8 +121,10 @@ mod tests {
use crate::db::stores::test_utils::{test_db, TestDatabase, TestDatabaseRunner};
use crate::db::traits::DocumentStore;

use super::SqlStorage;

/// Inserts a `DocumentView` into the db and returns its view id.
async fn insert_document_view(db: &TestDatabase) -> DocumentViewId {
async fn insert_document_view(db: &TestDatabase<SqlStorage>) -> DocumentViewId {
let author = Author::try_from(db.test_data.key_pairs[0].public_key().to_owned()).unwrap();
let entry = db
.store
Expand Down Expand Up @@ -153,7 +156,7 @@ mod tests {
#[with(1, 1, 1)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let document_view_id = insert_document_view(&db).await;
let result = db
.store
Expand All @@ -172,7 +175,7 @@ mod tests {
#[with(1, 1, 1)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let result = db
.store
.get_schema_by_document_view(&random_document_view_id)
Expand Down
80 changes: 0 additions & 80 deletions aquadoggo/src/db/request.rs

This file was deleted.

35 changes: 18 additions & 17 deletions aquadoggo/src/db/stores/document.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ impl DocumentStore for SqlStorage {
($1, $2, $3)
",
)
.bind(document_view.id().as_str())
.bind(document_view.id().to_string())
.bind(value.id().as_str().to_owned())
.bind(name)
.execute(&self.pool)
Expand All @@ -71,8 +71,8 @@ impl DocumentStore for SqlStorage {
($1, $2)
",
)
.bind(document_view.id().as_str())
.bind(schema_id.as_str())
.bind(document_view.id().to_string())
.bind(schema_id.to_string())
.execute(&self.pool)
.await
.map_err(|e| DocumentStorageError::FatalStorageError(e.to_string()))?;
Expand Down Expand Up @@ -129,7 +129,7 @@ impl DocumentStore for SqlStorage {
operation_fields_v1.list_index ASC
",
)
.bind(id.as_str())
.bind(id.to_string())
.fetch_all(&self.pool)
.await
.map_err(|e| DocumentStorageError::FatalStorageError(e.to_string()))?;
Expand Down Expand Up @@ -173,9 +173,9 @@ impl DocumentStore for SqlStorage {
",
)
.bind(document.id().as_str())
.bind(document.view_id().as_str())
.bind(document.view_id().to_string())
.bind(document.is_deleted())
.bind(document.schema().as_str())
.bind(document.schema().to_string())
.execute(&self.pool)
.await
.map_err(|e| DocumentStorageError::FatalStorageError(e.to_string()))?;
Expand Down Expand Up @@ -283,7 +283,7 @@ impl DocumentStore for SqlStorage {
operation_fields_v1.list_index ASC
",
)
.bind(schema_id.as_str())
.bind(schema_id.to_string())
.fetch_all(&self.pool)
.await
.map_err(|e| DocumentStorageError::FatalStorageError(e.to_string()))?;
Expand Down Expand Up @@ -333,6 +333,7 @@ mod tests {
};
use rstest::rstest;

use crate::db::provider::SqlStorage;
use crate::db::stores::document::{DocumentStore, DocumentView};
use crate::db::stores::entry::StorageEntry;
use crate::db::stores::test_utils::{test_db, TestDatabase, TestDatabaseRunner};
Expand Down Expand Up @@ -373,7 +374,7 @@ mod tests {
#[with(1, 1, 1)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let author =
Author::try_from(db.test_data.key_pairs[0].public_key().to_owned()).unwrap();

Expand Down Expand Up @@ -435,7 +436,7 @@ mod tests {
#[with(1, 1, 1)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let view_does_not_exist = db
.store
.get_document_view_by_id(&random_document_view_id)
Expand All @@ -452,7 +453,7 @@ mod tests {
#[with(10, 1, 1, false, SCHEMA_ID.parse().unwrap(), vec![("username", OperationValue::Text("panda".into()))], vec![("username", OperationValue::Text("PANDA".into()))])]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let author =
Author::try_from(db.test_data.key_pairs[0].public_key().to_owned()).unwrap();
let schema_id = SchemaId::from_str(SCHEMA_ID).unwrap();
Expand Down Expand Up @@ -510,7 +511,7 @@ mod tests {
#[from(test_db)] runner: TestDatabaseRunner,
operation: Operation,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let document_view = DocumentView::new(
&document_view_id,
&DocumentViewFields::new_from_operation_fields(
Expand All @@ -534,7 +535,7 @@ mod tests {
#[with(1, 1, 1)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let document_id = db.test_data.documents[0].clone();

let document_operations = db
Expand Down Expand Up @@ -581,7 +582,7 @@ mod tests {
#[with(1, 1, 1)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let document_id = db.test_data.documents[0].clone();

let document_operations = db
Expand Down Expand Up @@ -628,7 +629,7 @@ mod tests {
#[with(10, 1, 1, true)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let document_id = db.test_data.documents[0].clone();

let document_operations = db
Expand All @@ -655,7 +656,7 @@ mod tests {
#[with(10, 1, 1, true)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let document_id = db.test_data.documents[0].clone();

let document_operations = db
Expand Down Expand Up @@ -686,7 +687,7 @@ mod tests {
#[with(10, 1, 1)]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let document_id = db.test_data.documents[0].clone();

let document_operations = db
Expand Down Expand Up @@ -721,7 +722,7 @@ mod tests {
#[with(10, 2, 1, false, SCHEMA_ID.parse().unwrap())]
runner: TestDatabaseRunner,
) {
runner.with_db_teardown(|db: TestDatabase| async move {
runner.with_db_teardown(|db: TestDatabase<SqlStorage>| async move {
let schema_id = SchemaId::from_str(SCHEMA_ID).unwrap();

for document_id in &db.test_data.documents {
Expand Down
Loading