Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Refactor storage using triggers #174

Merged
merged 42 commits into from
Mar 31, 2022
Merged

Conversation

flemzord
Copy link
Member

@flemzord flemzord commented Mar 18, 2022

Replace #168 Refactor storage.

This PR add triggers on "transactions" table to automatically compute balances and volumes of accounts in "balances" and "volumes" tables.
It also create entries in a new table called "accounts". This table will replace "addresses" table.
Now, transaction postings are directly persist as json in the transactions table, as well as metadata.
Account metadata are persisted in "accounts" table, also with json type.

Previously, the GET /accounts was not retrieving accounts which was not part of a transaction.
The new table "accounts" fix defacto this issue.

Also, there is another new table, called "log" which is designed to model an event based store. The code now always write to this table, then a set of triggers generates
entries in the transaction table, or updates metadata of either accounts or transactions following the log "type" column.
The hash on transactions was removed and add on "log" table. As a result we have a complete hash chain taking in account, metadata set on transactions or accounts.
To complete the modification on hash, the "id" property of transactions has been changed from id to a string, and is set to be a uuid by the code.
Old transactions will not have a uuid , just a int to string convert is done by the migration, but since the id is an opaque string, it should not cause any problem. A column named "ord" has been added to keep transactions ordering without the need to read all logs. It is generate by triggers, and not by the code.

@flemzord flemzord requested a review from a team as a code owner March 18, 2022 15:34
@flemzord flemzord requested review from gfyrag and removed request for a team March 18, 2022 15:34
@flemzord flemzord requested a review from altitude March 18, 2022 15:37
@codecov
Copy link

codecov bot commented Mar 18, 2022

Codecov Report

Merging #174 (f96edb0) into main (d3f7ef0) will decrease coverage by 12.83%.
The diff coverage is 63.76%.

❗ Current head f96edb0 differs from pull request most recent head bc39b2e. Consider uploading reports for the commit bc39b2e to get more accurate results

@@             Coverage Diff             @@
##             main     #174       +/-   ##
===========================================
- Coverage   73.21%   60.37%   -12.84%     
===========================================
  Files          83       81        -2     
  Lines        3338     3541      +203     
===========================================
- Hits         2444     2138      -306     
- Misses        713     1212      +499     
- Partials      181      191       +10     
Impacted Files Coverage Δ
pkg/api/api.go 100.00% <ø> (ø)
pkg/api/controllers/controllers.go 100.00% <ø> (ø)
pkg/api/controllers/transaction_controller.go 11.11% <0.00%> (-41.89%) ⬇️
pkg/api/middlewares/ledger_middleware.go 57.89% <0.00%> (-3.22%) ⬇️
pkg/bus/message.go 71.42% <ø> (ø)
cmd/storage.go 16.26% <14.72%> (-83.74%) ⬇️
pkg/storage/driver.go 28.57% <28.57%> (ø)
pkg/api/internal/testing.go 21.73% <33.33%> (-76.53%) ⬇️
pkg/storage/storage.go 47.45% <50.00%> (-16.95%) ⬇️
pkg/storage/cache.go 53.84% <55.55%> (-10.44%) ⬇️
... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b1bb1f...bc39b2e. Read the comment docs.

@flemzord
Copy link
Member Author

flemzord commented Mar 18, 2022

Todo

  • TxID UUID or Incremental ID ?
  • Command cli for update schema ?
  • Create documentation for Upgrade Guide

sqlstorage.DB represents the entire set of ledgers.
sqlstorage.Schema represents a specific ledger.

In case of postgresql, sqlstorage.DB implement is a simple wrapper around *sql.DB struct, and Schema is a PostgresQL schema.
In case of sqlite, sqlstorage.DB is the directory holdings all the underlygin db files, and the Schema is the concrete *sql.DB.
We already have an abstraction over storage.Driver. No need to have twice.
Add following tables :
* log
* account
* volumes
* balances

Now, we only write on the "log" table.
There is a set of triggers which are in charge of filling tables when a write is make on "log" table.
Thus, the hash is now on logs instead of transactions.
This way the hash take in account, metadata applied to transactions or accounts.
The automatic migration fill this log table with the existing data.

Since, the new code use some json function, the code must be compiled with "json1" build tag to enable support on sqlite.
Add following tables :
* log
* account
* volumes
* balances

Now, we only write on the "log" table.
There is a set of triggers which are in charge of filling tables when a write is make on "log" table.
Thus, the hash is now on logs instead of transactions.
This way the hash take in account, metadata applied to transactions or accounts.
The automatic migration fill this log table with the existing data.

Since, the new code use some json function, the code must be compiled with "json1" build tag to enable support on sqlite.
@gfyrag gfyrag force-pushed the feature/num-427-ledger-db-triggers branch from cf2a5f0 to bb94821 Compare March 28, 2022 20:18
@gfyrag gfyrag force-pushed the feature/num-427-ledger-db-triggers branch from 425238d to c169c2e Compare March 30, 2022 09:22
@gfyrag gfyrag force-pushed the feature/num-427-ledger-db-triggers branch from 15b695f to bc39b2e Compare March 30, 2022 11:43
@gfyrag gfyrag merged commit b69f119 into main Mar 31, 2022
@gfyrag gfyrag deleted the feature/num-427-ledger-db-triggers branch March 31, 2022 11:55
flemzord pushed a commit that referenced this pull request Jun 7, 2023
flemzord pushed a commit that referenced this pull request Sep 20, 2023
flemzord pushed a commit that referenced this pull request Dec 4, 2023
flemzord pushed a commit that referenced this pull request Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants