feat: implement SQL lite storage #72

Stefan-Ethernal · 2024-09-27T06:12:47Z

What does this PR do?

This PR introduces SQL lite storage to the zkevm-eth-txmanager, in case the PersistenceFilename config param is provided.

Reviewers

Main reviewers:

common/common.go

ethtxmanager/sqlstorage/sqlstorage.go

joanestebanr

I'm wondering if make sense to support dbTx?

ethtxmanager/sqlstorage/sqlstorage.go

ethtxmanager/memory_storage/memstorage.go

ethtxmanager/sqlstorage/migrations/0001.sql

ethtxmanager/sqlstorage/sqlstorage.go

Stefan-Ethernal · 2024-10-01T15:51:23Z

I'm wondering if make sense to support dbTx?

Can you elaborate some more? Do you mean to be able to do multiple changes in the same transaction (basically to ensure transactional mode)?

IMO, probably doesn't make too much sense, because it is only a single table, but I'm not sure (it depends of the usage). There might be some valid use case for it 🤔

joanestebanr · 2024-10-02T08:22:51Z

I'm wondering if make sense to support dbTx?

Can you elaborate some more? Do you mean to be able to do multiple changes in the same transaction (basically to ensure transactional mode)?

IMO, probably doesn't make too much sense, because it is only a single table, but I'm not sure (it depends of the usage). There might be some valid use case for it 🤔

I don't known that is done internally but even with just 1 table make sense:

You do multiples INSERTS but want to rollback if any of them fails
You do some INSERTS and check something over them in another function (that need to be in the same set of data)
Basically, it depends how works internally. But in the future make sense, that if we modify it, we think in that model because is the common in the rest of Database usage

Stefan-Ethernal · 2024-10-02T11:29:25Z

I don't known that is done internally but even with just 1 table make sense:

You do multiples INSERTS but want to rollback if any of them fails
You do some INSERTS and check something over them in another function (that need to be in the same set of data)
Basically, it depends how works internally. But in the future make sense, that if we modify it, we think in that model because is the common in the rest of Database usage

I discussed this with @goran-ethernal and @joanestebanr, there is no use case, where we would benefit from introducing transactional model. The transactions are affected one by one.

ethtxmanager/sqlstorage/sqlstorage.go

@joanestebanr

…ments from @joanestebanr (part 1)

…d part)

@joanestebanr

…ts from @joanestebanr (part 3)

…@joanestebanr

…@joanestebanr part 4)

ethtxmanager/sqlstorage/sqlstorage.go

@arnaubennassar

…dress comment from @arnaubennassar)

sonarqubecloud · 2024-10-02T20:17:55Z

Quality Gate failed

Failed conditions
66.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

* tag 'v0.2.1': fix if a tx is finalized very fast (0xPolygon#84) Rename config parameter `DBPath` to `StoragePath` (0xPolygon#77) feat: implement SQL lite storage (0xPolygon#72) Nonce too low issues fix (0xPolygon#75) fix nonce too low (0xPolygon#67) change namespace (0xPolygon#65) ensure tx order (0xPolygon#64) Quality Gate (0xPolygon#61) option to avoid gas estimationi

@joanestebanr

* option to avoid gas estimationi * Quality Gate (0xPolygon#61) * add linters * fix: lint fix * fix: lint make file * feat: sonar * fix: remove unused files * try to include coverage * fix: change sonar cloud project key * small fix * fix: comments --------- Co-authored-by: Rachit Sonthalia <rachitsonthalia02@gmail.com> * ensure tx order (0xPolygon#64) * ensure tx order * fix: linter * change namespace (0xPolygon#65) * fix nonce too low (0xPolygon#67) * Nonce too low issues fix (0xPolygon#75) * Serial nonce update (0xPolygon#74) * feat: prepare txns before routines * fix: remove forcedNonce feature * feat: uts * fix: lint * lint fix --------- Co-authored-by: Goran Rojovic <100121253+goran-ethernal@users.noreply.github.com> Co-authored-by: Goran Rojovic <goran.rojovic@ethernal.tech> * feat: implement SQL lite storage (0xPolygon#72) * feat: initial scaffolding * feat: insert implementation * feat: change column types based on the SQL lite supported types * feat: changed schema, first version of CRUD operations implementation * refactor: extract common parts to helper functions * chore: add comments * fix: linter config warning * refactor: simplify in-memory storage and fix deadlocks in case of errors * feat: instantiate either in memory or sql storage * chore: add comment * test: fully covered memstorage feat minor renames * feat: add github.com/mattn/go-sqlite3 as direct dependency * fix: adapt the DB schema, fix issues and add unit tests for sql storage * feat: increase coverage * test: add some more unit tests * fix: group variables * refactor: rename PersistenceFilename config parameter * fix: rename helper function (address comment part 1) * feat: introduce meddler * chore: remove useless comments * fix: revert update logic to manually constructing SQL and address comments from @joanestebanr (part 1) * fix: change some columns types (address comment from @joanestebanr 2nd part) * fix: use meddler to construct on fly update statement, address comments from @joanestebanr (part 3) * feat: remove in memory storage, change block_number to BIGINT (address @joanestebanr part 4) * fix: remove redundant code * feat: simplify building of UPDATE clause * rebase fix * refactor: build base select and base delete statement dynamically (address comment from @arnaubennassar) * test: concurrent write to the database unit test --------- Co-authored-by: Goran Rojovic <goran.rojovic@ethernal.tech> * Rename config parameter `DBPath` to `StoragePath` (0xPolygon#77) * fix if a tx is finalized very fast (0xPolygon#84) * fix * comment * # This is a combination of 3 commits. # This is the 1st commit message: Merge `dev` into upstream v0.2.1 # This is the commit message #2: Fix lint errs # This is the commit message #3: Comment interface method * parent 4da7722 author Vui-Chee <vuicheesiew@gmail.com> 1731548943 +0800 committer Vui-Chee <vuicheesiew@gmail.com> 1731548943 +0800 parent 1b8de9d author Vui-Chee <vuicheesiew@gmail.com> 1731476344 +0800 committer Vui-Chee <vuicheesiew@gmail.com> 1731548582 +0800 Merge `dev` into upstream v0.2.1 Fix lint errs Comment interface method Fix configure (#3) * fix configure * keep the same with genesis Silence sonar cloud go mod tidy Add back hex ut to keep consistent with `dev` --------- Co-authored-by: Arnau <arnaubennassar5@gmail.com> Co-authored-by: Goran Rojovic <100121253+goran-ethernal@users.noreply.github.com> Co-authored-by: Rachit Sonthalia <rachitsonthalia02@gmail.com> Co-authored-by: Toni Ramírez <58293609+ToniRamirezM@users.noreply.github.com> Co-authored-by: Stefan Negovanović <93934272+Stefan-Ethernal@users.noreply.github.com> Co-authored-by: Goran Rojovic <goran.rojovic@ethernal.tech> Co-authored-by: Alonso Rodriguez <ARR552@users.noreply.github.com>

Stefan-Ethernal self-assigned this Sep 28, 2024

Stefan-Ethernal requested review from ToniRamirezM, goran-ethernal and joanestebanr September 30, 2024 08:32

Stefan-Ethernal marked this pull request as ready for review September 30, 2024 08:32

goran-ethernal reviewed Sep 30, 2024

View reviewed changes

common/common.go Outdated Show resolved Hide resolved

ethtxmanager/sqlstorage/sqlstorage.go Outdated Show resolved Hide resolved

joanestebanr reviewed Oct 1, 2024

View reviewed changes

joanestebanr approved these changes Oct 2, 2024

View reviewed changes

Stefan-Ethernal force-pushed the CDK-501-seq-sender-change-current-json-file-ethtxmanager-json-for-a-database-sqlite branch from 26fc997 to d3db367 Compare October 2, 2024 12:15

goran-ethernal approved these changes Oct 2, 2024

View reviewed changes

ethtxmanager/sqlstorage/sqlstorage.go Outdated Show resolved Hide resolved

ethtxmanager/sqlstorage/sqlstorage.go Show resolved Hide resolved

ToniRamirezM approved these changes Oct 2, 2024

View reviewed changes

Stefan-Ethernal added 16 commits October 2, 2024 16:02

feat: initial scaffolding

8bf6b8d

feat: insert implementation

f3b7e20

feat: change column types based on the SQL lite supported types

ccc8cd6

feat: changed schema, first version of CRUD operations implementation

ee4ed66

refactor: extract common parts to helper functions

0d72a77

chore: add comments

86cb156

fix: linter config warning

5e6f3c3

refactor: simplify in-memory storage and fix deadlocks in case of errors

630487a

feat: instantiate either in memory or sql storage

4f23f0e

chore: add comment

d46a33f

test: fully covered memstorage feat minor renames

8fedf41

feat: add github.com/mattn/go-sqlite3 as direct dependency

693e97d

fix: adapt the DB schema, fix issues and add unit tests for sql storage

3a2b2dd

feat: increase coverage

b5eaccf

test: add some more unit tests

ed8276b

fix: group variables

257c81e

Stefan-Ethernal and others added 11 commits October 2, 2024 16:04

refactor: rename PersistenceFilename config parameter

463dfcc

fix: rename helper function (address comment part 1)

9223da9

feat: introduce meddler

c10403b

chore: remove useless comments

d344af8

fix: revert update logic to manually constructing SQL and address com…

f0604dc

…ments from @joanestebanr (part 1)

fix: change some columns types (address comment from @joanestebanr 2n…

3ea3dc7

…d part)

fix: use meddler to construct on fly update statement, address commen…

aa54d6a

…ts from @joanestebanr (part 3)

feat: remove in memory storage, change block_number to BIGINT (address …

8196f5c

…@joanestebanr part 4)

fix: remove redundant code

40c398d

feat: simplify building of UPDATE clause

5077963

rebase fix

db40fc9

goran-ethernal force-pushed the CDK-501-seq-sender-change-current-json-file-ethtxmanager-json-for-a-database-sqlite branch from 4c36837 to db40fc9 Compare October 2, 2024 14:27

arnaubennassar approved these changes Oct 2, 2024

View reviewed changes

ethtxmanager/sqlstorage/sqlstorage.go Outdated Show resolved Hide resolved

refactor: build base select and base delete statement dynamically (ad…

c009e71

…dress comment from @arnaubennassar)

Stefan-Ethernal force-pushed the CDK-501-seq-sender-change-current-json-file-ethtxmanager-json-for-a-database-sqlite branch from d136818 to c009e71 Compare October 2, 2024 19:54

test: concurrent write to the database unit test

669a8d7

Stefan-Ethernal merged commit ec887ce into main Oct 3, 2024
5 of 6 checks passed

Stefan-Ethernal deleted the CDK-501-seq-sender-change-current-json-file-ethtxmanager-json-for-a-database-sqlite branch October 3, 2024 06:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement SQL lite storage #72

feat: implement SQL lite storage #72

Stefan-Ethernal commented Sep 27, 2024 •

edited

Loading

joanestebanr left a comment

Stefan-Ethernal commented Oct 1, 2024 •

edited

Loading

joanestebanr commented Oct 2, 2024

Stefan-Ethernal commented Oct 2, 2024

sonarqubecloud bot commented Oct 2, 2024

feat: implement SQL lite storage #72

feat: implement SQL lite storage #72

Conversation

Stefan-Ethernal commented Sep 27, 2024 • edited Loading

What does this PR do?

Reviewers

joanestebanr left a comment

Choose a reason for hiding this comment

Stefan-Ethernal commented Oct 1, 2024 • edited Loading

joanestebanr commented Oct 2, 2024

Stefan-Ethernal commented Oct 2, 2024

sonarqubecloud bot commented Oct 2, 2024

Quality Gate failed

Stefan-Ethernal commented Sep 27, 2024 •

edited

Loading

Stefan-Ethernal commented Oct 1, 2024 •

edited

Loading