feat: ADR 040: Implement in-memory DB backend #9952

roysc · 2021-08-17T08:53:41Z

Description

Implements an in-memory backend for the DB interface introduced by #9573 and specified by ADR-040. This expands on the btree-based MemDB from tm-db by using copy-on-write clones to implement versioning.

Resolves: vulcanize#2

Will move out of draft once #9573 is merged and rebased on.

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

redundant since dbWriter uses a clone

roysc · 2021-08-23T07:46:56Z

Now that the interface has been merged, this is good to review. This contains MemDB, unit tests, and supporting utilities, which logically fit together but make a large PR, so let me know if it would be better to break the code down further.

alexanderbez

Changes mainly LGTM, great work. Left some small remarks 👍

CHANGELOG.md

db/README.md

db/memdb/db.go

alexanderbez · 2021-08-23T19:38:17Z

db/memdb/db.go

+func (db *MemDB) Versions() (dbm.VersionSet, error) {
+	db.mtx.RLock()
+	defer db.mtx.RUnlock()
+	return db.vmgr, nil


do we not want to return a copy of the map?

It seems we already have a Copy method too.

I opted to make this copy-on-write. Versions is called whenever version history is queried, and if the history becomes large, it could be expensive. So a vmgr is just copied over whenever versions are saved/deleted, and the caller can safely keep the old reference.

vmgr is a pointer, so we don't make any object copy here

db/types.go

alexanderbez · 2021-08-23T19:42:07Z

db/version_manager.go

+	return &vmIterator{ch: ch}
+}
+
+// Count implements VersionSet.


If this type implements an interface, we don't need to add godocs for the sake of adding godocs 👍

Fine with me, I was just following the existing practice

alexanderbez · 2021-08-23T19:47:25Z

db/memdb/iterator.go

+
+// Error implements Iterator.
+func (i *memDBIterator) Error() error {
+	return nil // famous last words


lol, any reason not to track an error? Or can one never happen?

Yeah, there's no error case for this iterator - it's only invalidated when it reaches its end.

ACK. Let's remove // famous last words then

haha sure, that was from the tm-db code.

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

codecov · 2021-08-24T06:40:52Z

Codecov Report

Merging #9952 (3494b38) into master (54b0556) will increase coverage by 0.24%.
The diff coverage is n/a.

❗ Current head 3494b38 differs from pull request most recent head b56dd6e. Consider uploading reports for the commit b56dd6e to get more accurate results

@@            Coverage Diff             @@
##           master    #9952      +/-   ##
==========================================
+ Coverage   63.56%   63.80%   +0.24%     
==========================================
  Files         572      566       -6     
  Lines       53584    53363     -221     
==========================================
- Hits        34058    34047      -11     
+ Misses      17583    17397     -186     
+ Partials     1943     1919      -24

Impacted Files	Coverage Δ
server/mock/tx.go	`63.15% <0.00%> (-1.85%)`	⬇️
x/auth/keeper/grpc_query.go	`57.14% <0.00%> (-0.86%)`	⬇️
types/address.go	`68.69% <0.00%> (ø)`
x/auth/vesting/types/msgs.go	`0.00% <0.00%> (ø)`
x/auth/middleware/index_events.go
x/auth/middleware/run_msgs.go
x/auth/middleware/recovery.go
x/auth/middleware/middleware.go
x/auth/middleware/legacy_router.go
x/auth/middleware/gas.go
... and 18 more

tac0turtle

@robert-zaremba want to review this so we can merge?

alexanderbez · 2021-08-25T18:55:58Z

Let's give @robert-zaremba a day to review. Is that okay?

tac0turtle · 2021-08-26T06:54:15Z

Let's give @robert-zaremba a day to review. Is that okay?

haha, more than one day is fine as well. It was mainly a ping, not a "needs to be done now"

tac0turtle · 2021-08-31T07:41:57Z

@roysc once the PR is updated to master the bot will merge it

orijbot · 2021-08-31T08:09:48Z

Visit https://dashboard.github.orijtech.com?back=0&pr=9952&remote=false&repo=cosmos%2Fcosmos-sdk to see benchmark details.

odeke-em · 2021-10-07T03:32:07Z

Surprisingly the code from this change is failing tests locally

$ go test
# github.com/cosmos/cosmos-sdk/db/memdb [github.com/cosmos/cosmos-sdk/db/memdb.test]
./db.go:44:23: incompatible type: cannot use (*MemDB)(nil) (value of type *MemDB) as db.DBConnection value
./db_test.go:32:9: incompatible type: cannot use NewDB() (value of type *MemDB) as db.DBConnection value
FAIL	github.com/cosmos/cosmos-sdk/db/memdb [build failed]

roysc force-pushed the roysc/adr-040-db-memdb branch 3 times, most recently from a80c49d to f189a95 Compare August 18, 2021 11:36

roysc changed the title ~~feat: ADR 040: Implement in-memory DB~~ feat: ADR 040: Implement in-memory DB backend Aug 18, 2021

roysc force-pushed the roysc/adr-040-db-memdb branch 2 times, most recently from 9bdcfa2 to 15501e8 Compare August 22, 2021 01:02

roysc marked this pull request as ready for review August 22, 2021 01:03

roysc requested review from aaronc and alexanderbez as code owners August 22, 2021 01:03

changelog entry

9d58b92

roysc marked this pull request as draft August 22, 2021 06:47

roysc added 3 commits August 23, 2021 01:41

KVDB readme

9086cc1

Add dbtest package, test cases

caea36b

Add VersionManager

42306a3

roysc force-pushed the roysc/adr-040-db-memdb branch from 3867290 to 1358c4f Compare August 23, 2021 03:09

Add memdb package

c8cd01d

roysc force-pushed the roysc/adr-040-db-memdb branch from 1358c4f to c8cd01d Compare August 23, 2021 03:50

roysc added 2 commits August 23, 2021 11:56

readme nit

3665e04

fix testcase

e193214

roysc marked this pull request as ready for review August 23, 2021 05:05

clone is not needed here

e17792b

redundant since dbWriter uses a clone

roysc force-pushed the roysc/adr-040-db-memdb branch from 3638b40 to e17792b Compare August 23, 2021 07:21

alexanderbez added the C:Store label Aug 23, 2021

alexanderbez reviewed Aug 23, 2021

View reviewed changes

alexanderbez requested a review from robert-zaremba August 23, 2021 19:48

roysc and others added 2 commits August 24, 2021 13:53

Update CHANGELOG.md

0626049

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

Revise PR

0e04e23

alexanderbez approved these changes Aug 24, 2021

View reviewed changes

robert-zaremba mentioned this pull request Aug 24, 2021

Incorporate tm-db #9092

Closed

4 tasks

clean up

3494b38

tac0turtle approved these changes Aug 25, 2021

View reviewed changes

tac0turtle added the A:automerge Automatically merge PR once all prerequisites pass. label Aug 31, 2021

Merge branch 'master' into roysc/adr-040-db-memdb

b56dd6e

mergify bot merged commit 2c31451 into cosmos:master Aug 31, 2021

roysc deleted the roysc/adr-040-db-memdb branch August 31, 2021 10:33

i-norden mentioned this pull request Sep 7, 2021

ADR-040 Implementation: New DB interface #10094

Closed

8 tasks

odeke-em mentioned this pull request Oct 7, 2021

db/memdb: failing tests and non-compiling code from "cannot use (*MemDB)(nil) (value of type *MemDB) as db.DBConnection value" #10319

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: ADR 040: Implement in-memory DB backend #9952

feat: ADR 040: Implement in-memory DB backend #9952

roysc commented Aug 17, 2021 •

edited

Loading

roysc commented Aug 23, 2021

alexanderbez left a comment

alexanderbez Aug 23, 2021

alexanderbez Aug 23, 2021

roysc Aug 24, 2021

robert-zaremba Sep 3, 2021

alexanderbez Aug 23, 2021

roysc Aug 24, 2021

alexanderbez Aug 23, 2021

roysc Aug 24, 2021

alexanderbez Aug 24, 2021

roysc Aug 24, 2021

codecov bot commented Aug 24, 2021 •

edited

Loading

tac0turtle left a comment

alexanderbez commented Aug 25, 2021

tac0turtle commented Aug 26, 2021

tac0turtle commented Aug 31, 2021

orijbot commented Aug 31, 2021

odeke-em commented Oct 7, 2021

feat: ADR 040: Implement in-memory DB backend #9952

feat: ADR 040: Implement in-memory DB backend #9952

Conversation

roysc commented Aug 17, 2021 • edited Loading

Description

Author Checklist

Reviewers Checklist

roysc commented Aug 23, 2021

alexanderbez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 24, 2021 • edited Loading

Codecov Report

tac0turtle left a comment

Choose a reason for hiding this comment

alexanderbez commented Aug 25, 2021

tac0turtle commented Aug 26, 2021

tac0turtle commented Aug 31, 2021

orijbot commented Aug 31, 2021

odeke-em commented Oct 7, 2021

roysc commented Aug 17, 2021 •

edited

Loading

codecov bot commented Aug 24, 2021 •

edited

Loading