Add getEvents backed by DB #215

psheth9 · 2024-06-13T15:23:06Z

Overall design:

Ingest events into DB: store entire eventXDR along with metadata like contract_ids, topics etc..
Combine contract_ids and topics from all the filters for a given request and pass it along to DB (to save multiple DB calls)
Fetch events and apply filter again based on the request (on less data comparatively)

Other Refactor:

Remove in-memory events store
Introduce endLedgerSeq
Add migration related logic for events

Other related PR:

Migration failures and refactor daemon.go PR

Data Flow:

Design Docs:

Eng doc
Benchmarking doc here

Parent Epic #115
Fixes: #190

psheth9 · 2024-06-18T22:07:28Z

ledgerCloseMetaWithEvents in get_events_test does not seem to work during DB ingestion.

Error logs:
unknown tx hash in LedgerCloseMeta: 4705598f90aa1c2affc48ffb10f42ab1a4d9545fcb8ed0adfd6b2ae141aa1c21

But also at the same time txMetaWithEvents in transaction_test works fine for event ingestion so it has to do something with the mocked LCM object itself.

Shaptic · 2024-06-18T22:43:51Z

@psheth9 you're changing the hash of the transaction when you inject events into it after creation which makes it invalid

…re not indexed

psheth9 · 2024-06-20T15:16:59Z

you're changing the hash of the transaction when you inject events into it after creation which makes it invalid

This is not the case as I am not touching any fields related to hash. Also ledgerCloseMetaWithEvents is being used everywhere (in old code) to validate get_events RPC so this should work automatically with new code. Unfortunately it does not :(

cmd/soroban-rpc/internal/db/db.go

cmd/soroban-rpc/internal/db/event.go

Shaptic

Left a buncha comments, mostly small stuff and a few questions

cmd/soroban-rpc/internal/daemon/daemon.go

cmd/soroban-rpc/internal/db/cursor.go

cmd/soroban-rpc/internal/methods/get_events.go

* Fix migrations - 1 * Make migrations sequential - 1 * Make migrations sequential - 2 * Fix failing unittest - 1 * Fix linting errors - 1 * Fix failing integration test - 1 * Remove %w from Fatal strings * refactor migrationApplierFactoryF * Add ledger seq to fatal error string * Add comments - 1 * Fix - 1 * Optimise migrations - 1 * Optimise migrations - 2 * Optimise migrations - 3 * Fix linting - 1 * Fix linting - 2 * Remove dupicate latest ledger fetch code * Rollback db in daemon instead of migration * Remove unused constant * Remove unused constant - 2 * Add rollback() statement * Small change * Abstract transaction and rollback management inside migration code * Fix failing unittest

cmd/soroban-rpc/internal/db/event.go

cmd/soroban-rpc/internal/methods/get_events.go

cmd/soroban-rpc/internal/db/event.go

Shaptic

LGTM! I'd like @2opremio to have a last look, though, since he was driving most of the reviews. We should definitely add Prometheus metrics to these in a separate PR.

Shaptic · 2024-08-20T19:37:24Z

cmd/soroban-rpc/internal/methods/get_events.go

+func getEventTypeXDRFromEventType() map[string]xdr.ContractEventType {
+	return map[string]xdr.ContractEventType{
+		EventTypeSystem:     xdr.ContractEventTypeSystem,
+		EventTypeContract:   xdr.ContractEventTypeContract,
+		EventTypeDiagnostic: xdr.ContractEventTypeDiagnostic,
+	}
+}
+


Can/Should this just be a var at the top of the file? You could then use len(EventTypeMap) instead of a hardcoded 3 constant elsewhere.

well initially it was var same as eventTypeFromXDR but linter does not like global var :( so it was more of temp fix. it will all be gone when we target all of this lint error together to have consistency in code.

psheth9 · 2024-08-20T19:48:38Z

@Shaptic Thanks for review comments and stamp !!

Yes anyway I have to create final PR for events-db-backend to main. @2opremio you can take a final look there. Thank you :)

* Ingest events into DB (#192) * Ingest events into DB * Update tests * Update tests * Fix tests * Ignore ingestion when events are empty * Add getEvents backed by DB (#215) * Ingest events into DB * Update tests * Update tests * Fix tests * Ignore ingestion when events are empty * Add getEvents backed by DB * Refactor getEvents to call fetch events from db * Remove in-memory events store * Make use of cursor in pagination and combine filters * Update scan function logic in order to test * Move NewTestDb util * Trim events db and remove eventTypes in SELECT query as event types are not indexed * Introduce cursor range and add logs for latency * remove event memory store * remove event memory store from latest merge * Fix lint issues part 1 * Fix lint issues part 2 * Fix more lint and add ledger range code for events * Update eventHandler mock * Fix 2 major errors in tests: nil pointer reference and unknown hash * Fix contract Id filter logic and add cursor set to avoid duplicates * Validate requested start ledger with stored ledger range * Fix lint error part 4 * Add migration for events table * Fix lint error part 5 * Fix lint error part 6 * Address review comments pt1 * Make contract id a blob type * Remove events package and move cursor.go to db package * Fix lint error part 6 * Fix lint error part 7 * Optimize migration code * Introduce endLedger and remove Cursor id from schema * Use LedgerReader to get Ledger Range in events * Fix lint errors * Add benchmark for testing various load parameter * update benchmark * Comment benchmark events * Reduce allocs pt1 * Benchmark with 30 million events * Refactor getEvents to backed only by Events table * change test db path * Use Binary encoding for saving events into DB * Correct number of topics * reduce events in benchmark so that tests run * update events schema * Fix topic count * Fix fetch query to not stop if null topic * Fix lint pt1 * Fix trimEvents and lint errors * update log info * Fix more lint errors * Fix more lint errors pt 11 * Fix format in error * Fix linter error pt 12 * Add nolint for GetEvents as a temp fix. * Add events table migration (#262) * Fix migrations - 1 * Make migrations sequential - 1 * Make migrations sequential - 2 * Fix failing unittest - 1 * Fix linting errors - 1 * Fix failing integration test - 1 * Remove %w from Fatal strings * refactor migrationApplierFactoryF * Add ledger seq to fatal error string * Add comments - 1 * Fix - 1 * Optimise migrations - 1 * Optimise migrations - 2 * Optimise migrations - 3 * Fix linting - 1 * Fix linting - 2 * Remove dupicate latest ledger fetch code * Rollback db in daemon instead of migration * Remove unused constant * Remove unused constant - 2 * Add rollback() statement * Small change * Abstract transaction and rollback management inside migration code * Fix failing unittest * Address review comments * Store binary of topics instead of string * Unify min/max topic count in event.go * Address review comments for event Types and fix unit tests * cleanup * Fix linter errors for one last time --------- Co-authored-by: Aditya Vyas <aditya.vyas@stellar.org> * Fix linter errors pt 1 * Add nolint for InsertEvents * Add nolint for InsertEvents again * Address review comments * Fix lint * Fix lint pt1 * Fix lint pt2 * Fix index out of bound error and fix logging * Add changelog entry --------- Co-authored-by: Aditya Vyas <aditya.vyas@stellar.org>

psheth9 added 6 commits May 31, 2024 14:31

Ingest events into DB

0a7e05a

Update tests

9fa8c2a

Update tests

4cc58b9

Fix tests

3940d27

Ignore ingestion when events are empty

4d0c280

Add getEvents backed by DB

4f68a35

psheth9 mentioned this pull request Jun 13, 2024

Ingest events into DB #192

Merged

psheth9 added 2 commits June 17, 2024 12:42

Refactor getEvents to call fetch events from db

9519aa2

Remove in-memory events store

133ef97

Base automatically changed from ingest-events-in-db to events-db-backend June 18, 2024 04:42

psheth9 added 3 commits June 17, 2024 22:12

Make use of cursor in pagination and combine filters

c52f618

Update scan function logic in order to test

2a6435d

Move NewTestDb util

5c5b407

psheth9 added 3 commits June 18, 2024 15:51

Trim events db and remove eventTypes in SELECT query as event types a…

a566923

…re not indexed

Introduce cursor range and add logs for latency

c70c9e4

remove event memory store

e86cebc

psheth9 requested review from 2opremio and aditya1702 June 20, 2024 15:11

psheth9 marked this pull request as ready for review June 20, 2024 15:24

psheth9 changed the title ~~[DRAFT] Add getEvents backed by DB~~ [WIP] Add getEvents backed by DB Jun 20, 2024

psheth9 requested a review from Shaptic June 20, 2024 15:37

psheth9 added 4 commits June 20, 2024 12:21

Merge branch 'events-db-backend' into refactor-get-events

4d809d2

remove event memory store from latest merge

a3f8987

Fix lint issues part 1

1a03730

Fix lint issues part 2

6551dc1

psheth9 self-assigned this Jun 21, 2024

2opremio reviewed Jun 21, 2024

View reviewed changes

cmd/soroban-rpc/internal/db/db.go Outdated Show resolved Hide resolved

Fix lint pt1

905eaf0

tamirms reviewed Aug 9, 2024

View reviewed changes

cmd/soroban-rpc/internal/db/event.go Show resolved Hide resolved

tamirms reviewed Aug 9, 2024

View reviewed changes

cmd/soroban-rpc/internal/db/event.go Outdated Show resolved Hide resolved

psheth9 added 5 commits August 9, 2024 12:34

Fix trimEvents and lint errors

4b8726b

update log info

8b28d65

Fix more lint errors

051ab4f

Fix more lint errors pt 11

f284c9e

Fix format in error

9c7e870

psheth9 mentioned this pull request Aug 13, 2024

Add events table migration #262

Merged

Fix linter error pt 12

52bdf24

psheth9 requested a review from tamirms August 13, 2024 18:58

Add nolint for GetEvents as a temp fix.

99d5411

Shaptic reviewed Aug 15, 2024

View reviewed changes

aditya1702 and others added 4 commits August 16, 2024 13:21

Address review comments

8e33b1e

Store binary of topics instead of string

7d47427

Unify min/max topic count in event.go

435b7e0

Shaptic reviewed Aug 20, 2024

View reviewed changes

cmd/soroban-rpc/internal/db/event.go Show resolved Hide resolved

cmd/soroban-rpc/internal/methods/get_events.go Show resolved Hide resolved

Shaptic reviewed Aug 20, 2024

View reviewed changes

cmd/soroban-rpc/internal/methods/get_events.go Show resolved Hide resolved

Shaptic reviewed Aug 20, 2024

View reviewed changes

cmd/soroban-rpc/internal/db/event.go Show resolved Hide resolved

psheth9 added 2 commits August 19, 2024 20:29

Address review comments for event Types and fix unit tests

4f7b675

cleanup

1fbd381

psheth9 requested a review from Shaptic August 20, 2024 19:26

Shaptic approved these changes Aug 20, 2024

View reviewed changes

Fix linter errors for one last time

187e9d6

psheth9 merged commit a7e6459 into events-db-backend Aug 20, 2024
17 checks passed

psheth9 deleted the refactor-get-events branch August 20, 2024 20:19

psheth9 mentioned this pull request Aug 20, 2024

Events db backend #230

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add getEvents backed by DB #215

Add getEvents backed by DB #215

psheth9 commented Jun 13, 2024 •

edited

Loading

psheth9 commented Jun 18, 2024 •

edited

Loading

Shaptic commented Jun 18, 2024

psheth9 commented Jun 20, 2024

Shaptic left a comment

Shaptic left a comment

Shaptic Aug 20, 2024

psheth9 Aug 20, 2024

psheth9 commented Aug 20, 2024

Add getEvents backed by DB #215

Add getEvents backed by DB #215

Conversation

psheth9 commented Jun 13, 2024 • edited Loading

psheth9 commented Jun 18, 2024 • edited Loading

Shaptic commented Jun 18, 2024

psheth9 commented Jun 20, 2024

Shaptic left a comment

Choose a reason for hiding this comment

Shaptic left a comment

Choose a reason for hiding this comment

Shaptic Aug 20, 2024

Choose a reason for hiding this comment

psheth9 Aug 20, 2024

Choose a reason for hiding this comment

psheth9 commented Aug 20, 2024

psheth9 commented Jun 13, 2024 •

edited

Loading

psheth9 commented Jun 18, 2024 •

edited

Loading