Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

eth/filters: retrieve logs in async #27135

Merged
merged 26 commits into from
May 25, 2023
Merged

eth/filters: retrieve logs in async #27135

merged 26 commits into from
May 25, 2023

Conversation

jsvisa
Copy link
Contributor

@jsvisa jsvisa commented Apr 21, 2023 •
edited
Loading

Try to implement #15063 and discussed with @s1na,
In the first phase, we want to make the filter.Logs() function returns a channel instead of a large array

s1na
s1na requested changes Apr 21, 2023
View reviewed changes
Copy link
Contributor

@s1na s1na left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some early comments

eth/filters/filter.go Outdated Show resolved Hide resolved
eth/filters/filter.go Outdated Show resolved Hide resolved
eth/filters/filter.go Outdated Show resolved Hide resolved
@fjl fjl changed the title WIP: eth/filters: retrive logs in async WIP: eth/filters: retrieve logs in async Apr 24, 2023
fjl
fjl reviewed Apr 24, 2023
View reviewed changes
eth/filters/filter.go Outdated Show resolved Hide resolved
@jsvisa jsvisa changed the title WIP: eth/filters: retrieve logs in async eth/filters: retrieve logs in async Apr 24, 2023
@jsvisa
Copy link
Contributor Author

jsvisa commented Apr 24, 2023

@s1na please take another look, thanks

s1na
s1na reviewed May 4, 2023
View reviewed changes
eth/filters/filter.go Outdated Show resolved Hide resolved
holiman
holiman reviewed May 5, 2023
View reviewed changes
eth/filters/filter.go
} else {
logs, err = f.indexedLogs(ctx, indexed-1)

go func() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't reviewed this in any depth, so maybe I'm missing something... but it looks to me like this spins up two routines that may be stuck forever trying to deliver, in case the other side is no longer reading, for whatever reason.
Wouldn't it make sense it use the ctx to check if it's time to abort these routines?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. But seems we have handled this issue inside the indexedLogs https://github.com/ethereum/go-ethereum/blob/101ac683e447a2c23fc0141d33bbb3f2e0e32607/eth/filters/filter.go#L266

I wonder if I've lost something

Copy link
Contributor

@s1na s1na May 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to protect against potential receiver failure, we'd have to wrap every channel send with a select, as in:

select {
  case logchan <- log:
  case ctx.Done():
    return ctx.Error()
}

I guess I'm just wondering if this has an impact on performance or that should be negligible

@s1na
Copy link
Contributor

s1na commented May 8, 2023

This PR is looking very nice overall! I'm gonna do some testing.

s1na
s1na approved these changes May 12, 2023
View reviewed changes
Copy link
Contributor

@s1na s1na left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

holiman
holiman reviewed May 17, 2023
View reviewed changes
eth/filters/filter.go Outdated Show resolved Hide resolved
eth/filters/filter.go Outdated Show resolved Hide resolved
eth/filters/filter.go Outdated Show resolved Hide resolved
holiman
holiman reviewed May 17, 2023
View reviewed changes
eth/filters/filter.go Outdated Show resolved Hide resolved
@jsvisa
Copy link
Contributor Author

jsvisa commented May 18, 2023

@s1na @holiman please take another look, thanks

holiman
holiman reviewed May 19, 2023
View reviewed changes
eth/filters/filter.go Outdated Show resolved Hide resolved
holiman
holiman reviewed May 19, 2023
View reviewed changes
Copy link
Contributor

@holiman holiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining nit, otherwise lgtm

@s1na
Copy link
Contributor

s1na commented May 22, 2023

I saw the timeout testcase I added fail once so I re-adjusted the timeout. It might turn out to be a flaky one. In that case I'll remove it in a future PR.

holiman
holiman reviewed May 22, 2023
View reviewed changes
eth/filters/filter_test.go Outdated Show resolved Hide resolved
@s1na
Copy link
Contributor

s1na commented May 23, 2023

Needs rebase

@jsvisa
Copy link
Contributor Author

jsvisa commented May 23, 2023

I'll rebase it right now

@jsvisa
eth/filters: pendingLogs will not fail
78b2136 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: pass a channel to recv logs
58b35f7 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: retrive logs in async mode
4cdd665 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: backfilling it
4d33109 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: make LogsAsync private
ece83c7 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
Revert "eth/filters: backfilling it"
fc883c2 
This reverts commit fadf142d0a98d60acb0b4732e95e6bc77a5cbe8a.

Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: close logChan and errChan inside producer's side
9654751 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: let's resolve special in the begining of the goroutine
be1f183 
Signed-off-by: jsvisa <delweng@gmail.com>
jsvisa and others added 17 commits May 23, 2023 21:23
@jsvisa
eth/filters: resolve block number in Logs
9093799 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: handle pending logs outside async
368d09c 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: handle error
a2fcf1e 
Signed-off-by: jsvisa <delweng@gmail.com>
@s1na @jsvisa
add tests for single block and pending
4ebecb2 
Signed-off-by: jsvisa <delweng@gmail.com>
@s1na @jsvisa
use logger contract in tests
23e1876 
Signed-off-by: jsvisa <delweng@gmail.com>
@s1na @jsvisa
test full logs output
5625439 
Signed-off-by: jsvisa <delweng@gmail.com>
@s1na @jsvisa
expand tests
9cf858e 
Signed-off-by: jsvisa <delweng@gmail.com>
@s1na @jsvisa
add timeout case
4353b09 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @holiman
Update eth/filters/filter.go
aa2cf79 
Co-authored-by: Martin Holst Swende <martin@swende.se>
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @holiman
Update eth/filters/filter.go
a014cd3 
Co-authored-by: Martin Holst Swende <martin@swende.se>
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @holiman
Update eth/filters/filter.go
eda6214 
Co-authored-by: Martin Holst Swende <martin@swende.se>
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: fix typo
da62eac 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: no for loop
0ba4db0 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: no need to check 10th context error
2df3c84 
Signed-off-by: jsvisa <delweng@gmail.com>
@s1na @jsvisa
minor
5000eb7 
Signed-off-by: jsvisa <delweng@gmail.com>
@s1na @holiman @jsvisa
Update eth/filters/filter_test.go
5167b95 
Co-authored-by: Martin Holst Swende <martin@swende.se>
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
eth/filters: pending logs maybe null
f54f906 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa
Copy link
Contributor Author

jsvisa commented May 23, 2023

Rebased, and please take another look @s1na @holiman :)

holiman
holiman approved these changes May 23, 2023
View reviewed changes
Copy link
Contributor

@holiman holiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. We might be putting out a release pretty shortly, so I'd prefer to wait with merging until afterwards

@s1na s1na added this to the 1.12.1 milestone May 24, 2023
@holiman holiman merged commit db9a178 into ethereum:master May 25, 2023
@jsvisa jsvisa deleted the log-subscribe branch June 3, 2023 12:01
@0xcb9ff9 0xcb9ff9 mentioned this pull request Oct 30, 2023
Open
3 tasks
devopsbo3 pushed a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
@jsvisa @s1na
@holiman @devopsbo3
eth/filters: retrieve logs in async (ethereum#27135)
f875ab9 
This change implements async log retrievals via feeding logs in channels, instead of returning slices. This is a first step to implement ethereum#15063.  

---------

Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
devopsbo3 added a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
@devopsbo3
Revert "eth/filters: retrieve logs in async (ethereum#27135)"
d6f0b01 
This reverts commit f875ab9.
devopsbo3 added a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
@devopsbo3
Revert "eth/filters: retrieve logs in async (ethereum#27135)"
91ac8fd 
This reverts commit f875ab9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Macias131987 Macias131987 Macias131987 left review comments

@fjl fjl fjl left review comments

@s1na s1na s1na approved these changes

@holiman holiman holiman approved these changes

@karalabe karalabe Awaiting requested review from karalabe karalabe is a code owner

@rjl493456442 rjl493456442 Awaiting requested review from rjl493456442 rjl493456442 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.12.1
Development

Successfully merging this pull request may close these issues.
5 participants
@jsvisa @s1na @fjl @holiman @Macias131987