fix_: ensure individual store queries do not exceed 24h while still allowing to retrieve more than one day of messages #6115
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
richard-ramos
requested review from
plopezlpz,
kaichaosun,
chaitanyaprem,
igor-sirotin and
osmaczko
Jenkins BuildsClick to see older builds (32)
|
kaichaosun
reviewed
Nov 21, 2024
protocol/messenger_mailserver.go
Outdated
| @@ -880,6 +855,8 @@ loop: | |||
| contentTopics: w.contentTopics, | |||
| cursor: cursor, | |||
| limit: nextPageLimit, | |||
| from: w.from, | |||
| to: nextWorkTo, | |||
There was a problem hiding this comment.
What happens if above line cursor == nil?
It seems we need to first processNextPage with the cursor then process the nextWorkTo? Now it combines the cursor (processNextPage) and nextWorkTo.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #6115 +/- ##
===========================================
+ Coverage 60.95% 60.97% +0.01%
===========================================
Files 827 827
Lines 109734 109708 -26
===========================================
+ Hits 66890 66895 +5
+ Misses 34989 34964 -25
+ Partials 7855 7849 -6
Flags with carried forward coverage won't be shown. Click here to find out more.
|
jrainville
approved these changes
Nov 21, 2024
There was a problem hiding this comment.
LGTM.
BTW, we found today with @Ivansete-status that some of the store requests fetched a whole month. I assume it was an account recovery, but not sure.
So I guess this PR would fix it.
One additional improvement in that case would be to make the account recovery smarter. If you fetched all the needed info in the last 24 hours, you don't need the info from a week past, since you only need the most recent.
Maybe something to fix in another issue. Though I'm not sure if it should be fixed by the Status or Waku team 🤔
To fix that, we can do something similar to what @igor-sirotin implemented here: In which a function is passed toprocessMailserverBatchWithOptions to indicate whether to continue fetching messages from storenode or not.
|
kaichaosun
approved these changes
Nov 23, 2024
plopezlpz
approved these changes
Nov 25, 2024
|
This PR also fix #6046 |
richard-ramos
force-pushed
the
24h-batches
branch
from
38dce59 to
2244bfc
Compare
This was referenced Nov 25, 2024
richard-ramos
force-pushed
the
24h-batches
branch
2 times, most recently
from
757c9bb to
50f2c74
Compare
richard-ramos
force-pushed
the
24h-batches
branch
from
50f2c74 to
6d2a89a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #6057
Also, due to how we use seconds in some parts of the code, there's a loss of precision in the end date of the store queries which translates in potentially missed messages. To quickly fix this I just add a second and subtract a nanosecond to ensure that the end date will include the whole second.
This however should be properly fixed by using time.Time across all codebase instead as mentioned in #5857 (comment)
cc: @fryorcraken