-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HOLD [$250] Account not showing recent expenses/scans in my workspace chat #45546
Comments
Triggered auto assignment to @slafortune ( |
Triggered auto assignment to @techievivek ( |
Job added to Upwork: https://www.upwork.com/jobs/~0184e98d80e32fd2e3 |
Triggered auto assignment to Contributor-plus team member for initial proposal review - @ishpaul777 ( |
We are looking for proposals |
Does this only happen in HybridApp, or standalone NewDot? |
cc @staszekscp @mateuuszzzzz - Can one of you check this? |
I actually had this happen to me on Web, so I don't think this is limited to HybridApp and I think we should be looking at this in a larger context. e.g. It says receipt scanning in LHN and when I click onto the report. Only once I click into the report, does it "refresh" and show that the SmartScan has finished. |
I am able to reproduce as well on web Screen.Recording.2024-07-26.at.11.28.34.PM.mov |
In that case I think that HybridApp can be removed from the issue title to avoid confusion 😄 |
I have removed Hybrid web from the title. |
Closing this in favor of these two separate issues created after much discussion here - https://expensify.slack.com/archives/C05LX9D6E07/p1721152209600379 - |
Let's reopen this |
@rinej can you comment on this so I can assign you? |
📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸 |
Hello @muttmuure, sure please assign me. I've worked on tha logjam issue, so it might be related. |
Not overdue, @rinej will start to work on it this week. |
Hello! Here is my research and proposal: ProblemWe have encountered several instances where the Upon investigation, I discovered that the issue is related to multiple network retries. When there are too many retry requests, the If a request keeps failing and the retry mechanism increases the wait time exponentially, it can lead to significant delays. This can block the queue, causing other requests to be delayed or not processed at all. In our codebase, when a request fails, we retry it up to 10 times, with an increasing wait time for each retry. After 10 retries, the request is rejected. However, during this waiting period, the AnalysisHere are some measurements I took:
SolutionTo address this issue, in addition to limiting the maximum retry count, we can introduce a maximum cumulative wait time ( This constant sets a limit on the total time that can be spent waiting for retries across all requests. If the cumulative wait time exceeds this limit, the retry mechanism will stop, and all failing requests will be rejected. This ensures that the queue is not indefinitely blocked by failing requests, allowing other requests to be processed. Additionally, we’ve introduced a cooldown period ( BenchmarkHere are the results from testing 10 failing requests:
ConclusionWhile this is one approach to solve the problem, I'm open to further thoughts and suggestions! We can of course modify the MAX_CUMULATIVE_WAIT_TIME_MS to make the best balance between retrying period and unblocking the queue. |
Based on the slack discussion, we added more logs in the retry logic to identify the root cause -> #48983 |
@rinej The PR is deployed to PROD. Did you get a chance to dig into this further, given that we now have better logs for it? |
Right now we have to wait for the issue to happen again so we can dig into logs to confirm our investigation. Also there is outgoing PR, which might also fix the issue -> #47913 |
This is the discussion thread for this issue. The queue is not getting "stuck", the requests are getting stuck behind Putting this issue on HOLD |
This should be resolved as the reconnectApp calls been deduped, I will close this issue and feel free to reopen if this is still happening |
If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!
Version Number:
Reproducible in staging?: n/a
Reproducible in production?: n/a
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @danielrvidal
Slack conversation: https://expensify.slack.com/archives/C05LX9D6E07/p1721152209600379
Action Performed:
Expected Result:
All the past expense should be available
Actual Result:
Workaround:
unknown
Platforms:
Which of our officially supported platforms is this issue occurring on?
Screenshots/Videos
onyx-state.txt
logs-2024-07-16 17_45_38.974.txt
RPReplay_Final1721151916.mov
View all open jobs on GitHub
Upwork Automation - Do Not Edit
The text was updated successfully, but these errors were encountered: