HOLD [$250] Account not showing recent expenses/scans in my workspace chat

[m-natarajan]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Version Number:
Reproducible in staging?: n/a
Reproducible in production?: n/a
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @danielrvidal
Slack conversation: https://expensify.slack.com/archives/C05LX9D6E07/p1721152209600379

Action Performed:

1. Scan a receipt and completed successfully
2. Expense from yesterday is missing
3. Observe all the receipts scanned previously

Expected Result:

All the past expense should be available

Actual Result:

• All expenses just kept saying “Receipt scanning…” even though it was showing expenses from a month ago.
• In the video, you can see me switch to New Expensify the app rather than hybrid app and it shows my expenses.
• Seems some expenses stuck in a scanning state. After a while of having the app open, expense from yesterday came over and the one from today scanned successfully.
• But the two from June 17th remain in a “Receipt Scanning…” state.

Workaround:

unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android: Native
• Android: mWeb Chrome
• iOS: Native
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Screenshots/Videos

onyx-state.txt

logs-2024-07-16 17_45_38.974.txt

RPReplay_Final1721151916.mov

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~0184e98d80e32fd2e3
• Upwork Job ID: 1815375308907786402
• Last Price Increase: 2024-09-03

[melvin-bot]

Triggered auto assignment to @slafortune (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[melvin-bot]

Triggered auto assignment to @techievivek (AutoAssignerNewDotQuality)

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~0184e98d80e32fd2e3

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @ishpaul777 (External)

[ishpaul777]

We are looking for proposals

[AndrewGable]

Does this only happen in HybridApp, or standalone NewDot?

[danielrvidal]

It's only happening in Hybrid app. I just opened the app and it's still happening to me so let me know if I can do anything to help diagnose.

[AndrewGable]

cc @staszekscp @mateuuszzzzz - Can one of you check this?

[AndrewGable]

I actually had this happen to me on Web, so I don't think this is limited to HybridApp and I think we should be looking at this in a larger context.

e.g. It says receipt scanning in LHN and when I click onto the report.

Only once I click into the report, does it "refresh" and show that the SmartScan has finished.

[ishpaul777]

I am able to reproduce as well on web

Screen.Recording.2024-07-26.at.11.28.34.PM.mov

[staszekscp]

In that case I think that HybridApp can be removed from the issue title to avoid confusion 😄

[techievivek]

I have removed Hybrid web from the title.

[slafortune]

Closing this in favor of these two separate issues created after much discussion here - https://expensify.slack.com/archives/C05LX9D6E07/p1721152209600379 -
#47740
#47742

[muttmuure]

Let's reopen this

[muttmuure]

@rinej can you comment on this so I can assign you?

[melvin-bot]

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

[rinej]

Hello @muttmuure, sure please assign me. I've worked on tha logjam issue, so it might be related.
Also please remember I will be OOO from tomorrow till the end of the week

[techievivek]

Not overdue, @rinej will start to work on it this week.

[rinej]

Hello! Here is my research and proposal:

Problem

We have encountered several instances where the SequentialQueue stops executing without providing any feedback. For example, this issue was observed in the following tickets: #45546, #47740.

Upon investigation, I discovered that the issue is related to multiple network retries. When there are too many retry requests, the SequentialQueue can become blocked.

If a request keeps failing and the retry mechanism increases the wait time exponentially, it can lead to significant delays. This can block the queue, causing other requests to be delayed or not processed at all.

In our codebase, when a request fails, we retry it up to 10 times, with an increasing wait time for each retry. After 10 retries, the request is rejected. However, during this waiting period, the SequentialQueue is blocked, meaning that all new actions are halted. This is why users may experience issues like reports loading indefinitely or expenses not being loaded.

Analysis

Here are some measurements I took:

• For 1 failing API call (which is retried 10 times), the user has to wait approximately 20 seconds before the request fails and all other actions are blocked during this time.
• For 10 failing API calls, the wait time increases significantly to 6 minutes and 40 seconds. This is an excessively long time, during which the user perceives the app as unresponsive.

Solution

To address this issue, in addition to limiting the maximum retry count, we can introduce a maximum cumulative wait time (MAX_CUMULATIVE_WAIT_TIME_MS).

This constant sets a limit on the total time that can be spent waiting for retries across all requests. If the cumulative wait time exceeds this limit, the retry mechanism will stop, and all failing requests will be rejected. This ensures that the queue is not indefinitely blocked by failing requests, allowing other requests to be processed.

Additionally, we’ve introduced a cooldown period (CLEAR_RETRY_TOTAL_TIME_COOLDOWN) to reset the cumulative wait time after it has been exceeded. This allows the system to recover and handle new requests without being negatively impacted by previous failures.

Benchmark

Here are the results from testing 10 failing requests:

• Before the change: The app is blocked for ~6 minutes and 40 seconds while retries are processed.
• After the change (with the maximum cumulative time set to 10 seconds): The app is blocked for only ~15 seconds.

Conclusion

While this is one approach to solve the problem, I'm open to further thoughts and suggestions!

We can of course modify the MAX_CUMULATIVE_WAIT_TIME_MS to make the best balance between retrying period and unblocking the queue.

[rinej]

Based on the slack discussion, we added more logs in the retry logic to identify the root cause -> #48983

[techievivek]

@rinej The PR is deployed to PROD. Did you get a chance to dig into this further, given that we now have better logs for it?

[rinej]

Right now we have to wait for the issue to happen again so we can dig into logs to confirm our investigation.
Here is the slack message created by Matt -> https://callstack-hq.slack.com/archives/C05LX9D6E07/p1726224795425099

Also there is outgoing PR, which might also fix the issue -> #47913

[muttmuure]

This is the discussion thread for this issue. The queue is not getting "stuck", the requests are getting stuck behind ReconnectApp calls.

Putting this issue on HOLD

[mountiny]

This should be resolved as the reconnectApp calls been deduped, I will close this issue and feel free to reopen if this is still happening