[$250] Web - Search - New messages are not updated automatically in chats section

[lanitochka17]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Version Number: 9.0.59-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught on HybridApp, is this reproducible on New Expensify Standalone?: N/A
If this was caught during regression testing, add the test name, ID and link from TestRail: https://expensify.testrail.io/index.php?/tests/view/5192577&group_by=cases:section_id&group_order=asc&group_id=296775
Issue reported by: Applause - Internal Team

Action Performed:

1. Open the staging.new.expensify.com website
2. Click on "Search" on the bottom of the screen
3. Click on "Chats"
4. From another browser or device, send a message to the main account
5. Verify the new message is automatically updated on the chat section

Expected Result:

New messages should be updated in real time on chats section on main device, as it happens with new expenses or deleted messages

Actual Result:

New messages on chats section are not displayed until navigating out of the section and returning. Behaviour with deleted messages or new expenses is different, as they are displayed in real time

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android: Standalone
• Android: HybridApp
• Android: mWeb Chrome
• iOS: Standalone
• iOS: HybridApp
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

Bug6657705_1730991181627.Search_Chats.mp4

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~021856702046390987958
• Upwork Job ID: 1856702046390987958
• Last Price Increase: 2024-11-20
• Automatic offers:
• FitseTLT | Contributor | 105128912

Issue Owner

Current Issue Owner: @mananjadhav

[melvin-bot]

Triggered auto assignment to @CortneyOfstad (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[melvin-bot]

@CortneyOfstad Huh... This is 4 days overdue. Who can take care of this?

[CortneyOfstad]

Was able to consistently recreate this between my cortney@realtownhomes.com and cortney@expensifail.com accounts.

The only way I could get the messages to appear was a forced refresh or actively clicking out and back into the chats page.

Going to get eyes on this!

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~021856702046390987958

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @mananjadhav (External)

[FitseTLT]

Edited by proposal-police: This proposal was edited at 2024-11-22 18:50:08 UTC.

Proposal

Please re-state the problem that we are trying to solve in this issue.

Web - Search - New messages are not updated automatically in chats section

What is the root cause of that problem?

New Feature

What changes do you think we should make in order to solve the problem?

We can update useSearchHighlightAndScroll which is handling only the searchHighlight of expenses to start supporting chat too. We will update it to handle report action highlighting for chat case and transaction highlighting for expenses

1. We will listen to reportActions_ key and hold prevReportActions value to compare the new report action added

 const [reportActions] = useOnyx(ONYXKEYS.COLLECTION.REPORT_ACTIONS);
    const previousReportActions = usePrevious(reportActions);

and pass it to the hook here

App/src/components/Search/index.tsx

Lines 180 to 185 in 4bb19d7

	const {newSearchResultKey, handleSelectionListScroll} = useSearchHighlightAndScroll({
	searchResults,
	transactions,
	previousTransactions,
	queryJSON,
	offset,

        reportActions,
        previousReportActions,

2. inside the hook we will trigger a search action when there is new report action added
   

   App/src/hooks/useSearchHighlightAndScroll.ts

   Lines 42 to 47 in 4bb19d7

   	if (transactionsLength && typeof previousTransactionsLength === 'number' && transactionsLength > previousTransactionsLength) {
   	// Set the flag indicating the search is triggered by the hook
   	triggeredByHookRef.current = true;
   	// Trigger the search
   	SearchActions.search({queryJSON, offset});

        const reportActionsLength = reportActions && Object.values(reportActions).reduce((sum, curr) => sum + Object.keys(curr).length, 0);
        const prevReportActionsLength = previousReportActions && Object.values(previousReportActions).reduce((sum, curr) => sum + Object.keys(curr).length, 0);
        // Return early if search was already triggered or there's no change in transactions length
        if (searchTriggeredRef.current || (isExpense && previousTransactionsLength === transactionsLength) || (isChat && reportActionsLength === prevReportActionsLength)) {
            return;
        }
        const newTransactionAdded = transactionsLength && typeof previousTransactionsLength === 'number' && transactionsLength > previousTransactionsLength;

        const newReportActionAdded = typeof reportActionsLength === 'number' && typeof prevReportActionsLength === 'number' && reportActionsLength > prevReportActionsLength;
        // Check if a new transaction was added
        if ((!isChat && newTransactionAdded) || (isChat && newReportActionAdded)) {
            // Set the flag indicating the search is triggered by the hook
           

3. We will identify the id of the new report action add and set newSearchResultKey
   

   App/src/hooks/useSearchHighlightAndScroll.ts

   Lines 87 to 90 in 4bb19d7

   	const newTransactionKey = `${ONYXKEYS.COLLECTION.TRANSACTION}${newTransactionID}`;
   	setNewSearchResultKey(newTransactionKey);
   	highlightedTransactionIDs.current.add(newTransactionID);

  if (!isChat) {
            const previousTransactionIDs = extractTransactionIDsFromSearchResults(previousSearchResults);
            const currentTransactionIDs = extractTransactionIDsFromSearchResults(searchResults.data);

            // Find new transaction IDs that are not in the previousTransactionIDs and not already highlighted
            const newTransactionIDs = currentTransactionIDs.filter((id) => !previousTransactionIDs.includes(id) && !highlightedIDs.current.has(id));

            if (!triggeredByHookRef.current || newTransactionIDs.length === 0) {
                return;
            }

            const newTransactionID = newTransactionIDs.at(0) ?? '';
            const newTransactionKey = `${ONYXKEYS.COLLECTION.TRANSACTION}${newTransactionID}`;

            setNewSearchResultKey(newTransactionKey);
            highlightedIDs.current.add(newTransactionID);
        }
        if (isChat) {
            const previousReportActionIDs = extractReportActionIDsFromSearchResults(previousSearchResults);
            const currentReportActionIDs = extractReportActionIDsFromSearchResults(searchResults.data);

            // Find new transaction IDs that are not in the previousTransactionIDs and not already highlighted
            const newReportActionIDs = currentReportActionIDs.filter((id) => !previousReportActionIDs.includes(id) && !highlightedIDs.current.has(id));

            if (!triggeredByHookRef.current || newReportActionIDs.length === 0) {
                return;
            }

            const newReportActionID = newReportActionIDs.at(0) ?? '';
            const newReportActionKey = `${ONYXKEYS.COLLECTION.REPORT_ACTIONS}${newReportActionID}`;

            setNewSearchResultKey(newReportActionKey);
            highlightedIDs.current.add(newReportActionID);
        }
    }, [searchResults?.data, previousSearchResults, isExpense, isChat]);

4. Then add the necessary styling in ChatItem .
   And also other minor changes I think a Test branch is better approach here.

POC:

2024-11-16.03-28-00.mp4

What alternative solutions did you explore? (Optional)

[huult]

Proposal

Please re-state the problem that we are trying to solve in this issue.

New messages are not updated automatically in chats section

What is the root cause of that problem?

This is a new feature of react-native-onyx.

When User A sends a message to User B, User B will receive new data through the Pusher event.

App/src/libs/PusherUtils.ts

Line 29 in 4ffbd42

function subscribeToPrivateUserChannelEvent(eventName: string, accountID: string, onEvent: (pushJSON: OnyxUpdatesFromServer) => void) {

And User B listens to this event to receive new data from User A and stores it in Onyx.

• Listen event Pusher
  

  App/src/libs/Navigation/AppNavigator/AuthScreens.tsx

  Line 118 in a4e78aa

  function initializePusher() {

• Handle storing new data in Onyx
  

  App/src/libs/actions/User.ts

  Line 895 in ee38879

  applyOnyxUpdatesReliably(updates);

In this new data included new message from User A sent we have update it to reportAction_ and already called to update to snapshot_ but this value key not exisitng in snapshot_ so we can't update it.

This new data includes a new message sent by User A, which we have updated in reportAction_ and already called to update in snapshot_. However, the value key does not exist in snapshot_, so we can't update it. The reason it doesn't exist is that this is a new message.

https://github.com/Expensify/react-native-onyx/blob/11407f663b605591acfe7e5d5a4265466b7f61ce/lib/Onyx.ts#L576-L579

const newValue = lodashPick(value, Object.keys(snapshotData[key]));

As per this code, we only have a new value when the value key exists, and then we are able to pick it.

To debug this, we can add the log following the path node_modules/react-native-onyx/dist/Onyx.js. and go to updateSnapshots function and add this code under let newValue = (0, pick_1.default)(value, Object.keys(snapshotData[key])); to see what i mentioned

            console.log('****** snapshotData[key] ******', snapshotData[key]);
            console.log('****** value key ******', Object.keys(value)[0]);
            if (snapshotData[key] && Object.keys(snapshotData[key]).length > 0) {
                const firstValueKey = Object.keys(value)[0];
                const snapshotKeys = Object.keys(snapshotData[key]);

                if (!snapshotKeys.includes(firstValueKey)) {
                    console.log(`****** ${firstValueKey} key not exisitng in ${key} ******`, );
                }
            }

So, because we can't add new data to snapshot_, the message will not be displayed when User A sends a new message.

VIDEO: Debug this issue.

What changes do you think we should make in order to solve the problem?

To resolve this issue, we must update react-native-onyx to allow adding new values to snapshot_ if the value key does not exist in snapshot_. The code change will be:

1. Create a new function to add a new value to snapshot_.

// node_modules/react-native-onyx/dist/Onyx.js#L472
function addSnapshots(data) {
    const snapshotCollectionKey = OnyxUtils_1.default.getSnapshotKey();
    if (!snapshotCollectionKey)
        return;
    const promises = [];
    const snapshotCollection = OnyxUtils_1.default.getCachedCollection(snapshotCollectionKey);
    const snapshotCollectionKeyLength = snapshotCollectionKey.length;

    Object.entries(snapshotCollection).forEach(([snapshotKey, snapshotValue]) => {
        // Snapshots may not be present in cache. We don't know how to update them so we skip.
        if (!snapshotValue) {
            return;
        }
        let addedData = {};
        data.forEach(({ key, value }) => {
            // snapshots are normal keys so we want to skip update if they are written to Onyx
            if (OnyxUtils_1.default.isCollectionMemberKey(snapshotCollectionKey, key, snapshotCollectionKeyLength)) {
                return;
            }
            if (typeof snapshotValue !== 'object' || !('data' in snapshotValue)) {
                return;
            }
            const snapshotData = snapshotValue.data;
            if (!snapshotData || !snapshotData[key]) {
                return;
            }

            let newValue = {};

            // Check if snapshotData contains the key and has values
            if (snapshotData[key] && Object.keys(snapshotData[key]).length > 0) {
                const firstValueKey = Object.keys(value)[0];
                const snapshotKeys = Object.keys(snapshotData[key]);

                // Check if the first key in `value` doesn't exist in `snapshotData[key]`
                if (!snapshotKeys.includes(firstValueKey)) {
                    const existingValue = Object.values(snapshotData[key])[0];
                    const newValues = Object.values(value)[0];

                    // Update `existingValue` with matching keys from `newValues`
                    Object.keys(newValues).forEach(subKey => {
                        if (subKey in existingValue) {
                            existingValue[subKey] = newValues[subKey];
                        }
                    });

                    newValue = existingValue;
                }
            }

            addedData = {
                ...addedData,
                [key]: newValue,
            };

        });
        // Skip the update if there's no data to be merged
        if (utils_1.default.isEmptyObject(addedData)) {
            return;
        }
       
        
        promises.push(() => merge(snapshotKey, { data: addedData }));
    });
    return Promise.all(promises.map((p) => p()));
}

2. Call it inside Onyx Update

// node_modules/react-native-onyx/dist/Onyx.js#L675
    return clearPromise
        .then(() => Promise.all(promises.map((p) => p())))
+        .then(() => addSnapshots(data))
        .then(() => updateSnapshots(data))
        .then(() => undefined);

2. If we need to highlight and scroll to the message, we will use useSearchHighlightAndScroll and create currentReportAction and previousReportAction to identify the new item and scroll to it. We will discuss this behavior in the pull request if needed.

To test my proposal, you can download the file Onyx.js and replace the Onyx.js file in your node modules node_modules/react-native-onyx/dist/Onyx.js.

POC

Screen.Recording.2024-11-16.at.14.55.36.mp4

[mananjadhav]

This one's tricky. Reviewing the issue as well as proposals. Will post an update by Monday.

[mananjadhav]

Still reviewing.

[melvin-bot]

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

[melvin-bot]

@mananjadhav @CortneyOfstad this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[FitseTLT]

@luacmartins What do you think about my proposal here? I have proposed to auto updating and highlighting new chat messages in search the same as we are doing for expenses. I have also uploaded the result. It would also be nice if you self-assign as you are the head of search 👍 Thx

[melvin-bot]

@mananjadhav, @CortneyOfstad Huh... This is 4 days overdue. Who can take care of this?

[mananjadhav]

Reviewed this finally (took longer to navigate through all the code chaos). I think @FitseTLT's proposal should be enough to solve this one.

I don't think we should be updating onyx for this. But I'll let internal engineer confirm this one.

🎀 👀 🎀 C+ reviewed.

[huult]

#52826 Same RCA with this ticket

[melvin-bot]

@mananjadhav, @marcochavezf, @CortneyOfstad Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[mananjadhav]

@marcochavezf @luacmartins quick bump on this.

[marcochavezf]

Thanks for the review @mananjadhav! Looks like we don't need to update onyx from the backend, but I think we can confirm in the PR. Assigning @FitseTLT 🚀

[melvin-bot]

📣 @FitseTLT 🎉 An offer has been automatically sent to your Upwork account for the Contributor role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job
Please accept the offer and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻
Keep in mind: Code of Conduct | Contributing 📖

[melvin-bot]

@mananjadhav, @marcochavezf, @CortneyOfstad, @FitseTLT Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[muttmuure]

@FitseTLT do you have an ETA for your PR? Thanks!

[FitseTLT]

PR will be ready by Tomorrow 👍

[CortneyOfstad]

Awesome — thanks @FitseTLT!

[melvin-bot]

@mananjadhav @marcochavezf @CortneyOfstad @FitseTLT this issue is now 4 weeks old, please consider:

• Finding a contributor to fix the bug
• Closing the issue if BZ has been unable to add the issue to a VIP or Wave project
• If you have any questions, don't hesitate to start a discussion in #expensify-open-source

Thanks!

[melvin-bot]

@mananjadhav, @marcochavezf, @CortneyOfstad, @FitseTLT Eep! 4 days overdue now. Issues have feelings too...

[mananjadhav]

@FitseTLT Can you please post an update on the PR?

[FitseTLT]

It's ready now

[melvin-bot]

⚠️ Looks like this issue was linked to a Deploy Blocker here

If you are the assigned CME please investigate whether the linked PR caused a regression and leave a comment with the results.

If a regression has occurred and you are the assigned CM follow the instructions here.

If this regression could have been avoided please consider also proposing a recommendation to the PR checklist so that we can avoid it in the future.

[CortneyOfstad]

Hello! Just a heads up that I will be OOO starting this afternoon (December 20) and will be returning January 6th. A handful of folks on the BZ team will be online for a few days in between the 25th and the 1st, but we'll be operating with a skeleton crew. If any action is needed during that time, please post this issue in #expensify-open-source and someone on the team will jump in. Otherwise I will review when I am back on 1/6 — thanks!