Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: only use shards #5474

Merged
merged 3 commits into from
Jul 15, 2024
Merged

refactor: only use shards #5474

merged 3 commits into from
Jul 15, 2024

Conversation

chaitanyaprem
Copy link
Contributor

Closes #5277

@chaitanyaprem chaitanyaprem changed the title fix: make tests use default shard fix: update tests to use default shard Jul 3, 2024
@chaitanyaprem chaitanyaprem changed the base branch from develop to feat/optimize-filter-subs July 3, 2024 09:29
@status-im-auto
Copy link
Member

status-im-auto commented Jul 3, 2024

Jenkins Builds

Click to see older builds (126)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 547a3e5 #1 2024-07-03 09:32:09 ~4 min linux 📦zip
✔️ 547a3e5 #1 2024-07-03 09:32:42 ~4 min ios 📦zip
✔️ 547a3e5 #1 2024-07-03 09:33:40 ~5 min android 📦aar
✔️ 547a3e5 #2 2024-07-03 09:35:58 ~2 min ios 📦zip
✖️ 547a3e5 #1 2024-07-03 09:36:42 ~8 min tests 📄log
✔️ 547a3e5 #2 2024-07-03 09:44:03 ~11 min linux 📦zip
✔️ 547a3e5 #2 2024-07-03 09:45:47 ~11 min android 📦aar
✖️ 547a3e5 #2 2024-07-03 10:23:49 ~46 min tests 📄log
✔️ 5d80b57 #3 2024-07-03 14:22:24 ~2 min ios 📦zip
✖️ 5d80b57 #3 2024-07-03 14:30:57 ~11 min tests 📄log
✔️ 5d80b57 #3 2024-07-03 14:31:26 ~11 min linux 📦zip
✔️ 5d80b57 #3 2024-07-03 14:33:45 ~14 min android 📦aar
✔️ 3369d26 #4 2024-07-03 15:55:21 ~5 min ios 📦zip
✔️ 3369d26 #4 2024-07-03 15:59:59 ~10 min android 📦aar
✔️ 3369d26 #4 2024-07-03 16:02:17 ~12 min linux 📦zip
✖️ 3369d26 #4 2024-07-03 16:03:15 ~13 min tests 📄log
✖️ 519e54d #5 2024-07-03 17:22:58 ~1 min tests 📄log
✔️ 519e54d #5 2024-07-03 17:23:44 ~2 min linux 📦zip
✔️ 519e54d #5 2024-07-03 17:24:31 ~2 min ios 📦zip
✔️ 519e54d #5 2024-07-03 17:26:56 ~5 min android 📦aar
✔️ cbbdb93 #6 2024-07-03 17:34:54 ~2 min ios 📦zip
✔️ cbbdb93 #6 2024-07-03 17:38:01 ~5 min android 📦aar
✔️ cbbdb93 #6 2024-07-03 17:38:59 ~6 min linux 📦zip
✖️ cbbdb93 #6 2024-07-03 17:42:36 ~10 min tests 📄log
✔️ 3a2a08f #7 2024-07-03 18:29:42 ~1 min android 📦aar
✔️ 3a2a08f #7 2024-07-03 18:29:44 ~1 min linux 📦zip
✔️ 3a2a08f #7 2024-07-03 18:30:20 ~2 min ios 📦zip
✖️ 3a2a08f #7 2024-07-03 19:06:12 ~38 min tests 📄log
✔️ 3a2a08f #9 2024-07-03 21:42:21 ~1 min android 📦aar
✔️ 3a2a08f #9 2024-07-03 21:43:01 ~1 min linux 📦zip
✔️ 3a2a08f #9 2024-07-03 21:43:33 ~2 min ios 📦zip
✖️ 3a2a08f #9 2024-07-03 21:46:25 ~5 min tests 📄log
✔️ 639c571 #8 2024-07-03 21:01:36 ~2 min linux 📦zip
✔️ 639c571 #8 2024-07-03 21:01:39 ~2 min android 📦aar
✔️ 639c571 #8 2024-07-03 21:02:32 ~3 min ios 📦zip
✖️ 639c571 #8 2024-07-03 21:36:28 ~36 min tests 📄log
✖️ 958551c #10 2024-07-03 22:17:01 ~1 min tests 📄log
✔️ 958551c #10 2024-07-03 22:18:02 ~2 min android 📦aar
✔️ 958551c #10 2024-07-03 22:18:08 ~2 min linux 📦zip
✔️ 958551c #10 2024-07-03 22:18:47 ~2 min ios 📦zip
✔️ f4ad281 #11 2024-07-03 22:25:45 ~1 min android 📦aar
✔️ f4ad281 #11 2024-07-03 22:26:11 ~1 min linux 📦zip
✔️ f4ad281 #11 2024-07-03 22:26:46 ~2 min ios 📦zip
✖️ f4ad281 #11 2024-07-03 22:47:58 ~23 min tests 📄log
✔️ f4ad281 #12 2024-07-03 23:43:44 ~43 min tests 📄log
✔️ f4ad281 #12 2024-07-04 05:07:23 ~2 min android 📦aar
✔️ f4ad281 #12 2024-07-04 05:07:26 ~2 min linux 📦zip
✔️ f4ad281 #12 2024-07-04 05:08:09 ~2 min ios 📦zip
✔️ f4ad281 #13 2024-07-04 05:50:13 ~44 min tests 📄log
f4d63d5 #13 2024-07-04 05:41:59 ~27 sec android 📄log
f4d63d5 #13 2024-07-04 05:42:03 ~30 sec ios 📄log
f4d63d5 #13 2024-07-04 05:42:06 ~31 sec linux 📄log
✔️ 5759c16 #14 2024-07-04 05:45:58 ~2 min android 📦aar
✔️ 5759c16 #14 2024-07-04 05:46:02 ~2 min linux 📦zip
✔️ 5759c16 #14 2024-07-04 05:46:43 ~2 min ios 📦zip
✔️ 4ae62d6 #15 2024-07-04 05:49:40 ~2 min android 📦aar
✔️ 4ae62d6 #15 2024-07-04 05:49:40 ~2 min linux 📦zip
✔️ 4ae62d6 #15 2024-07-04 05:50:05 ~2 min ios 📦zip
✔️ 4ae62d6 #14 2024-07-04 06:34:13 ~43 min tests 📄log
✔️ d59d451 #16 2024-07-05 00:39:57 ~2 min android 📦aar
✔️ d59d451 #16 2024-07-05 00:40:05 ~2 min linux 📦zip
✔️ d59d451 #1 2024-07-05 00:40:17 ~2 min tests-rpc 📄log
✔️ d59d451 #16 2024-07-05 00:41:25 ~3 min ios 📦zip
✔️ d59d451 #15 2024-07-05 01:22:08 ~44 min tests 📄log
✖️ d72fee2 #16 2024-07-10 09:04:52 ~1 min tests 📄log
✔️ d72fee2 #17 2024-07-10 09:06:36 ~3 min ios 📦zip
✔️ d72fee2 #17 2024-07-10 09:06:46 ~3 min linux 📦zip
✔️ d72fee2 #17 2024-07-10 09:07:29 ~4 min android 📦aar
✔️ 436ad54 #18 2024-07-10 09:09:09 ~2 min linux 📦zip
✖️ 436ad54 #3 2024-07-10 09:09:20 ~2 min tests-rpc 📄log
✔️ 436ad54 #18 2024-07-10 09:09:26 ~1 min android 📦aar
✔️ 436ad54 #18 2024-07-10 09:09:49 ~2 min ios 📦zip
✖️ 436ad54 #17 2024-07-10 09:13:21 ~6 min tests 📄log
✔️ 2470244 #19 2024-07-10 09:34:06 ~2 min ios 📦zip
✔️ 2470244 #5 2024-07-10 09:34:19 ~2 min tests-rpc 📄log
✔️ 2470244 #19 2024-07-10 09:34:49 ~3 min linux 📦zip
✔️ 2470244 #19 2024-07-10 09:35:35 ~4 min android 📦aar
✔️ 2470244 #18 2024-07-10 10:15:34 ~44 min tests 📄log
✖️ ea9fc20 #19 2024-07-13 01:15:57 ~1 min tests 📄log
✔️ ea9fc20 #6 2024-07-13 01:16:24 ~2 min tests-rpc 📄log
✔️ ea9fc20 #20 2024-07-13 01:16:43 ~2 min linux 📦zip
✔️ ea9fc20 #20 2024-07-13 01:16:43 ~2 min android 📦aar
✔️ ea9fc20 #20 2024-07-13 01:17:36 ~3 min ios 📦zip
✔️ 66fec5d #21 2024-07-13 01:20:52 ~1 min linux 📦zip
✔️ 66fec5d #7 2024-07-13 01:21:02 ~2 min tests-rpc 📄log
✔️ 66fec5d #21 2024-07-13 01:21:37 ~2 min ios 📦zip
✔️ 66fec5d #21 2024-07-13 01:23:19 ~4 min android 📦aar
✖️ 66fec5d #20 2024-07-13 01:57:34 ~38 min tests 📄log
✖️ 66fec5d #21 2024-07-13 02:58:30 ~37 min tests 📄log
✖️ 66fec5d #22 2024-07-13 04:54:17 ~37 min tests 📄log
✖️ 66fec5d #23 2024-07-13 05:34:38 ~37 min tests 📄log
✔️ 66fec5d #23 2024-07-15 07:02:08 ~1 min android 📦aar
✔️ 66fec5d #23 2024-07-15 07:02:34 ~1 min linux 📦zip
✔️ 66fec5d #9 2024-07-15 07:02:53 ~2 min tests-rpc 📄log
✔️ 66fec5d #23 2024-07-15 07:03:33 ~2 min ios 📦zip
✖️ 66fec5d #25 2024-07-15 07:38:43 ~37 min tests 📄log
✔️ a4ada5a #8 2024-07-13 23:13:33 ~2 min tests-rpc 📄log
✔️ a4ada5a #22 2024-07-13 23:13:47 ~2 min android 📦aar
✔️ a4ada5a #22 2024-07-13 23:14:33 ~3 min ios 📦zip
✔️ a4ada5a #22 2024-07-13 23:15:20 ~3 min linux 📦zip
✖️ a4ada5a #24 2024-07-13 23:49:21 ~37 min tests 📄log
✔️ ad22714 #10 2024-07-15 07:43:09 ~2 min tests-rpc 📄log
✔️ ad22714 #24 2024-07-15 07:43:11 ~2 min android 📦aar
✔️ ad22714 #24 2024-07-15 07:43:17 ~2 min linux 📦zip
✔️ ad22714 #24 2024-07-15 07:44:07 ~3 min ios 📦zip
✔️ ad22714 #26 2024-07-15 08:25:49 ~44 min tests 📄log
✔️ dc8ea45 #25 2024-07-15 08:15:20 ~1 min android 📦aar
✔️ dc8ea45 #25 2024-07-15 08:15:43 ~1 min linux 📦zip
✔️ dc8ea45 #11 2024-07-15 08:15:59 ~2 min tests-rpc 📄log
✔️ dc8ea45 #25 2024-07-15 08:16:39 ~2 min ios 📦zip
✔️ dc8ea45 #27 2024-07-15 09:10:21 ~44 min tests 📄log
✖️ 4853a00 #28 2024-07-15 10:13:23 ~56 sec tests 📄log
✔️ 4853a00 #12 2024-07-15 10:14:39 ~2 min tests-rpc 📄log
✔️ 4853a00 #26 2024-07-15 10:14:48 ~2 min linux 📦zip
✔️ 4853a00 #26 2024-07-15 10:15:49 ~3 min ios 📦zip
✔️ 4853a00 #26 2024-07-15 10:16:35 ~4 min android 📦aar
4d84752 #27 2024-07-15 10:36:35 ~32 sec ios 📄log
4d84752 #27 2024-07-15 10:36:38 ~32 sec linux 📄log
4d84752 #27 2024-07-15 10:36:40 ~36 sec android 📄log
✖️ 4d84752 #13 2024-07-15 10:37:10 ~1 min tests-rpc 📄log
✖️ 4d84752 #29 2024-07-15 10:37:26 ~1 min tests 📄log
✔️ f9ab393 #28 2024-07-15 10:54:43 ~1 min android 📦aar
✔️ f9ab393 #14 2024-07-15 10:55:22 ~2 min tests-rpc 📄log
✔️ f9ab393 #28 2024-07-15 10:55:30 ~2 min linux 📦zip
✔️ f9ab393 #28 2024-07-15 10:56:10 ~2 min ios 📦zip
✔️ f9ab393 #30 2024-07-15 11:37:59 ~44 min tests 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
✖️ 2d2be5c #31 2024-07-15 14:21:59 ~1 min tests 📄log
✔️ 2d2be5c #29 2024-07-15 14:23:09 ~2 min linux 📦zip
✔️ 2d2be5c #15 2024-07-15 14:23:22 ~2 min tests-rpc 📄log
✔️ 2d2be5c #29 2024-07-15 14:23:36 ~2 min android 📦aar
✔️ 2d2be5c #29 2024-07-15 14:24:42 ~3 min ios 📦zip
✔️ 2b5cde1 #30 2024-07-15 14:32:05 ~2 min android 📦aar
✔️ 2b5cde1 #16 2024-07-15 14:32:12 ~2 min tests-rpc 📄log
✔️ 2b5cde1 #30 2024-07-15 14:33:14 ~3 min ios 📦zip
✔️ 2b5cde1 #30 2024-07-15 14:33:41 ~3 min linux 📦zip
✖️ 2b5cde1 #32 2024-07-15 14:36:51 ~6 min tests 📄log
✔️ 2b5cde1 #33 2024-07-15 15:21:35 ~43 min tests 📄log

@richard-ramos richard-ramos changed the title fix: update tests to use default shard refactor: only use shards Jul 3, 2024
Copy link
Collaborator

@igor-sirotin igor-sirotin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙌

protocol/messenger_communities.go Outdated Show resolved Hide resolved
protocol/messenger_communities.go Outdated Show resolved Hide resolved
@richard-ramos richard-ramos force-pushed the fix/tests-shard branch 3 times, most recently from cbbdb93 to 3a2a08f Compare July 3, 2024 18:27
@richard-ramos
Copy link
Member

For the scenarios that are failing, I think we're sending the messages on the wrong pubsub topic, i.e.

rawMessage := common.RawMessage{
		Payload:             payload,
		CommunityID:         community.ID(),
		SkipEncryptionLayer: true,
		MessageType:         protobuf.ApplicationMetadataMessage_COMMUNITY_EDIT_SHARED_ADDRESSES,
		PubsubTopic:         community.PubsubTopic(), // TODO: confirm if it should be sent in community pubsub topic
		ResendType:          common.ResendTypeRawMessage,
	}

I think this should have probably been sent on shard.DefaultNonProtectedPubsubTopic() (If I change that TestMessengerCommunitiesTokenPermissionsSuite/TestEditSharedAddresses passes).

I'm investigating the other cases.

@richard-ramos
Copy link
Member

@jrainville, @mprakhov
Please do check the latest commit. I did change the pubsub topic of some messages.

The question is:
Can these messages be sent before the user has joined a community? If so, they should be send/received in either the default non protected topic or in the default shard, and not in the community shard. Pls let me know.

@jrainville
Copy link
Member

Can these messages be sent before the user has joined a community? If so, they should be send/received in either the default non protected topic or in the default shard, and not in the community shard. Pls let me know.

@richard-ramos both of these can only be called when the user has already joined the community.

@richard-ramos
Copy link
Member

I'll revert last commit then

@richard-ramos richard-ramos force-pushed the fix/tests-shard branch 2 times, most recently from 958551c to f4ad281 Compare July 3, 2024 22:24
@richard-ramos
Copy link
Member

'kay, all green now :)

@chaitanyaprem chaitanyaprem marked this pull request as ready for review July 4, 2024 01:22
Base automatically changed from feat/optimize-filter-subs to develop July 4, 2024 05:04
@chaitanyaprem chaitanyaprem force-pushed the fix/tests-shard branch 2 times, most recently from f4d63d5 to 5759c16 Compare July 4, 2024 05:43
@igor-sirotin
Copy link
Collaborator

Manual testing probably is good idea for this one?
cc @anastasiyaig @churik

@chaitanyaprem
Copy link
Contributor Author

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Yep, was going to create sttus-desktop PR with this.
But both of them seem to be OOO , not sure who will test.

@chaitanyaprem
Copy link
Contributor Author

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Here is the status-desktop PR status-im/status-desktop#15455

@igor-sirotin
Copy link
Collaborator

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Yep, was going to create sttus-desktop PR with this. But both of them seem to be OOO , not sure who will test.

Let's ping @glitchminer and @qoqobolo 🙂

@pavloburykh
Copy link

pavloburykh commented Jul 4, 2024

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Yep, was going to create sttus-desktop PR with this. But both of them seem to be OOO , not sure who will test.

@chaitanyaprem thank you for the PR and pinging QA team. Could you please rebase status go branch to the latest develop and create status mobile PR as well?

Also, would appreciate some details on what exactly has been affected and how should we test this PR from user perspective (what test scenarios/areas)? Thank you.

@chaitanyaprem
Copy link
Contributor Author

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Yep, was going to create sttus-desktop PR with this. But both of them seem to be OOO , not sure who will test.

@chaitanyaprem thank you for the PR and pinging QA team. Could you please rebase status go branch to the latest develop and create status mobile PR as well?

Also, would appreciate some details on what exactly has been affected and how should we test this PR from user perspective (what test scenarios/areas)? Thank you.

Done and here it is status-im/status-mobile#20658.
Not sure if many scenarios can be tested from mobile though, because they involve creating commuities and assigning shards to them. Maybe mobile can be used as a device to test send/receive messages and joini communities in this case.

@pavloburykh
Copy link

@chaitanyaprem I have left a comment in mobile PR, posting it here as well, so everyone is aware about the problem.

Unfortunately we cannot proceed with testing as currently we experience global Waku issues which result in communities failed to be fetched. This is a reason why e2e run in this PR has mostly failed.

I have already pinged Waku team about the problem, waiting for the fix.

Here is example of logs of a user from this PR who failed to fetch community when following community link.

Status-debug-logs - 2024-07-05T101140.382.zip

@chaitanyaprem chaitanyaprem force-pushed the fix/tests-shard branch 2 times, most recently from 436ad54 to 2470244 Compare July 10, 2024 09:30
@pavloburykh
Copy link

Hey @cammellos! What do you think, is this a priority to test this PR on mobile or we can rely on e2e results from our side? Note that our e2e do not cover sharded communities.

@cammellos
Copy link
Contributor

@pavloburykh thanks, given the priorities for 2.30, I don't think we have enough capacity in the QA team to test something that's not strictly needed for 2.30. If @chaitanyaprem thinks that this is not impacting anything unless the community is manually sharded, then e2e tests is probably good enough, but if it impacts production code, best to leave it for later

@chaitanyaprem
Copy link
Contributor Author

chaitanyaprem commented Jul 13, 2024

@pavloburykh thanks, given the priorities for 2.30, I don't think we have enough capacity in the QA team to test something that's not strictly needed for 2.30. If @chaitanyaprem thinks that this is not impacting anything unless the community is manually sharded, then e2e tests is probably good enough, but if it impacts production code, best to leave it for later

afaik it doesn't affect any other code apart from sharded communities.
Also, i have tested by creating a community and assigning it a shard 128 via RPC (which requires a privateKey) to be specified and i was able to fetch the community from both mobile and desktop.
Maybe there is a bug in the UI for assigning shard which is causing the issue.

But i'll want a yes from the great @richard-ramos as well to be sure.

@richard-ramos
Copy link
Member

IMO it should be fine to merge this PR. The normal behavior of the app is maintained, and I think that as long as the custom shard RPC methods still work fine, which as far as we know, it is the case, we should be good to go!

@chaitanyaprem chaitanyaprem force-pushed the fix/tests-shard branch 4 times, most recently from ad22714 to dc8ea45 Compare July 15, 2024 08:13
@chaitanyaprem chaitanyaprem force-pushed the fix/tests-shard branch 2 times, most recently from 4853a00 to 4d84752 Compare July 15, 2024 10:35
@chaitanyaprem chaitanyaprem merged commit 9a70316 into develop Jul 15, 2024
10 checks passed
@chaitanyaprem chaitanyaprem deleted the fix/tests-shard branch July 15, 2024 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

chore: update all tests that use default pubsubTopic to use shard based pubsubTopic
8 participants