An event was sent twice down /sync (and down application service txns?) consistently across clients; ghost notifications
#14539
Labels
Comments
matrixbot
changed the title
/sync (and down application service txns?) consistently across clients; ghost notifications
|
I haven't had any issues which I described as "ghost notifications" for quite some time now. So I think that was fixed somehow. The situation with the "Event duplication" relaxed a lot for a long time, but increased noticeably in the past couple of weeks. The events that were "duplicated" all seem to come from a particular time span, as before, where we experienced a distributed join spam attack with >35k users. On Dec 16 2023 we had an interesting "conversation" with our bot (mjolnir). We told it to ban a user, just as usual, and it told us that it isn't in the room anymore: So I first checked the room member list in element-web (with sliding-sync) on my instance, and it was still there, but it had the avatar and display name from over a year ago. On the The last join the bot logged happened at 2023-12-07 11:02 and the next join took place at 2023-12-07 11:34:00 (which was not logged anymore by the bot). So, whatever went sideways happened, with high certainty between Asking the database what the current state event of the bot is-- Query the current membership state event for the bot
SELECT json::JSONB, event_id
FROM event_json
WHERE event_id IN
(SELECT event_id
FROM current_state_events
WHERE room_id='!iuyQXswfjgxQMZGrfQ:matrix.org'
AND TYPE='m.room.member'
AND state_key='@bot.mjolnir:michaelsasser.org'
);// The response to that query was an old state event which should not be the
// current one. Though, from the origin_server_ts it seems to be consistent
// with the old avatar and displayname I initially saw in the room user
// directory on element-web which sliding-sync enabled.
// $cb_QySiCLd5FLanxgrgnYzPLG83U2zO69PHwGj4Cxu8
{
type: "m.room.member",
depth: 45711,
hashes: {
sha256: "Y4IQHedNuOc1PQ/bZAEUnZw3asz/dcUr8vINr3hhPN0",
},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
membership: "leave",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org", // Python Room: @python:matrix.org
unsigned: {
replaces_state: "$qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8",
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "L3HB/fwK4lpQRMcBcXSJQkc9PnY1zxucjJ8w25jAwLwHINT1HWPt4d1IHOg7ldIg3/+MbW/L8ax+RRRg0aYmCA",
},
},
auth_events: [
"$qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
],
prev_events: ["$6DeuJJjdF7lkemjqQOxCJavCfg0_-cixJ_vYyTaShrw"],
origin_server_ts: 1643233431682, // Wednesday, January 26, 2022 9:43:51.682 PM
}Normally, it should be or more precisely, it should have responded to with this state event: -- Query the latest (by time) membership state event for the bot
SELECT json::JSONB, event_id
FROM event_json
WHERE event_id IN
(SELECT event_id
FROM state_events
WHERE room_id='!iuyQXswfjgxQMZGrfQ:matrix.org'
AND TYPE='m.room.member'
AND state_key='@bot.mjolnir:michaelsasser.org'
)
ORDER BY json::JSONB->'origin_server_ts' DESC
LIMIT 1;// $_6N5ltJ9hg35sk4j8Do7PTNNzlMjcZSk7p8qlJ51fFA
{
type: "m.room.member",
depth: 118579,
hashes: {
sha256: "YZ/qfnBF+RlFUO8vaXpPep6UVcAgc9sVbue8oUDVaAU",
},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/kPMJgudCvuUzCIrOeJuUrPTn",
membership: "join",
displayname: "Administrator",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
age_ts: 1669213202762,
replaces_state: "$rST-rCiCfl8z0zZVSlBWZEeuVX8oy40SFOkrlNoJ5RY",
},
state_key: "@bot.mjolnir:michaelsasser.org",
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "gfeD/i2blF1OSX7A3D59lGsDMBeQx0zdfmmP6ZK7D/VKcwjI1tTm9zDLS+//Nq07qKUsGx9KJWF9WuhUTdg7CQ",
},
},
auth_events: [
"$rST-rCiCfl8z0zZVSlBWZEeuVX8oy40SFOkrlNoJ5RY",
"$Ur-nGignA9Bqyyv-YRC5oH8UkGHzqgoqevuzM_m8Jn8",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$tAy72IFE7UsohvKVrhH8jbolxKq0N0qM1P7cClN9xIE",
],
prev_events: ["$_rx-nDqmJuZW5Tu5qRP3xdLbuFSm3wby970RG3JP1wU"],
origin_server_ts: 1669213202762, // Wednesday, November 23, 2022 2:20:02.762 PM
}Just for reference: Asking the database about all the membership state events of the bot-- Get all membership state events for the bot and sort them by `depth`
SELECT json::JSONB, event_id
FROM event_json
WHERE event_id IN
(SELECT event_id
FROM state_events
WHERE room_id='!iuyQXswfjgxQMZGrfQ:matrix.org'
AND TYPE='m.room.member'
AND state_key='@bot.mjolnir:michaelsasser.org'
)
ORDER BY json::JSONB->'depth';// Result
[
// $Mq-oxPctUHcgjuhnqv3HCcK9q-shzshhjrVVHkb-wv4
{
type: "m.room.member",
depth: 14212,
hashes: {sha256: "9xr2Ni1HNNarZySoGM26qaVElKgUEA1FAky3zwr2iWE"},
origin: "michaelsasser.org",
sender: "@michael:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/XGOSHXVNMogAChjWByqRsYTT",
membership: "invite",
displayname: "Abuse Management",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
age: 48290275580,
invite_room_state: [
{
type: "m.room.avatar",
sender: "@michael:michaelsasser.org",
content: {
url: "mxc://michaelsasser.org/LQdedDsAKKrwxEFWZvRLuSfa",
},
state_key: "",
},
{
type: "m.room.canonical_alias",
sender: "@michaelsasser:matrix.org",
content: {alias: "#Python3:matrix.org"},
state_key: "",
},
{
type: "m.room.join_rules",
sender: "@michaelsasser:matrix.org",
content: {join_rule: "public"},
state_key: "",
},
{
type: "m.room.member",
sender: "@michael:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/gHeydUcioJByDfoMSwRRfNHc",
membership: "join",
displayname: "Michael",
},
state_key: "@michael:michaelsasser.org",
},
{
type: "m.room.name",
sender: "@michaelsasser:matrix.org",
content: {name: "Python"},
state_key: "",
},
],
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "lzQWtRz17YC9EONtraKyCSbO/EZtN4y9FJ/YZfM0Tc8xfUuqB6VdzpSRH0Jyl31tCTeRtyKwmrNmsi4MkKrkAQ",
},
},
auth_events: [
"$nrVVXusAZIdWQ2aODw5XXs-A5F7iUvvEEytQzheVuY0",
"$wslk8IDCsILP43g8VqaAH8TsThP8_zsC4Vzr_Fi6GYw",
"$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
],
prev_events: ["$Fd0Y1VSUzbxMenKzpWZ_sm08VGVMuBmVje8u--ftM3c"],
origin_server_ts: 1620665527678, // Monday, May 10, 2021 4:52:07.678 PM
},
// $CosN_FgJRmvn-xqxywY5R3XVsdv5Wig6yetgygioAcw
{
type: "m.room.member",
depth: 14213,
hashes: {sha256: "WRC7Z8HHRTvHYQ/KLZDqBzh8Ipo+aOzLepXlwjxe6N0"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/XGOSHXVNMogAChjWByqRsYTT",
membership: "join",
displayname: "Abuse Management",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
age: 48290275158,
replaces_state: "$Mq-oxPctUHcgjuhnqv3HCcK9q-shzshhjrVVHkb-wv4",
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "6BX/DbQRP6jNgLvxevgOPp0w1RJoRl+w6cT8Tls1CXUTTYQpszbXdGDFBpINK6qBYq+crnxNgGuA+nMhaZkXCA",
},
},
auth_events: [
"$Mq-oxPctUHcgjuhnqv3HCcK9q-shzshhjrVVHkb-wv4",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$wslk8IDCsILP43g8VqaAH8TsThP8_zsC4Vzr_Fi6GYw",
"$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
],
prev_events: ["$Mq-oxPctUHcgjuhnqv3HCcK9q-shzshhjrVVHkb-wv4"],
origin_server_ts: 1620665528100, // Monday, May 10, 2021 4:52:08.100 PM
},
// $kj0hopGM88K3qvvIJXTFrTZP7kaLtFdDCqBNg__RVXg
{
type: "m.room.member",
depth: 14314,
hashes: {sha256: "vCrtMIDJwJ0b31ZDJXnfwblMJBED/ysnhpn8duqVKFs"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
membership: "join",
displayname: "Abuse Management",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
age: 48095980698,
replaces_state: "$CosN_FgJRmvn-xqxywY5R3XVsdv5Wig6yetgygioAcw",
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "4pBBdTMrUXURB41sgE2hHr4PKfaZlTlMWrQLNhQ505K8say2vkeCiMuNd6tsNgmeQ8vwtwNalbN3S9EmLygzAw",
},
},
auth_events: [
"$CosN_FgJRmvn-xqxywY5R3XVsdv5Wig6yetgygioAcw",
"$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
],
prev_events: ["$uP758I6_pK_L_LqUvLzNPB18ZRCpkg7XtbDRV1oFkiQ"],
origin_server_ts: 1620859822560, // Wednesday, May 12, 2021 10:50:22.560 PM
},
// $-n0TlZQ3ae9mXQKeAVFdBfxbO_-0wIwY4bIguMkaAgU
{
type: "m.room.member",
depth: 45689,
hashes: {sha256: "YqYhi1uyc9Bii1PF9M+vEg5vNhWhaD2kKCgS50vIa7U"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {membership: "leave"},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
replaces_state: "$kj0hopGM88K3qvvIJXTFrTZP7kaLtFdDCqBNg__RVXg",
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "NZCj21eFB0BUe5y8XRhwzPd4yr7woH1bYUXOidmljzxF3YMvLLBbYES2VfA3OOHkJL4Viws3Y7RbNEwHNfhzDA",
},
},
auth_events: [
"$kj0hopGM88K3qvvIJXTFrTZP7kaLtFdDCqBNg__RVXg",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
],
prev_events: ["$cgxH6iDVSpvPSZXsQSEWZB-ZJGIzMeFd9uL2SDGihvU"],
origin_server_ts: 1643233112395, // Wednesday, January 26, 2022 9:38:32.395 PM
},
// $qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8
{
type: "m.room.member",
depth: 45706,
hashes: {sha256: "/uCHlAoGBl3Ge9ZrTRRuubvdgMExJj6eVBIQT2M9KTU"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
membership: "join",
displayname: "Abuse Management",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
replaces_state: "$-n0TlZQ3ae9mXQKeAVFdBfxbO_-0wIwY4bIguMkaAgU",
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "X4Nm2EtObxA5L4jEs40uLJA90lq61RCZB0CW7d0pY0EFfCAnB2Ur69n81FW6fJLHkaNOr42p6NKSFGFvLB/lAA",
},
},
auth_events: [
"$-n0TlZQ3ae9mXQKeAVFdBfxbO_-0wIwY4bIguMkaAgU",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
"$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
],
prev_events: ["$rmOzmrZbOut2UK43mAASkCHZBeL2duUte5PbjgvUH0E"],
origin_server_ts: 1643233352820, // Wednesday, January 26, 2022 9:42:32.820 PM
},
// $cb_QySiCLd5FLanxgrgnYzPLG83U2zO69PHwGj4Cxu8
{
type: "m.room.member",
depth: 45711,
hashes: {sha256: "Y4IQHedNuOc1PQ/bZAEUnZw3asz/dcUr8vINr3hhPN0"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {membership: "leave"},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
replaces_state: "$qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8",
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "L3HB/fwK4lpQRMcBcXSJQkc9PnY1zxucjJ8w25jAwLwHINT1HWPt4d1IHOg7ldIg3/+MbW/L8ax+RRRg0aYmCA",
},
},
auth_events: [
"$qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
],
prev_events: ["$6DeuJJjdF7lkemjqQOxCJavCfg0_-cixJ_vYyTaShrw"],
origin_server_ts: 1643233431682, // Wednesday, January 26, 2022 9:43:51.682 PM
},
// $bycEakqs8rCaoj4R6p5d8SMEw3u-X95a5cPcTkhjQUo
{
type: "m.room.member",
depth: 45759,
hashes: {sha256: "lUbjxLBtAMdFsP2eCzehw+/q61eewB2lUd6Db2O/RRM"},
origin: "michaelsasser.org",
sender: "@michael:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
membership: "invite",
displayname: "Abuse Management",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
replaces_state: "$cb_QySiCLd5FLanxgrgnYzPLG83U2zO69PHwGj4Cxu8",
invite_room_state: [
{
type: "m.room.avatar",
sender: "@michael:michaelsasser.org",
content: {
url: "mxc://michaelsasser.org/LQdedDsAKKrwxEFWZvRLuSfa",
},
state_key: "",
},
{
type: "m.room.canonical_alias",
sender: "@michaelsasser:matrix.org",
content: {alias: "#python:matrix.org", alt_aliases: []},
state_key: "",
},
{
type: "m.room.create",
sender: "@michaelsasser:matrix.org",
content: {
creator: "@michaelsasser:matrix.org",
room_version: "6",
},
state_key: "",
},
{
type: "m.room.join_rules",
sender: "@michaelsasser:matrix.org",
content: {join_rule: "public"},
state_key: "",
},
{
type: "m.room.member",
sender: "@michael:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/gHeydUcioJByDfoMSwRRfNHc",
membership: "join",
displayname: "Michael",
},
state_key: "@michael:michaelsasser.org",
},
{
type: "m.room.name",
sender: "@michaelsasser:matrix.org",
content: {name: "Python"},
state_key: "",
},
],
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "YUR6B4c3y7gvB3NW00Aq4Q3D1aUhr/1OqBmKTGvgxpFfUWMeqnYZV2PnQgGy/dfhVRno3iEuKnKKh/WfJoXBDw",
},
},
auth_events: [
"$cb_QySiCLd5FLanxgrgnYzPLG83U2zO69PHwGj4Cxu8",
"$nrVVXusAZIdWQ2aODw5XXs-A5F7iUvvEEytQzheVuY0",
"$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
"$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
],
prev_events: ["$wbhx9QDWZzbc60qvGmSpfpr8F-WGA6_hF98lU03O7Rc"],
origin_server_ts: 1643273377451, // Thursday, January 27, 2022 8:49:37.451 AM
},
// $AwKIg7YRtAfl1wbsrm40_gqsXJsibib3FpJX7I7CRrI
{
type: "m.room.member",
depth: 45760,
hashes: {sha256: "tw+taBgFb6YyFQfuK1jIYpixXGDVdci7kn7Fzv3AAlc"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
membership: "join",
displayname: "Abuse Management",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
replaces_state: "$bycEakqs8rCaoj4R6p5d8SMEw3u-X95a5cPcTkhjQUo",
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "ufky5xnc8k6NfeT/pii6NGqeh3EexIY3ftygEwZrldlbVIwiIt2nY56J8CySWEN+mRNjl8C8uTJp9w66Ja4tCQ",
},
},
auth_events: [
"$bycEakqs8rCaoj4R6p5d8SMEw3u-X95a5cPcTkhjQUo",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
"$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
],
prev_events: ["$bycEakqs8rCaoj4R6p5d8SMEw3u-X95a5cPcTkhjQUo"],
origin_server_ts: 1643273378028, // Thursday, January 27, 2022 8:49:38.028 AM
},
{
type: "m.room.member",
depth: 50867,
hashes: {sha256: "Yeo4hdZuxBzZElV1uQ93Sl1YryUPg3LqdcGbSp/V2tA"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
membership: "join",
displayname: "Administrator",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
replaces_state: "$AwKIg7YRtAfl1wbsrm40_gqsXJsibib3FpJX7I7CRrI",
},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "kKrHKoi/r6mhu2VdRZzlRDpEFfKImyLGuy0AL6NcLekhctEk/xAW73HLR0531K+fQmjvcyoCOuy9C9KxuMzBCQ",
},
},
auth_events: [
"$AwKIg7YRtAfl1wbsrm40_gqsXJsibib3FpJX7I7CRrI",
"$tUYE1mFeqtmd09uaK36r5eW_m1KFEWzYa4ji7ga2EKg",
"$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
],
prev_events: ["$xKb1Fofi9ztN23BeejTeFYOBxETkwrly0zkTXYYKPzg"],
origin_server_ts: 1647259553837, // Monday, March 14, 2022 12:05:53.837 PM
},
// $NfdoJih78gBbTys_V3FeE9tv9-JZG69Lbqqz8fWjeoA
{
type: "m.room.member",
depth: 59399,
hashes: {sha256: "45CskOjxJmqcDc+6mvUV6g0d2a4M+59nweH9eIqvCv8"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/FtcRMfFgpJnpLzWZoFKQNMMd",
membership: "join",
displayname: "Administrator",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "2IAZoyf4yTlYoCjtFX1IOKV5xLWhTC+NmY/3axu0sWYn9V21m3sXiU7928p1gqf/AYRG2nNMIpg0L/+XeiOqCg",
},
},
auth_events: [
"$NfdoJih78gBbTys_V3FeE9tv9-JZG69Lbqqz8fWjeoA",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$hUZdUKqNn6maEll77yE0z2467P4HQgVg95wTygi5Gvg",
"$9sNpU5w38YrbcRBzx6IKGRVfv-T8-D6IS_lTVBEckro",
],
prev_events: [
"$BnWnDOK5G73hBFDCoR3wmOy2zgFfmGtUp_AzLvEBFzs",
"$s5L-e6Vmbo3R3DVLXNYEncx_GWOH3DEEFoVGgkFeGEg",
],
origin_server_ts: 1651987205907, // Sunday, May 8, 2022 5:20:05.907 AM
},
// $b0o1xm8AIuTFDAz7f_lC89f1JEvMGO5rjEdyNffd2Wg
{
type: "m.room.member",
depth: 59400,
hashes: {sha256: "R1HILYpnatdcckdcCm+WdkqswRCi0ZKaYqOVAegO7LU"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/kPMJgudCvuUzCIrOeJuUrPTn",
membership: "join",
displayname: "Administrator",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {},
state_key: "@bot.mjolnir:michaelsasser.org",
prev_state: [],
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "p4Lae6lJ060FkmXB5TLk2hgKLwAapYculSa2tyRgk5DTp8d4WIFr9T6TZZ8yU1w0u1ybsziqPGjymf60hB36DA",
},
},
auth_events: [
"$qX_SKHxi7hCBj7OL4PEhT8rM4f3_MX33Asx95z5kCJQ",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$hUZdUKqNn6maEll77yE0z2467P4HQgVg95wTygi5Gvg",
"$9sNpU5w38YrbcRBzx6IKGRVfv-T8-D6IS_lTVBEckro",
],
prev_events: ["$qX_SKHxi7hCBj7OL4PEhT8rM4f3_MX33Asx95z5kCJQ"],
origin_server_ts: 1651990924046, // Sunday, May 8, 2022 6:22:04.046 AM
},
// $SKK1_K51jxGTJcUueDFVoHajkXGTRFMa-IMoIrmz6q4
{
type: "m.room.member",
depth: 118048,
hashes: {sha256: "5rDK7MwQnO8NWFkWyf1tzvPAXwkLUXcp3iYaM8rFHuk"},
origin: "matrix.org",
sender: "@michaelsasser:matrix.org",
content: {membership: "leave"},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
age: 8635818,
replaces_state: "$b0o1xm8AIuTFDAz7f_lC89f1JEvMGO5rjEdyNffd2Wg",
},
state_key: "@bot.mjolnir:michaelsasser.org",
signatures: {
"matrix.org": {
"ed25519:a_RXGa": "nI72s0AUVy1v6k/la2SeDUSdxvdqp+SZuURrOAA5dU40kIS+BMtDpd8ft39iqT6iw+ryXD6MQ4ZletG8TPMGCA",
},
},
auth_events: [
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$A-7fXhjCI7UazqdD0cz_1Srzg-493XoCBsq2abAkSNc",
"$b0o1xm8AIuTFDAz7f_lC89f1JEvMGO5rjEdyNffd2Wg",
"$rGDjDeraGK57FgbQOhS6nfpiJ-Y-4jKiKEcpYtVQcUI",
],
prev_events: ["$A-7fXhjCI7UazqdD0cz_1Srzg-493XoCBsq2abAkSNc"],
origin_server_ts: 1668947167440, // Sunday, November 20, 2022 12:26:07.440 PM
},
// $iexffkStREHb4KcvmBLS4QiOlxvUouKvWKUdp6vuojk
{
type: "m.room.member",
depth: 118151,
hashes: {sha256: "V7lanv+6Ib+1vB/8gG3tiH4pB58gUTU3Qe/9gzanhM0"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/kPMJgudCvuUzCIrOeJuUrPTn",
membership: "join",
displayname: "Administrator",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
age_ts: 1668989318053,
replaces_state: "$SKK1_K51jxGTJcUueDFVoHajkXGTRFMa-IMoIrmz6q4",
},
state_key: "@bot.mjolnir:michaelsasser.org",
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "LCJBRa5oliq/RfahJa8NxmSAGrfbdzdss3AyG3XABSx7MbBUoHzsu2Ati78mv4EgdoeRvX9LS+UN5a8X9s2GBQ",
},
},
auth_events: [
"$tAy72IFE7UsohvKVrhH8jbolxKq0N0qM1P7cClN9xIE",
"$Ur-nGignA9Bqyyv-YRC5oH8UkGHzqgoqevuzM_m8Jn8",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$SKK1_K51jxGTJcUueDFVoHajkXGTRFMa-IMoIrmz6q4",
],
prev_events: ["$UChrHiRIGdgPpDGD4jFa4G5oB-umimKO3yjr8YrfFyo"],
origin_server_ts: 1668989318053, // Monday, November 21, 2022 12:08:38.053 AM
},
// $iexffkStREHb4KcvmBLS4QiOlxvUouKvWKUdp6vuojk
{
type: "m.room.member",
depth: 118570,
hashes: {sha256: "Dhd8n5Jq0HAY+AgWIseGV5BVCmXDGJLSHRcFLC79Gis"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {membership: "leave"},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
age_ts: 1669200418456,
replaces_state: "$iexffkStREHb4KcvmBLS4QiOlxvUouKvWKUdp6vuojk",
},
state_key: "@bot.mjolnir:michaelsasser.org",
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "6KBJbMEQ4cbEV7O1I/5sjg/i4+8gmf4xzBoOHlSi5fkgU/WtOuFsszZXzjOaO9+F3tVufQON/OSXw9YBhD6mBg",
},
},
auth_events: [
"$iexffkStREHb4KcvmBLS4QiOlxvUouKvWKUdp6vuojk",
"$Ur-nGignA9Bqyyv-YRC5oH8UkGHzqgoqevuzM_m8Jn8",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
],
prev_events: ["$U7DFEXnnyuZ2ez6znqUaLeJodQTAcTN1hsDDX6f5ZTg"],
origin_server_ts: 1669200418456, // Wednesday, November 23, 2022 10:46:58.456 AM
},
// $_6N5ltJ9hg35sk4j8Do7PTNNzlMjcZSk7p8qlJ51fFA
{
type: "m.room.member",
depth: 118579,
hashes: {sha256: "YZ/qfnBF+RlFUO8vaXpPep6UVcAgc9sVbue8oUDVaAU"},
origin: "michaelsasser.org",
sender: "@bot.mjolnir:michaelsasser.org",
content: {
avatar_url: "mxc://michaelsasser.org/kPMJgudCvuUzCIrOeJuUrPTn",
membership: "join",
displayname: "Administrator",
},
room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
unsigned: {
age_ts: 1669213202762,
replaces_state: "$rST-rCiCfl8z0zZVSlBWZEeuVX8oy40SFOkrlNoJ5RY",
},
state_key: "@bot.mjolnir:michaelsasser.org",
signatures: {
"michaelsasser.org": {
"ed25519:a_QNjX": "gfeD/i2blF1OSX7A3D59lGsDMBeQx0zdfmmP6ZK7D/VKcwjI1tTm9zDLS+//Nq07qKUsGx9KJWF9WuhUTdg7CQ",
},
},
auth_events: [
"$rST-rCiCfl8z0zZVSlBWZEeuVX8oy40SFOkrlNoJ5RY",
"$Ur-nGignA9Bqyyv-YRC5oH8UkGHzqgoqevuzM_m8Jn8",
"$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
"$tAy72IFE7UsohvKVrhH8jbolxKq0N0qM1P7cClN9xIE",
],
prev_events: ["$_rx-nDqmJuZW5Tu5qRP3xdLbuFSm3wby970RG3JP1wU"],
origin_server_ts: 1669213202762, // Wednesday, November 23, 2022 2:20:02.762 PM
},
]//edit 2024-01-09
|
|
Thanks for the update and your thorough investigation. I'm no longer working on Matrix, but I'm sure the Synapse team will take a look at this as part of issue review. |
|
Since the last edit (2024-01-09), where power levels reverted suddenly, we have seen something like this happening a few times, Topics revert to a previous state, Users are suddenly no longer joined to the room, Homeservers who have joined the room before and have not purged the room from the homeserver can no longer join the room (we asked for logs, or more information but none of them were able to provide them). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue has been migrated from #14539.
Description
Ghost notifications
Since synapse v1.70 or v1.71, I've noticed randomly occurring "ghost notification", happening on element-web, element-desktop and element-android, hydrogen. They usually come in small, fast bursts of two to four notifications (I hear an audible sound telling me about new events in a room). When I check for new events, there aren't any. Nothing has changed. I have the options
Developer modeandShow hidden events in timelineenabled in element, so I would assume, I'd see any changes to a room.Event duplication
Around the same Time, I've noticed some events, some from weeks or months ago, from being randomly duplicated and pushed as new ones. This happens independently of the "ghost notification issue". At first, I've only noticed this in one particular room, where the same messages were duplicated a few times over a couple of days (Example: see "Image 1" below), possibly leading to an unstable room (see "Possible implications). Today, the same thing happened in our mjolnir control room, where mjolnir (configured as a client and not as an appservice) executed the duplicated command (Example: see "Image 2" below). Actually, I'm actually not certain, if my homeserver is the one, the issues originate from, as the duplicated events are visible on other instances too, including
matrix.organd maybe others as well.Image 1: Duplicated messages in the python room
Image 2: Mjolnir bot reacts to duplicated events
The "failed to leave" error from mjolnir is unrelated (and already fixed). The duplication errors happened before the room, it was trying to leave, had issues, too, and might have led to the issue, where the room was becoming unstable on my homeserver.
Possible implications
After this happened a few times, in the Python room
#python:matrix.org, the room became unstable on my end, and I was unable to send events to that room any more because of missing state events. I also wasn't able to remove the room with the admin API, so, I manually had to remove the room from the database to gain access again:(Because I had to roll back a few times, while I was trying to figure out, how I can remove the room, I accidentally deleted the logs as well.)
Steps to reproduce
Homeserver
matrix.org; michaelsasser.org
Synapse Version
matrixdotorg/synapse; tags: v1.71.0, v1.72.0, possibly v1.70.0
Installation Method
Docker (matrixdotorg/synapse)
Database
postgres:15.0-alpine on same machine, It was recently restored to upgrade to v15, not sure, if it happeded before; recently restored while trying to removing room, issues happened before
Workers
Multiple workers
Platform
Environment: CX31 Hetzner Cloud
Using the Ansible playbook: spantaleev/matrix-docker-ansible-deploy
Debian: Linux matrix 5.10.0-19-amd64 element-hq/element-web#1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux
Docker: Docker version 20.10.21, build baeda1f
Containerd: containerd containerd.io 1.6.10 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
Configuration
Playbook
Using the Ansible playbook: [spantaleev/matrix-docker-ansible-deploy], contains multiple files (result below):
Playbook/config
Results in:
homeserver.yaml
Workers
Relevant log output
Anything else that would be useful to know?
Full log during the time the duplication, see "Image 2" happened: message_duplication.log
All times & dates are UTC+1.
The text was updated successfully, but these errors were encountered: