self-signing keys are not returned from CS-API for user on different HS

[bwindels]

@lampholder is not seeing self-signing keys when calling /keys/query on his homeserver for neilj who's on matrix.org. This seems to happen only for this user (neilj), but is 100% reproducable. The master key is the correct one, with "self_signing_keys": {} at the end. Tom says he's running 1.12.3, I've asked him to repro on 1.12.4, but already wanted to report here as 1.12.4 apparently only contained fixes for workerized instances, which is not the case for Tom.

Exact output returned from the CS-API can be found at https://github.com/matrix-org/riot-web-rageshakes/issues/2658

This causes breakage in Riot as even after verifying somebody, they still appear with a red shield, as trust chain to their devices is incomplete.

[babolivier]

Wouldn't that be fixed by #7160? It was also shipped in 1.12.4.

[bwindels]

Wouldn't that be fixed by #7160? It was also shipped in 1.12.4.

Hmm, I don't see the immediate connection between that issue and this, but if you think so. Ideally Tom would retest with 1.12.4 but he is OOO as of today, so don't think he will be updating his synapse soon.

[anoadragon453]

Hm, yeah looking at the list of updates I don't think any of the fixes here would help with this.

Looking at my own non-worker HS (v1.12.4), I get the self-signing keys just fine:

{
  "device_keys": {
     ... blabla ...
  },
  "self_signing_keys": {
    "@neilj:matrix.org": {
      "keys": {
        "ed25519:PjWOJEVY2WL6po+pbkw1180RUBCrrGwnEw00fHB5pYc": "PjWOJEVY2WL6po+pbkw1180RUBCrrGwnEw00fHB5pYc"
      },
      "signatures": {
        "@neilj:matrix.org": {
          "ed25519:LL7r5ZWgrQlO5V/Yhus/+fGdv1iOTmyRt0geOMW8PSw": "0ucKZKR7nR4IzNOlg/kllX0gqB/BhoBCw9i1/dRCEe22GSy8jZCgtTsU2aL3AtjuzrvKj1mr02+SjwuiQwvJCw"
        }
      },
      "usage": [
        "self_signing"
      ],
      "user_id": "@neilj:matrix.org"
    }
  },
  "user_signing_keys": {}
}

[bwindels]

Tom told me he had seen Neil as red since the moment he started using cross-signing. So maybe some federation or other bug was persisted on his HS and never refetched from matrix.org afterwards 🤷‍♂️

[anoadragon453]

@lampholder Did you get a chance to test on v1.12.4 just in case?

[bwindels]

@lampholder is OOO fwiw

[neilisfragile]

@bwindels are you aware of any other occurrences in the wild? It would be good to reproduce.

[bwindels]

I am not, no.

[flackr]

I seem to also be running into this. My user @flackr:flack.undo.it has self_signing_keys when queried on my HS and my friend's HS, but not when queried from matrix.org. I set up my keys when cross signing support was first added to https://riot.im/develop. However, other users that have set up self_signing keys more recently (on the now stable cross signing support at https://riot.im/app) have had those keys federated to matrix.org.

Update: I tried wiping my cross signing keys and creating new ones and this fixed the issue.

[richvdh]

ok, as long as this is unreproducible, I don't think there's anything we can do here. matrix.org was struggling yesterday which could explain the earlier problems there.

[flackr]

I suspect that if you deploy an old version (whichever version was on Matrix.org around the initial cross signing riot release) of synapse on a homeserver A and then generate cross signing keys on another homeserver (newer version) B participating in some of the same rooms, that it may reproduce the issue even once A is upgraded. This seems to have happened to me and every one of my friends who generated cross signing keys on other homeservers as soon as the feature was available on the develop branch of riot.

[babolivier]

FTR I believe this issue might have the same cause as #7418

[flackr]

I just ran into this again on a new homeserver I created. I am missing everyone's master_keys and self_signing_keys except for people who have since set up their E2E cross signing keys. I added some details to #7418 in case it is the same issue however I figured I'd also comment here since having recently set up the homeserver I am pretty sure it could be reproduced.

During the initial setup, at one point I had not noticed that my dns provider had implicitly created a redirect for the root domain and was claiming the root domain, which meant that when I initially connected to a matrix.org channel my .well-known/matrix/server may not have been accessible at the root domain. This may have lead to it never synchronizing.

[flackr]

This may be naive, but could we force synchronizing keys when a verification request is received / sent to a user on another homeserver?