Hash 3PID lookups

[anoadragon453]

The implementation of matrix-org/matrix-spec-proposals#2134

TODO:

• Support hashed lookups
• Add column to the db for pre-computed hashes
• Re-compute hashes for all 3PIDs on startup when pepper changes
• Store previous pepper in the db so we know it changed
• Hash addresses as they come in via /bind and replication servlets
• Recompute lookup hashes if the algorithms in the config changed

We need to have a think about how this will work in the replication use case in general. Obviously we don't want to replicate lookup hashes as the receiving server may have a different pepper/algorithm in use than the sending server.

When we receive an association, we just rehash the address/medium combo with our own lookup pepper.

[anoadragon453]

Heads up that CI is probably never going to pass on this.

[anoadragon453]

Removed my assignment as this PR is currently waiting for review.

[erikjohnston]

Some preliminary thoughts.

Whats the deal with CI? This can't really get merged with it failing. Also, what about some unit tests?

[anoadragon453]

Whats the deal with CI? This can't really get merged with it failing.

CI has been turned on but without a pipeline, hence it failing.

Also, what about some unit tests?

Blocked on #171

[erikjohnston]

Don't we need to go back and hash the existing stuff?

[anoadragon453]

That will be done on first run. Sydent checks to see if a pepper is configured in the database, and if not, generate one and rehash all 3PIDs:

sydent/sydent/sydent.py

Lines 186 to 194 in 424270b

	hashing_metadata_store = HashingMetadataStore(self)
	lookup_pepper = hashing_metadata_store.get_lookup_pepper()
	if not lookup_pepper:
	# No pepper defined in the database, generate one
	lookup_pepper = generateAlphanumericTokenOfLength(5)
	# Store it in the database and rehash 3PIDs
	hashing_metadata_store.store_lookup_pepper(sha256_and_url_safe_base64,
	lookup_pepper)

[erikjohnston]

Hmm, OK. Though please add a comment there to say that it must be run before we start serving requests.

[erikjohnston]

Does this all work correctly with canonicalised emails? I.e. things like case-insensitive etc.

[anoadragon453]

Does this all work correctly with canonicalised emails? I.e. things like case-insensitive etc.

Sydent continues to make use of the lower() function here: https://github.com/matrix-org/sydent/pull/184/files#diff-c5c56bd2a12d20e5d6f24879fd48d15eR154

@babolivier What is the status of updating this on master?

[anoadragon453]

@babolivier says the current state using lower() is fine. We're using lower case emails everywhere.

[erikjohnston]

Just a couple of trivial nits now

[anoadragon453]

@erikjohnston https://matrix.org/docs/spec/identity_service/unstable#get-matrix-identity-api-v1 oh, maybe we should keep the empty 200 return on GET .../v2.

[anoadragon453]

@erikjohnston https://matrix.org/docs/spec/identity_service/unstable#get-matrix-identity-api-v1 oh, maybe we should keep the empty 200 return on GET .../v2.

Have done this. Simple to remove in the future if we want to clean up that part of the spec for any reason.

[dbkr]

Only other question is why the local threepid assocs have hashes? Wouldn't we only ever be matching against the global ones?

[anoadragon453]

Only other question is why the local threepid assocs have hashes? Wouldn't we only ever be matching against the global ones?

Conclusion is it's not necessary but we'll remove it in a later PR. (issue #196)