Link preview duplicate url filter considers anchors as unique element

[eras]

Copied from matrix-org/synapse#10088, but Element is the correct component for this issue.

Description

When receiving an URL in a room, a link preview is generated. If the same URL is seen twice, only one preview is generated. However, if the second link is the same but with an #anchor, the link preview is generated twice for the same content as, I assume, the homeserver will not consider the anchor at all for the actual preview generation process.

In client this is perhaps a bit tricky case because the homeserver could consider the anchor e.g. by running javascript or extracting some other part from the document for the preview, but as it is, it doesn't. To robustly handle this the client could send a request for both, but remove duplicate responses.

Steps to reproduce

• Using Element Web, send a URL that produces a link preview twice to a room, but add #anchor to the end of the second URL, e.g. https://github.com/matrix-org/synapse/issues/10088 and https://github.com/matrix-org/synapse/issues/10088#foo
• Witness two link preview appearing for the same content

Expected behaviour: only one link preview is generated.

Version information

• Homeserver: matrix.org
• Riot version: 1.7.28