Improve reference metadata handling for EventSource #4934

Job	Run time
docs	8m 54s
lint	19s
tests (Linux (3.10, mamba), ubuntu-latest, 3.10, mamba, tests,all)	11m 29s
tests (Linux (3.10, pip), ubuntu-latest, 3.10, pip, tests,all)	12m 52s
tests (Linux (3.11, pip, minimal-dependencies), ubuntu-latest, 3.11, pip, tests)	11m 40s
tests (Linux (3.12, mamba, coverage), ubuntu-latest, 3.12, mamba, codecov, tests,all)	12m 42s
tests (Linux (3.12, pip), ubuntu-latest, 3.12, pip, tests,all)	16m 11s
tests (macos (3.10, x86_64, pip), macos-13, 3.10, pip, tests,all)	24m 30s
tests (macos (3.12, arm64, mamba), macos-14, 3.12, mamba, tests,all)	10m 48s
	1h 49m 25s

Provide feedback