Exception from 'rotate_notifs' shown when running the synapse_port_db script

[blabug]

Description

When migrating an ~5Gb SQLite3 Database to PostgreSQL as described in the documentation, the script printed an exception repeatedly from the rotate_notifs background process.

After about 10 minutes the script started to output database migration progress and I didn't see this exception again.

Steps to reproduce

• Set up the Postgres database as described in the docs
• systemctl stop matrix-synapse
• Added the Postgres configuration section (copied from the docs) in a new file /etc/matrix-synapse/conf.d/postgres.yaml
• Concatenated homeserver.yaml and all files in the conf.d directory into a single configuration for the migration script
• Ran synapse_port_db --sqlite-database /var/lib/matrix-synapse/homeserver.db --postgres-config concatenated_config.yaml
• After some log messages I've got the exception message printed. As it repeated, I thought that the script was stuck on the exception and killed it and tried again with an updated system. But this also created the same exception. While the migration worked after waiting some time, I don't expect the script to throw any exceptions when migrating (or don't show them, if they are ok).

Homeserver

Self hosted homeserver

Synapse Version

1.81.0+bullseye1 and 1.85.2+bookworm1 after server upgrade

Installation Method

Debian packages from packages.matrix.org

Database

SQLite3 3.40.1-2

Workers

I don't know

Platform

Debian Bullseye and Debian Bookworm on a VM in a Proxmox Cluster

Configuration

No response

Relevant log output

synapse.metrics.background_process_metrics - 244 - ERROR - Background proces
s 'rotate_notifs' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/metrics/background_process_metr
ics.py", line 242, in run
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/event_pu
sh_actions.py", line 1297, in _rotate_notifs
    caught_up = await self.db_pool.runInteraction(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 924,
 in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, 
in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/failure.py", line 518, i
n throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 890,
 in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 1019
, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 1012, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 752, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1332, in _handle_new_receipts_for_notifs_txn
    min_receipts_stream_id = self.db_pool.simple_select_one_onecol_txn(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 1741, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found

Anything else that would be useful to know?

Maybe the issue is caused by the large DB size of ~5Gb and while the DB is loaded the exceptions are thrown due to missing data (just my assumption).

[erikjohnston]

Hmm, this looks like its trying to run the rotate_notifs background process, which shouldn't be run during the port DB script.