Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Handle pusher being deleted during processing. #5809

Merged
merged 4 commits into from
Aug 13, 2019

Conversation

erikjohnston
Copy link
Member

Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.

Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
@erikjohnston erikjohnston requested a review from a team August 1, 2019 12:45
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you record the text of the exception for future reference?

if not pusher_still_exists:
# The pusher has been deleted while we were processing, so
# lets just stop and return.
self.on_stop()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this done when the pusher is deleted?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I mostly put this in for paranoia to make sure we don't go round the loop again.

synapse/storage/pusher.py Outdated Show resolved Hide resolved
@erikjohnston
Copy link
Member Author

Example stack trace:

Exception processing notifs
StoreError: 404: No row found (pushers)
  File "synapse/push/httppusher.py", line 171, in _process
    yield self._unsafe_process()
  File "twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/push/httppusher.py", line 216, in _unsafe_process
    self.app_id, self.pushkey, self.user_id, self.failing_since
  File "twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/storage/pusher.py", line 327, in update_pusher_failing_since
    desc="update_pusher_failing_since",
  File "twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/storage/_base.py", line 506, in runInteraction
    **kwargs
  File "twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/storage/_base.py", line 554, in runWithConnection
    result = yield self._db_pool.runWithConnection(inner_func, *args, **kwargs)
  File "twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "twisted/enterprise/adbapi.py", line 306, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "twisted/python/compat.py", line 464, in reraise
    raise exception.with_traceback(traceback)
  File "twisted/enterprise/adbapi.py", line 297, in _runWithConnection
    result = func(conn, *args, **kw)
  File "synapse/storage/_base.py", line 551, in inner_func
    return func(conn, *args, **kwargs)
  File "synapse/storage/_base.py", line 423, in _new_transaction
    r = func(txn, *args, **kwargs)
  File "synapse/storage/_base.py", line 1233, in _simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

# The pusher has been deleted while we were processing, so
# lets just stop and return.
self.on_stop()
return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is redundant...

@erikjohnston erikjohnston merged commit af18780 into develop Aug 13, 2019
babolivier added a commit that referenced this pull request Aug 15, 2019
Synapse 1.3.0 (2019-08-15)
==========================

Bugfixes
--------

- Fix 500 Internal Server Error on `publicRooms` when the public room list was
  cached. ([\#5851](#5851))

Synapse 1.3.0rc1 (2019-08-13)
==========================

Features
--------

- Use `M_USER_DEACTIVATED` instead of `M_UNKNOWN` for errcode when a deactivated user attempts to login. ([\#5686](#5686))
- Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. ([\#5732](#5732))
- Synapse will no longer serve any media repo admin endpoints when `enable_media_repo` is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. ([\#5754](#5754), [\#5848](#5848))
- Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. ([\#5783](#5783))
- Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. ([\#5807](#5807))

Bugfixes
--------

- Fix UISIs during homeserver outage. ([\#5693](#5693), [\#5789](#5789))
- Fix stack overflow in server key lookup code. ([\#5724](#5724))
- start.sh no longer uses deprecated cli option. ([\#5725](#5725))
- Log when we receive an event receipt from an unexpected origin. ([\#5743](#5743))
- Fix debian packaging scripts to correctly build sid packages. ([\#5775](#5775))
- Correctly handle redactions of redactions. ([\#5788](#5788))
- Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. ([\#5798](#5798))
- Fix check that tombstone is a state event in push rules. ([\#5804](#5804))
- Fix error when trying to login as a deactivated user when using a worker to handle login. ([\#5806](#5806))
- Fix bug where user `/sync` stream could get wedged in rare circumstances. ([\#5825](#5825))
- The purge_remote_media.sh script was fixed. ([\#5839](#5839))

Deprecations and Removals
-------------------------

- Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration. ([\#5678](#5678), [\#5729](#5729))
- Remove non-functional 'expire_access_token' setting. ([\#5782](#5782))

Internal Changes
----------------

- Make Jaeger fully configurable. ([\#5694](#5694))
- Add precautionary measures to prevent future abuse of `window.opener` in default welcome page. ([\#5695](#5695))
- Reduce database IO usage by optimising queries for current membership. ([\#5706](#5706), [\#5738](#5738), [\#5746](#5746), [\#5752](#5752), [\#5770](#5770), [\#5774](#5774), [\#5792](#5792), [\#5793](#5793))
- Improve caching when fetching `get_filtered_current_state_ids`. ([\#5713](#5713))
- Don't accept opentracing data from clients. ([\#5715](#5715))
- Speed up PostgreSQL unit tests in CI. ([\#5717](#5717))
- Update the coding style document. ([\#5719](#5719))
- Improve database query performance when recording retry intervals for remote hosts. ([\#5720](#5720))
- Add a set of opentracing utils. ([\#5722](#5722))
- Cache result of get_version_string to reduce overhead of `/version` federation requests. ([\#5730](#5730))
- Return 'user_type' in admin API user endpoints results. ([\#5731](#5731))
- Don't package the sytest test blacklist file. ([\#5733](#5733))
- Replace uses of returnValue with plain return, as returnValue is not needed on Python 3. ([\#5736](#5736))
- Blacklist some flakey tests in worker mode. ([\#5740](#5740))
- Fix some error cases in the caching layer. ([\#5749](#5749))
- Add a prometheus metric for pending cache lookups. ([\#5750](#5750))
- Stop trying to fetch events with event_id=None. ([\#5753](#5753))
- Convert RedactionTestCase to modern test style. ([\#5768](#5768))
- Allow looping calls to be given arguments. ([\#5780](#5780))
- Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO. ([\#5785](#5785))
- Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests. ([\#5787](#5787))
- Remove some spurious exceptions from the logs where we failed to talk to a remote server. ([\#5790](#5790))
- Improve performance when making `.well-known` requests by sharing the SSL options between requests. ([\#5794](#5794))
- Disable codecov GitHub comments on PRs. ([\#5796](#5796))
- Don't allow clients to send tombstone events that reference the room it's sent in. ([\#5801](#5801))
- Deny redactions of events sent in a different room. ([\#5802](#5802))
- Deny sending well known state types as non-state events. ([\#5805](#5805))
- Handle incorrectly encoded query params correctly by returning a 400. ([\#5808](#5808))
- Handle pusher being deleted during processing rather than logging an exception. ([\#5809](#5809))
- Return 502 not 500 when failing to reach any remote server. ([\#5810](#5810))
- Reduce global pauses in the events stream caused by expensive state resolution during persistence. ([\#5826](#5826))
- Add a lower bound to well-known lookup cache time to avoid repeated lookups. ([\#5836](#5836))
- Whitelist history visbility sytests in worker mode tests. ([\#5843](#5843))
@erikjohnston erikjohnston deleted the erikj/handle_pusher_stop branch January 9, 2020 15:48
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants