Fresh install error: AssertionError sentry.eventstream.kafka.backend in _build_consumer

[ssjunior]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Version

Sentry 22.10.0.dev0

Steps to Reproduce

Do a fresh install using debian bulls-eye.

git clone https://github.com/getsentry/self-hosted.git sentry
cd sentry
./install.sh

go to webpage at port 9000

Expected Result

No issues reported after fresh install

Actual Result

Installed 3 times, same result. After a few seconds an issue is reported:

The issues continue to be reported growing the number of incidents.

[emmatyping]

Hm, sorry you are running into this issue. Are there any more details if you click through to the issue? I cannot reproduce this. You might also want to check if there is any hints in the docker compose logs.

[ssjunior]

The rest of traceback:

 Traceback (most recent call last):
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/bin/sentry", line 8, in <module>
sentry-self-hosted-post-process-forwarder-1                    |     sys.exit(main())
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 188, in main
sentry-self-hosted-post-process-forwarder-1                    |     func(**kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
sentry-self-hosted-post-process-forwarder-1                    |     return self.main(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
sentry-self-hosted-post-process-forwarder-1                    |     rv = self.invoke(ctx)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return ctx.invoke(self.callback, **ctx.params)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-post-process-forwarder-1                    |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner
sentry-self-hosted-post-process-forwarder-1                    |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-post-process-forwarder-1                    |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
sentry-self-hosted-post-process-forwarder-1                    |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 363, in post_process_forwarder
sentry-self-hosted-post-process-forwarder-1                    |     eventstream.run_post_process_forwarder(
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 127, in <lambda>
sentry-self-hosted-post-process-forwarder-1                    |     context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 288, in run_post_process_forwarder
sentry-self-hosted-post-process-forwarder-1                    |     self.run_batched_consumer(
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 254, in run_batched_consumer
sentry-self-hosted-post-process-forwarder-1                    |     consumer = self._build_consumer(
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 218, in _build_consumer
sentry-self-hosted-post-process-forwarder-1                    |     assert self.topic == self.transactions_topic
sentry-self-hosted-post-process-forwarder-1                    | AssertionError

[lbogdan]

Getting this too, with latest master, on a minimal Ubuntu Server 22.04 install.

Weird thing is that I don't even see that error in Sentry, under the internal project. Do I have to explicitly configure something for that to happen? I created other projects, and I can see their errors.

[lbogdan]

Did a bit of troubleshooting, added a debug log

logger.info(f"topic={self.topic} transactions_topic={self.transactions_topic}")

before the assert

# /usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py:218
assert self.topic == self.transactions_topic

and got

10:14:52 [INFO] sentry.eventstream.kafka.backend: Starting post process forwarder to consume all messages
10:14:52 [INFO] sentry.eventstream.kafka.backend: topic=events transactions_topic=transactions

There's a comment above that says:

# Default implementation which processes both errors and transactions
# irrespective of values in the header. This would most likely be the case
# for development environments. For the combined post process forwarder
# to work KAFKA_EVENTS and KAFKA_TRANSACTIONS must be the same currently.

Not sure what that means, but maybe it helps. 🙂

[long2ice]

Same here

[hubertdeng123]

Seems like this may be a regression of some sort from the kafka backend on sentry, not an issue introduced recently with self-hosted. I'll be transferring this to getsentry/sentry for further investigation

[getsentry-release]

Routing to @getsentry/sns for triage. ⏲️

[hubertdeng123]

Seems like the file in question, sentry/eventstream/kafka/backend.py has been changed in the past week if that's helpful 🤔

[lynnagara]

Errors and transactions pipelines are fully separate now, previously they shared infra. We would need to have two separate post process forwarders, as running both in a single container is no longer supported.

Errors one would need to be like:
run post-process-forwarder --commit-batch-size 1 --entity errors

Transactions one would be like:
run post-process-forwarder --commit-batch-size 1 --entity transactions --commit-log-topic=snuba-transactions-commit-log --synchronize-commit-group transactions_group

[hubertdeng123]

So I'm guessing this is how the docker-compose.yml file would look like

post-process-forwarder-errors:
    <<: *sentry_defaults
    # Increase `--commit-batch-size 1` below to deal with high-load environments.
    command: run post-process-forwarder --commit-batch-size 1 --entity errors

  post-process-forwarder-transactions:
    <<: *sentry_defaults
    command: run post-process-forwarder --commit-batch-size 1 --entity transactions --commit-log-topic=snuba- transactions-commit-log --synchronize-commit-group transactions_group

[chadwhitacre]

Ftr here is where we flipped the switch last week:

• feat: Remove default mapping of "transaction" topic to "events" snuba#3250
• feat: Switch default transactions topic #39988

We're working through an attempted fix in getsentry/self-hosted#1759.

[hubertdeng123]

Able to repro on master:

[chadwhitacre]

Should be fixed in getsentry/self-hosted#1759.