-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to get messages from non-persistent topics #2009
Labels
Comments
nikita-tomilov
changed the title
Unable to get messages from non-persistent topics while using wildcard subscription
Unable to get messages from non-persistent topics
Jun 21, 2018
@jiazhai can you help check if regex subscription works for non-persistent? |
This is being fixed by #2025, will be included in the upcoming release. |
sijie
pushed a commit
that referenced
this issue
Sep 25, 2018
… from the "GetTopicsOfNamespace" method (#2025) ### Motivation Please see issue #2009 for a detailed bug report. In our use case we require using java client with pattern subscription to read from a set of non-persistent topics. Unfortunately, right now this feature doesn't work. After researching the cause for this I have found out that under the hood the client is requesting a list of topics by namespace from the server and then filters them out by pattern and subscribes to them. The method in Pulsar broker NamespaceService class that is responsible for searching for required topics only uses ledgers, thus returning only persistent topics to the client. The goal of this pull request is to provide a solution for that problem. ### Modifications This pull request updates `getListOfTopics` method of NamespaceService class to also include active non-pesistent topics from local broker cache inside the `multiLayerTopicsMap` collection of BrokerService in the result. ### Result As a result, requesting a list of topics by namespace using the HTTP API or binary API (and thus via the clients) will add non-persistent topics to search result, allowing pattern subscription to be used with non-persistent topics. ### Considerations 1. Since this method pulls non-persistent topics from local broker cache, this probably means that this solution will only work for Pulsar installations with a single broker. And if there are multiple brokers, results might be inconsistent. Unfortunately I don't really know if non-persistent themselves work in multi-broker setups. I have recently asked on Slack if non-persistent topics are being replicated in any way and @merlimat's response was that they don't. Also it seems to be that some other methods that are working with non-persistent topics are using this very same collection. 2. It seems to me that unit tests have made sure that Java client can work with this setup, but this might still be a breaking change for other clients or if applications working with this API are not expecting non-persistent topics in result. 3. I have made sure that old unit tests inside the `pulsar-broker` subproject are still working and updated some old tests for this particular use case. Are there any more tests that I can add. Overall, we really need this and I would appreciate if maintainers could share their opinion. Thanks in advance.
this has been addressed by #2025 |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue
I am unable to get messages from non-persistent topics while using wildcard subscription.
Publisher:
Subscriber:
Expected behavior
I receive the message.
Actual behavior
No messages can be received. While debugging, we noticed that the topic list that client gets from the server by specified regexp is empty.
Server says that the message is actually sent.
After changing
non-persistent
topersistent
, everything works just fine.While trying to investigate this, we have changed the
pulsar://
prefix tohttp://
and inspected http requests. Client tried to perform requestGET /admin/v2/namespaces/public/default/destinations
, but failed, and in the server logs there was aTooLongFrameException : Adjusted frame length exceeds 5242880: 1195725860 - discarded
. From the Wireshark dump, the request's frame length is totally fine.Steps to reproduce
System configuration
Pulsar version:
2.0.1-incubating
The text was updated successfully, but these errors were encountered: