Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to get messages from non-persistent topics #2009

Closed
nikita-tomilov opened this issue Jun 21, 2018 · 3 comments
Closed

Unable to get messages from non-persistent topics #2009

nikita-tomilov opened this issue Jun 21, 2018 · 3 comments
Labels
area/client type/bug The PR fixed a bug or issue reported a bug

Comments

@nikita-tomilov
Copy link

Issue

I am unable to get messages from non-persistent topics while using wildcard subscription.

Publisher:

PulsarClient client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();
Producer<String> stringProducer = client.newProducer(Schema.STRING)
        .topic("non-persistent://public/default/test1")
        .create();
stringProducer.send(line);

Subscriber:

 client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();
consumer = client.newConsumer()
        .topicsPattern(Pattern.compile("non-persistent://public/default/test[12]"))
        .subscriptionName(new Date().toString())
        .subscribe();
Message msg = consumer.receive();

Expected behavior

I receive the message.

Actual behavior

No messages can be received. While debugging, we noticed that the topic list that client gets from the server by specified regexp is empty.

Server says that the message is actually sent.

After changing non-persistent to persistent, everything works just fine.

While trying to investigate this, we have changed the pulsar:// prefix to http:// and inspected http requests. Client tried to perform request GET /admin/v2/namespaces/public/default/destinations, but failed, and in the server logs there was a TooLongFrameException : Adjusted frame length exceeds 5242880: 1195725860 - discarded. From the Wireshark dump, the request's frame length is totally fine.

Steps to reproduce

  1. Run Pulsar server
  2. Run consumer
  3. Run producer
  4. Try to send the message from producer
  5. See that the message is not received

System configuration

Pulsar version: 2.0.1-incubating

@nikita-tomilov nikita-tomilov changed the title Unable to get messages from non-persistent topics while using wildcard subscription Unable to get messages from non-persistent topics Jun 21, 2018
@sijie
Copy link
Member

sijie commented Jun 21, 2018

@jiazhai can you help check if regex subscription works for non-persistent?

@sijie
Copy link
Member

sijie commented Sep 19, 2018

This is being fixed by #2025, will be included in the upcoming release.

sijie pushed a commit that referenced this issue Sep 25, 2018
… from the "GetTopicsOfNamespace" method (#2025)

### Motivation

Please see issue #2009 for a detailed bug report.

In our use case we require using java client with pattern subscription to read from a set of non-persistent topics. Unfortunately, right now this feature doesn't work. After researching the cause for this I have found out that under the hood the client is requesting a list of topics by namespace from the server and then filters them out by pattern and subscribes to them. The method in Pulsar broker NamespaceService class that is responsible for searching for required topics only uses ledgers, thus returning only persistent topics to the client. The goal of this pull request is to provide a solution for that problem.

### Modifications

This pull request updates `getListOfTopics` method of NamespaceService class to also include active non-pesistent topics from local broker cache inside the `multiLayerTopicsMap` collection of BrokerService in the result.

### Result

As a result, requesting a list of topics by namespace using the HTTP API or binary API (and thus via the clients) will add non-persistent topics to search result, allowing pattern subscription to be used with non-persistent topics.

### Considerations

1. Since this method pulls non-persistent topics from local broker cache, this probably means that this solution will only work for Pulsar installations with a single broker. And if there are multiple brokers, results might be inconsistent. Unfortunately I don't really know if non-persistent themselves work in multi-broker setups. I have recently asked on Slack if non-persistent topics are being replicated in any way and @merlimat's response was that they don't. Also it seems to be that some other methods that are working with non-persistent topics are using this very same collection.

2. It seems to me that unit tests have made sure that Java client can work with this setup, but this might still be a breaking change for other clients or if applications working with this API are not expecting non-persistent topics in result.

3. I have made sure that old unit tests inside the `pulsar-broker` subproject are still working and updated some old tests for this particular use case. Are there any more tests that I can add.

Overall, we really need this and I would appreciate if maintainers could share their opinion. Thanks in advance.
@sijie
Copy link
Member

sijie commented Jan 25, 2019

this has been addressed by #2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/client type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

No branches or pull requests

2 participants