Removes bindOnLocalhost=boolean. Adds bindAddress and advertisedAddress. #26

radekg · 2016-09-20T12:50:55Z

Motivation

The motivation has been documented and discussed in #23.

Modifications

Adds bindAddress and advertisedAddress configuration settings.
Removes bindOnLocalhost configuration setting.

Result

The user can how set the address the service will bind on, for example: 0.0.0.0. The user can also advertise a different address via ZooKeeper. The latter setting makes it much easier to setup Pulsar in environments such as Amazon EC2.

…Address.

yahoocla · 2016-09-20T12:50:55Z

CLA is valid!

radekg · 2016-09-20T12:53:50Z

conf/broker.conf

-# Control whether to bind directly on localhost rather than on normal hostname
-bindOnLocalhost=false
+# Hostname or IP address the service binds on, default is InetAddress.getLocalHost().getHostName().
+bindAddress=


Leaving empty for clarity.

Shouldn't this be 0.0.0.0 instead of hostname by default?

I guess that's a personal preference. Some people like things wide open by default, some do not. I don't mind making it 0.0.0.0.

radekg · 2016-09-20T12:54:03Z

conf/broker.conf

+bindAddress=
+
+# Hostname or IP address the service advertises to the outside world. If not set, the value of bindAddress is used.
+advertisedAddress=


Leaving empty for clarity.

If the bind address was 0.0.0.0, then defaulting to bind address might not work. Should we default to hostname for advertisedAddress ?

Addressed in 3745033

Update the comment with "default to hostname"

radekg · 2016-09-20T12:55:23Z

pulsar-broker/src/test/resources/configurations/pulsar_broker_test.conf

@@ -5,7 +5,8 @@ brokerServicePort=6650
 brokerServicePortTls=6651
 webServicePort=8080
 webServicePortTls=4443
-bindOnLocalhost=false
+bindAddress=


Left empty for clarity.

radekg · 2016-09-20T12:59:44Z

It might be worth adding advertised ports settings for services. 100% in the short term someone is going to ask for it when setting Pulsar in Mesos with Docker.

radekg · 2016-09-20T13:06:59Z

It isn't clear to me how the ServiceConfiguration is populated from file. That might be the only missing link.

merlimat

The change looks good to me. I think the only part missing is to enforce the bindAddress settings by have sockets effectively binding on the desired IP/Host. Before, the bindOnLocalhost was just a trick used for unit tests to advertise localhost instead of the real hostname, but that was not really affecting the "bind" IP address.

For example, https://github.com/yahoo/pulsar/blob/master/pulsar-broker/src/main/java/com/yahoo/pulsar/broker/web/WebService.java#L87 we'd have to add a connector.setHost() to use it.

Same thing with the Netty server socket, at https://github.com/yahoo/pulsar/blob/master/pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/BrokerService.java#L223 we need to specify the bindAddress.

merlimat · 2016-09-20T17:17:01Z

conf/broker.conf

-# Control whether to bind directly on localhost rather than on normal hostname
-bindOnLocalhost=false
+# Hostname or IP address the service binds on, default is InetAddress.getLocalHost().getHostName().
+bindAddress=


Shouldn't this be 0.0.0.0 instead of hostname by default?

merlimat · 2016-09-20T17:17:41Z

conf/broker.conf

+bindAddress=
+
+# Hostname or IP address the service advertises to the outside world. If not set, the value of bindAddress is used.
+advertisedAddress=


If the bind address was 0.0.0.0, then defaulting to bind address might not work. Should we default to hostname for advertisedAddress ?

merlimat · 2016-09-20T17:19:00Z

conf/websocket.conf

+bindAddress=
+
+# Hostname or IP address the service advertises to the outside world. If not set, the value of bindAddress is used.
+advertisedAddress=


Web socket server component doesn't need to advertise itself

merlimat · 2016-09-20T17:28:35Z

pulsar-broker-common/src/main/java/com/yahoo/pulsar/broker/ServiceConfiguration.java

@@ -291,12 +301,32 @@ public void setWebServicePortTls(int webServicePortTls) {
        this.webServicePortTls = webServicePortTls;
    }

-    public boolean isBindOnLocalhost() {
-        return bindOnLocalhost;
+    public String getBindAddress() {


Preferably, the ServiceConfiguration should be a simple POJO. This initialization of the bind address is potentially at risk if multiple threads are initializing. It'd be better to return null (or even better Optional<String>) and have the caller cope with the missing value.

I was trying to avoid throws on this method as the caller would have to require try / catch everywhere where this is used. Unless we can settle on RuntimeException or IllegalArgumentException?

Right, scratch that, that's a silly idea. I'll just re-throw the exception and handle where used.

Addressed in 3745033

@merlimat Why would multiple threads try to initialize a single instance of ServiceConfiguration? Is the service configuration stored somewhere as static?

merlimat · 2016-09-20T17:30:37Z

pulsar-broker-common/src/main/java/com/yahoo/pulsar/broker/ServiceConfiguration.java

+                LOG.debug("bindAddress not set, attempting default configuration.");
+                this.setBindAddress(InetAddress.getLocalHost().getHostName());
+            } catch (UnknownHostException ex) {
+                LOG.warn(ex.getMessage(), "Using localhost as bindAddress.");


We should just re-trow an unchecked exception to have the broker exit if the hostname is invalid. Anyway it will probably not reachable on localhost.

merlimat · 2016-09-20T17:35:21Z

pulsar-broker-common/src/main/java/com/yahoo/pulsar/broker/ServiceConfiguration.java

+        this.bindAddress = bindAddress;
+    }
+
+    public String getAdvertisedAddress() {


Same as above, we should just return what was configured, and in any case, the fallback here should be on hostname rather than the bind address.

merlimat · 2016-09-20T17:36:49Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/PulsarService.java

-            LOG.error(e.getMessage(), e);
-            throw new IllegalStateException("failed to find host", e);
-        }
+        return config.getBindAddress();


These 2 methods should have the above logic to check whether bindAddress and advertisedAddress were configured.

merlimat · 2016-09-20T17:50:59Z

It might be worth adding advertised ports settings for services. 100% in the short term someone is going to ask for it when setting Pulsar in Mesos with Docker.

If you're in the mood, I'm all for it! 😄

It isn't clear to me how the ServiceConfiguration is populated from file. That might be the only missing link.

Ok, that's a bit mysterious, but there's a ServiceConfigurationLoader class that uses reflections to associate the ServiceConfiguration fields to variables in the config file.

@merlimat

…suggested by @merlimat.

radekg · 2016-09-20T21:48:33Z

@merlimat Addressed the comments related to bindAddress / advertisedAddress configuration setup. Web socket and broker now bind on the bindAddress.

merlimat · 2016-09-20T21:50:10Z

@rdhabalia Can you take a look as well?

merlimat · 2016-09-20T21:55:43Z

pulsar-broker-common/src/main/java/com/yahoo/pulsar/broker/ServiceConfiguration.java

+    // If not set, InetAddress.getLocalHost().getHostName() will be used.
+    private String bindAddress;
+    // Controls which hostname is advertised to the discovery service via ZooKeeper.
+    // If not set, bindAddress is used.


I think the comment is out of date now, since it defaults to hostname

merlimat · 2016-09-20T21:56:01Z

conf/standalone.conf

+# Hostname or IP address the service binds on, default is InetAddress.getLocalHost().getHostName().
+bindAddress=
+
+# Hostname or IP address the service advertises to the outside world. If not set, the value of bindAddress is used.


default to hostname

merlimat

Very nice, thank you!

radekg · 2016-09-20T22:18:26Z

@merlimat I'm looking at travis, here: https://travis-ci.org/yahoo/pulsar/builds/161469038, I've been noticing similar situation on my machine:

Running TestSuite
2016-09-20 22:04:02,984 - INFO  - [main:MockedBookKeeperTestCase@65] - >>>>>> starting void com.yahoo.pulsar.broker.service.PersistentMessageFinderTest.testPersistentMessageFinder() throws java.lang.Exception
2016-09-20 22:04:05,505 - INFO  - [main:MockedBookKeeperTestCase@81] - @@@@@@@@@ stopping void com.yahoo.pulsar.broker.service.PersistentMessageFinderTest.testPersistentMessageFinder() throws java.lang.Exception
2016-09-20 22:04:05,543 - INFO  - [main:MockedBookKeeperTestCase@88] - --------- stopped void com.yahoo.pulsar.broker.service.PersistentMessageFinderTest.testPersistentMessageFinder() throws java.lang.Exception
Exception in thread "Thread-7" java.lang.AssertionError: expected [false] but found [true]
    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:494)
    at org.testng.Assert.assertFalse(Assert.java:63)
    at org.testng.Assert.assertFalse(Assert.java:73)
    at com.yahoo.pulsar.broker.service.PersistentTopicTest$5.run(PersistentTopicTest.java:483)
Sep 20, 2016 10:04:09 PM org.glassfish.jersey.server.ApplicationHandler initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
Sep 20, 2016 10:04:10 PM org.glassfish.jersey.server.ApplicationHandler initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
Sep 20, 2016 10:04:11 PM org.glassfish.jersey.server.ApplicationHandler initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
2016-09-20 22:06:18,260 - ERROR - [main:Slf4JLogger@171] - LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 0
Created at:
    io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:241)
    io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:192)
    io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:174)
    com.yahoo.pulsar.client.impl.PulsarClientImpl.<init>(PulsarClientImpl.java:109)
    com.yahoo.pulsar.client.impl.PulsarClientImpl.<init>(PulsarClientImpl.java:82)
    com.yahoo.pulsar.client.api.PulsarClient.create(PulsarClient.java:55)
    com.yahoo.pulsar.broker.service.BacklogQuotaManagerTest.testConcurrentAckAndEviction(BacklogQuotaManagerTest.java:205)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:483)
    org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
    org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
    org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
    org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
    org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
    org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
    org.testng.TestRunner.privateRun(TestRunner.java:767)
    org.testng.TestRunner.run(TestRunner.java:617)
    org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    org.testng.SuiteRunner.run(SuiteRunner.java:240)
    org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    org.testng.TestNG.run(TestNG.java:1057)
    org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:132)
    org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
    org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
    org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:147)
    org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
    org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
    org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

I believe none of my changes are responsible for this, obviously I could be wrong. I do not remember having such earlier today, when I started working on this PR. Do you know what happened in the meantime?

rdhabalia

minor thing. but looks good to me.

rdhabalia · 2016-09-20T22:25:41Z

pulsar-broker/src/test/java/com/yahoo/pulsar/broker/service/AdvertisedAddressTest.java

+    public void setup() throws Exception {
+        bkEnsemble = new LocalBookkeeperEnsemble(3, ZOOKEEPER_PORT, 5001);
+        bkEnsemble.start();
+        ServiceConfiguration config = create(new Properties(System.getProperties()));


as we are not loading ServiceConfiguration from map/properties: it is fine to directly instantiate
ServiceConfiguration config = new ServiceConfiguration()

merlimat · 2016-09-20T23:19:33Z

@radekg I think the test issue might be related to the fact that without specifying the IP at bind time, we were always binding on 0.0.0.0. Now, we're specifying the bind address, by passing the hostname, so 127.0.0.1 will be excluded. There might be few unit tests that weren't passing the bindOnLocalhost=true, that are affected.

radekg · 2016-09-20T23:29:35Z

The default binding is still InetAddress.getLocalHost().getHostName() (https://github.com/yahoo/pulsar/pull/26/files#diff-5ff5807c8da7704d2729f62605e53e88R571) which should by default go to localhost. What I see locally, where there is no 10 minute timeout, is that the test continues just fine but it takes about 20 minutes just in that step.

When running these tests locally, the progress can be seen with lsof -i -n -P | grep TCP | grep java | wc -l. The number changes, the test is much, much slower though. I'll give it a shot after a few hours of sleep.

radekg · 2016-09-26T14:42:02Z

@rdhabalia that does the job. This is now passing the tests. Thank you! Can we review again?
I was also wondering if it isn't worth changing the PulsarService.getHost() to a little bit more descriptive PulsarService.getBindAddress(). What do you think?

radekg · 2016-09-26T14:47:07Z

I believe there is one more change required, here: https://github.com/yahoo/pulsar/blob/master/pulsar-broker/src/main/java/com/yahoo/pulsar/broker/admin/PersistentTopics.java#L906

…dAddress()

radekg · 2016-09-26T15:58:18Z

The failure seems to be one of those intermittent issues. It successfully ran here: https://travis-ci.org/radekg/pulsar/builds/162817512. I guess the build needs to be restarted?

merlimat

Change looks good, thanks for going after all test cases. Just one very minor comment

merlimat · 2016-09-26T17:55:00Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/PulsarService.java

@@ -565,41 +567,56 @@ public MessagingServiceShutdownHook getShutdownService() {
    /**
     * Derive the host
     *
-     * @param isBindOnLocalhost
-     * @return
+     * @return Hostname or IP address the service binds on.
     */
    public static String host(ServiceConfiguration config) {


this host(...) method could probably be removed now, given that we can directly use config.getBindAddress() which defaults to a non-null value.

@merlimat is it okay to change getHost() to getBindAddress()? Also, would you like me to rebase?

Sure getBindAddress() sounds good, but we probably don't even need to have the static method, since we could just call config.getBindAddress() as it is.

For rebasing, I will squash it anyway when merging. I don't expect particular conflicts with current master.

…rService.getHost() to PulsarService.getBindAddress().

merlimat · 2016-09-26T21:48:33Z

Nice work!

radekg · 2016-09-26T21:56:31Z

Thank you!

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Introduced the concept of systemException to differentiate from userException

Add Producer/Consumer/EndToEnd Tests

### Motivation When a broker server is restarted, the Pulsar proxy outputs the following warning: ``` 15:09:01.486 [main-EventThread] INFO o.a.p.z.ZooKeeperChildrenCache - reloadCache called in zookeeperChildrenCache for path /loadbalance/brokers 15:09:31.487 [main-EventThread] WARN o.a.p.p.s.util.ZookeeperCacheLoader - Error updating broker info after broker list changed. java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.updateBrokerList(ZookeeperCacheLoader.java:118) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.lambda$new$0(ZookeeperCacheLoader.java:82) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache.lambda$0(ZooKeeperChildrenCache.java:89) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$22(ZooKeeperCache.java:415) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:592) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` This is due to a deadlock occurring in the proxy. While this deadlock occurs, `main-EventThread` is stopped in the following states: ``` "main-EventThread" #26 daemon prio=5 os_prio=0 tid=0x00007fa9b55ff000 nid=0x3d98 waiting on condition [0x00007fa923dfc000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000eef2ad80> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1709) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1788) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.updateBrokerList(ZookeeperCacheLoader.java:123) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.lambda$new$0(ZookeeperCacheLoader.java:84) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader$$Lambda$40/1450652220.onUpdate(Unknown Source) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache.lambda$0(ZooKeeperChildrenCache.java:89) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache$$Lambda$365/191748085.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$22(ZooKeeperCache.java:415) at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$46/1819738265.processResult(Unknown Source) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:592) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` `ZookeeperCacheLoader` will try to get load information for all available brokers from ZK or its cache if there is a change in the znode `/loadbalance/brokers`. The problem is that it is `main-EventThread` that performs the above process, and it is also `main-EventThread` that gets the data from ZK. `main-EventThread` is blocked at the following line, so deadlock occurs. https://github.com/apache/pulsar/blob/9d3ab60efddd3f4064789293fd8e9c4d4388b6c7/pulsar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/ZooKeeperDataCache.java#L97 ### Modifications ~Make the method `ZookeeperCacheLoader#updateBrokerList` run on a thread separate from `main-EventThread`.~ Made `updateBrokerList()` method asynchronous so that `main-EventThread` is not blocked.

### Motivation When a broker server is restarted, the Pulsar proxy outputs the following warning: ``` 15:09:01.486 [main-EventThread] INFO o.a.p.z.ZooKeeperChildrenCache - reloadCache called in zookeeperChildrenCache for path /loadbalance/brokers 15:09:31.487 [main-EventThread] WARN o.a.p.p.s.util.ZookeeperCacheLoader - Error updating broker info after broker list changed. java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.updateBrokerList(ZookeeperCacheLoader.java:118) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.lambda$new$0(ZookeeperCacheLoader.java:82) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache.lambda$0(ZooKeeperChildrenCache.java:89) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$22(ZooKeeperCache.java:415) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:592) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` This is due to a deadlock occurring in the proxy. While this deadlock occurs, `main-EventThread` is stopped in the following states: ``` "main-EventThread" apache#26 daemon prio=5 os_prio=0 tid=0x00007fa9b55ff000 nid=0x3d98 waiting on condition [0x00007fa923dfc000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000eef2ad80> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1709) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1788) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.updateBrokerList(ZookeeperCacheLoader.java:123) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.lambda$new$0(ZookeeperCacheLoader.java:84) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader$$Lambda$40/1450652220.onUpdate(Unknown Source) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache.lambda$0(ZooKeeperChildrenCache.java:89) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache$$Lambda$365/191748085.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$22(ZooKeeperCache.java:415) at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$46/1819738265.processResult(Unknown Source) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:592) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` `ZookeeperCacheLoader` will try to get load information for all available brokers from ZK or its cache if there is a change in the znode `/loadbalance/brokers`. The problem is that it is `main-EventThread` that performs the above process, and it is also `main-EventThread` that gets the data from ZK. `main-EventThread` is blocked at the following line, so deadlock occurs. https://github.com/apache/pulsar/blob/9d3ab60efddd3f4064789293fd8e9c4d4388b6c7/pulsar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/ZooKeeperDataCache.java#L97 ### Modifications ~Make the method `ZookeeperCacheLoader#updateBrokerList` run on a thread separate from `main-EventThread`.~ Made `updateBrokerList()` method asynchronous so that `main-EventThread` is not blocked.

### Motivation When a broker server is restarted, the Pulsar proxy outputs the following warning: ``` 15:09:01.486 [main-EventThread] INFO o.a.p.z.ZooKeeperChildrenCache - reloadCache called in zookeeperChildrenCache for path /loadbalance/brokers 15:09:31.487 [main-EventThread] WARN o.a.p.p.s.util.ZookeeperCacheLoader - Error updating broker info after broker list changed. java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.updateBrokerList(ZookeeperCacheLoader.java:118) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.lambda$new$0(ZookeeperCacheLoader.java:82) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache.lambda$0(ZooKeeperChildrenCache.java:89) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$22(ZooKeeperCache.java:415) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:592) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` This is due to a deadlock occurring in the proxy. While this deadlock occurs, `main-EventThread` is stopped in the following states: ``` "main-EventThread" #26 daemon prio=5 os_prio=0 tid=0x00007fa9b55ff000 nid=0x3d98 waiting on condition [0x00007fa923dfc000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000eef2ad80> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1709) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1788) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.updateBrokerList(ZookeeperCacheLoader.java:123) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.lambda$new$0(ZookeeperCacheLoader.java:84) at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader$$Lambda$40/1450652220.onUpdate(Unknown Source) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache.lambda$0(ZooKeeperChildrenCache.java:89) at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache$$Lambda$365/191748085.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$22(ZooKeeperCache.java:415) at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$46/1819738265.processResult(Unknown Source) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:592) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` `ZookeeperCacheLoader` will try to get load information for all available brokers from ZK or its cache if there is a change in the znode `/loadbalance/brokers`. The problem is that it is `main-EventThread` that performs the above process, and it is also `main-EventThread` that gets the data from ZK. `main-EventThread` is blocked at the following line, so deadlock occurs. https://github.com/apache/pulsar/blob/9d3ab60efddd3f4064789293fd8e9c4d4388b6c7/pulsar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/ZooKeeperDataCache.java#L97 ### Modifications ~Make the method `ZookeeperCacheLoader#updateBrokerList` run on a thread separate from `main-EventThread`.~ Made `updateBrokerList()` method asynchronous so that `main-EventThread` is not blocked. (cherry picked from commit d7c1cfa)

1. Add kafka-examples to easy verify broker. 2. update readme.

update log4j dependency to 2.17.0 Fixes apache#26 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve.* ### Modifications *Describe the modifications you've done.* ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the chan

…plication (#26) * [feat][broker] Add ResourceGroup rate limit for topic dispatch and replication Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][broker] Fix ownership loss Signed-off-by: Zixuan Liu <nodeces@gmail.com> * Fix test Signed-off-by: Zixuan Liu <nodeces@gmail.com> --------- Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Changes bindOnLocalhost=boolean to bindAddress. Introduces advertised…

c7e3a7a

…Address.

radekg commented Sep 20, 2016

View reviewed changes

radekg changed the title ~~Changes bindOnLocalhost=boolean to bindAddress. Introduces advertised…~~ Removes bindOnLocalhost=boolean. Adds bindAddress and advertisedAddress Sep 20, 2016

radekg changed the title ~~Removes bindOnLocalhost=boolean. Adds bindAddress and advertisedAddress~~ Removes bindOnLocalhost=boolean. Adds bindAddress and advertisedAddress. Sep 20, 2016

merlimat reviewed Sep 20, 2016

View reviewed changes

merlimat added this to the 1.15 milestone Sep 20, 2016

merlimat assigned radekg Sep 20, 2016

merlimat added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Sep 20, 2016

radekg added 4 commits September 20, 2016 19:57

Removes advertisement from web socket config.

cc08ed3

ENsure services bind on the bind address.

6fa126f

Change bind address and advertised address hostname resolution to as …

3745033

…suggested by @merlimat.

Remove unnecessary LOG in ServiceConfiguration.

393e10e

Removed unused imports.

3ed746e

merlimat reviewed Sep 20, 2016

View reviewed changes

Corrections in comments.

1bec010

merlimat approved these changes Sep 20, 2016

View reviewed changes

rdhabalia reviewed Sep 20, 2016

View reviewed changes

ServiceConfigurration in tests not loaded from Properties.

bf27260

Last place where the getHost() needs to be replaced with getAdvertise…

1ac44bf

…dAddress()

merlimat reviewed Sep 26, 2016

View reviewed changes

Address the final review point, adjust the metrics test, rename Pulsa…

d4c76f5

…rService.getHost() to PulsarService.getBindAddress().

merlimat merged commit 8df6b74 into apache:master Sep 26, 2016

massakam pushed a commit to massakam/pulsar that referenced this pull request Aug 6, 2020

Merge pull request apache#26 from k2la/add_test_code

4398c89

Add Producer/Consumer/EndToEnd Tests

LutzWeischerFujitsu mentioned this pull request Mar 17, 2021

test fails on AArch64, Fedora 33 #9936

Closed

hangc0276 pushed a commit to hangc0276/pulsar that referenced this pull request May 26, 2021

add kafka-client-examples, update readme (apache#26)

4dab945

1. Add kafka-examples to easy verify broker. 2. update readme.

xiaotongwang1 mentioned this pull request Aug 4, 2021

Pulsar 2.7.0+ KOP 2.7.2.x getPartitionedTopicMetadata timeout #11532

Closed

paladin-dalao mentioned this pull request Jan 13, 2022

cpp client coredump #13735

Closed

dragonls pushed a commit to dragonls/pulsar that referenced this pull request Oct 21, 2022

Optimize Dockerfile. (apache#26)

cb88b78

dragonls pushed a commit to dragonls/pulsar that referenced this pull request Oct 21, 2022

Optimize Dockerfile. (apache#26)

e70e649

dragonls pushed a commit to dragonls/pulsar that referenced this pull request Oct 21, 2022

Optimize image and zk-shell. (apache#26)

b30c97d

ZHr-UChiHa mentioned this pull request Jun 3, 2023

[Bug] python pulsar client create producer thread suspend because of deadlock apache/pulsar-client-python#129

Open

2 tasks

Removes bindOnLocalhost=boolean. Adds bindAddress and advertisedAddress. #26

Removes bindOnLocalhost=boolean. Adds bindAddress and advertisedAddress. #26

Conversation

radekg commented Sep 20, 2016

Motivation

Modifications

Result

yahoocla commented Sep 20, 2016

Choose a reason for hiding this comment

merlimat Sep 20, 2016 • edited Loading

Choose a reason for hiding this comment

radekg Sep 20, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radekg commented Sep 20, 2016

radekg commented Sep 20, 2016

merlimat left a comment

Choose a reason for hiding this comment

merlimat Sep 20, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radekg Sep 21, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlimat commented Sep 20, 2016

radekg commented Sep 20, 2016

merlimat commented Sep 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlimat left a comment

Choose a reason for hiding this comment

radekg commented Sep 20, 2016 • edited Loading

rdhabalia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlimat commented Sep 20, 2016

radekg commented Sep 20, 2016 • edited Loading

radekg commented Sep 26, 2016 • edited Loading

radekg commented Sep 26, 2016

radekg commented Sep 26, 2016

merlimat left a comment

Choose a reason for hiding this comment

merlimat Sep 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlimat commented Sep 26, 2016

radekg commented Sep 26, 2016

merlimat Sep 20, 2016 •

edited

Loading

radekg Sep 20, 2016 •

edited

Loading

merlimat Sep 20, 2016 •

edited

Loading

radekg Sep 21, 2016 •

edited

Loading

radekg commented Sep 20, 2016 •

edited

Loading

radekg commented Sep 20, 2016 •

edited

Loading

radekg commented Sep 26, 2016 •

edited

Loading

merlimat Sep 26, 2016 •

edited

Loading