Advertised broker address

[radekg]

Expected behavior

Connecting through the discovery to the broker in the cluster setup should redirect to the public address of the broker. Preferably there should be an option to set an advertised hostname, this should come together with an option to set a bind host for the broker (however, this might not be necessary).

Actual behavior

Broker advertises the hostname as the address for the client to connect to. On EC2, it results in the client trying to connect to something like ip-xx-xxx-xxx-xxx which is not resolvable from outside of the same region. The error I see on my local machine while trying to publish messages to Pulsar is the following:

2016-09-19 17:45:38,592 - ERROR - [main:CmdProduce@186] - java.net.UnknownHostException: ip-10-239-182-4: unknown error
com.yahoo.pulsar.client.api.PulsarClientException: java.net.UnknownHostException: ip-10-239-182-4: unknown error
    at com.yahoo.pulsar.client.impl.HttpClient$1.onThrowable(HttpClient.java:139)
    at com.ning.http.client.providers.netty.future.NettyResponseFuture.abort(NettyResponseFuture.java:238)
    at com.ning.http.client.providers.netty.request.NettyRequestSender.abort(NettyRequestSender.java:422)
    at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:290)
    at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142)
    at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:117)
    at com.ning.http.client.providers.netty.request.NettyRequestSender.sendNextRequest(NettyRequestSender.java:493)
    at com.ning.http.client.providers.netty.handler.Protocol.exitAfterHandlingRedirect(Protocol.java:189)
    at com.ning.http.client.providers.netty.handler.HttpProtocol.handleHttpResponse(HttpProtocol.java:427)
    at com.ning.http.client.providers.netty.handler.HttpProtocol.handle(HttpProtocol.java:470)
    at com.ning.http.client.providers.netty.handler.Processor.messageReceived(Processor.java:88)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: ip-10-239-182-4: unknown error
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at java.net.InetAddress.getByName(InetAddress.java:1076)
    at com.ning.http.client.NameResolver$JdkNameResolver.resolve(NameResolver.java:28)
    at com.ning.http.client.providers.netty.request.NettyRequestSender.remoteAddress(NettyRequestSender.java:358)
    at com.ning.http.client.providers.netty.request.NettyRequestSender.connect(NettyRequestSender.java:369)
    at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:283)
    ... 37 more

I could set the hostname on the machine to a public IP address / public hostname but it is an ugly workaround. I do not have any DNS other than EC2 defaults available right now.

Steps to reproduce

Launch the cluster setup in EC2 (without any DNS other than EC2 defaults) and try producing messages.

System configuration

Pulsar version: 1.14

If the suggestions sound okay, I can prepare a PR.

[merlimat]

I'm not a big fan of setting the hostname to bind on the config file, since it would require to have a different config file on each host.

What about a setting to advertise the broker address with just the IP ?

Eg: something like :

# Control whether the broker should advertise itself by using the hostname
# or its own IP address. 
advertiseBrokerWithHostname=false

[radekg]

This will not work on EC2, the public IP address is not available via ifconfig. Broker would always advertise private 10.x.x.x address. What about doing it similar to how Kafka does it?

# If set, broker will advertise to discovery using this address;
# if empty, broker will advertise using the hostname, unless bindOnLocalhost
advertiseBrokerAddress=<ip address or hostname>

Yet to figure out this will work if the broker binds on the hostname. It might be a good idea to allow people set the bind address / host. For example:

# to bind on all interfaces, set to 0.0.0.0
brokerBindAddress=<ip or hostname>

What do you think?

[merlimat]

Oh, I see. Then your proposal sounds good.

[radekg]

Shall I target for brokerBindAddress and advertiseBrokerAddress? This might deprecate the bindOnLocalhost as the user could do brokerBindAddress=localhost # or 127.0.0.1.

[merlimat]

Yes, having both gives the most flexibility, and please go ahead in removing bindOnLocalhost.

[radekg]

@merlimat Please see attached PR. I went for brokerAddress and advertisedAddress as these settings relate to broker.conf file.