advertise address != bind address

[ChristianKniep]

Hey there,

I used etcd/skydns and loosely couple check scripts before and I love consul just for making my live easier!
Currently I am consolidating our AWS infrastructure and I got a docker server instance which runs the consul server.

Ontop of that a handfull of containers provide the services I desire (influxdb, nginx, elasticsearch, ...).

If I expose the service using consul the advertised adress is equal to the bind address, which is not accessable from other instances.
I decided to use the DOCKER_HOST adress to advertise the services, but unfortunately if I do so the clients fighting for this IP address to check their serfHealth as well.

    "bind_addr": "DOCKER_CONTAINER",
    "advertise_addr": "DOCKER_HOST",

Is it possible to overcome this issue? I learned that each agent is eager to talk to all other agents.
So maybe it's not that easy to hide containers behind a common IP address. Maybe I have to use weaver to get real IPs for all of them. :(

Or could it be solved by using multiple data centers? One global and one per docker server domain?

Any clues on how to architect this in a smart way?

[sielaq]

Why do You need consul in every container on one same DOCKER_HOST ?
What you need is one consul per docker host - much easier and comfortable.

[ChristianKniep]

Ahh... I thing I gotcha. On second thought I was thinking about such a construct.

But on third thought you are saying I should kick out the consul agents on each docker container, but only use one consul agent on the bare-metal that deals with all containers as well, correct?

con_srv01 will advertise and check all services that are provided by 'his' docker container friends?

[sielaq]

correct, even better now you can use registrator (https://github.com/progrium/registrator) so all services gonna be register in consul automaticaly (just if you need this funtionallity)

[ChristianKniep]

Dude, you are kidding me! :) I love this stuff. What exiting time we are living in...
I have to digest it over night, but it looks like gold.

[sielaq]

actually to be honest - I'm using consul also in a container, but only one per docker host.
So docker host doen't have ANY additional services which are NOT running in containers - I don't need to use puppet for that anymore.

[ChristianKniep]

yeah... that's my goal as well. Provisioning is reduce to setup the docker host and start the essential containers (registry, consul, registrator) and off you go.
http://qnib.org/2014/12/19/Docker-and-ConfigMgmt/

[sielaq]

so you can try this one:
https://github.com/eBayClassifiedsGroup/paas-in-a-box/
you have all pieces integrated together already :)

[ChristianKniep]

Uhi, please keep posting comments. 👍
You are located in Berlin? I work at locafox.de (Kreuzberg); we should grab a drink some day...
Over and out.

[rotatingJazz]

@sielaq "Why do You need consul in every container on one same DOCKER_HOST ?" -> multi-tenancy is one reason.

[sielaq]

it is not logical, you scale consul on multiple docker hosts.

If you want in every container then you run into netowrk problems so you need to run https://github.com/jpetazzo/pipework to manipulate additional interface to be able to crate your private consul network that is routable between docker hosts

[rotatingJazz]

@sielaq I suppose you assume everyone runs docker hosts on elastic cloud and not on dedis. On dedis the illogical thing is to under-use your hardware, not the opposite because of software limitations.

Besides, having multiple consul instances will be trivially possible once #550 lands.