Document how clients not on K8s can join a DC in K8s

[ndhanushkodi]

Documents hashicorp/consul-helm#740.

Also resolves/helps with hashicorp/consul-helm#743.

[ndhanushkodi]

In my testing, I did not need to add this annotation, even though I configured my server gossip port to be 9301. consul members still successfully showed all members of the cluster.

[ndhanushkodi]

I think it's still useful documentation to have, but that's why I didn't include this annotation in the instructions below.

[ishustava]

Hmm that's interesting. Did you change the gossip ports on the servers or just the external client? I definitely see in the code that it's being used.

[lkysow]

weird, I don't know how the join could have worked without this? Wouldn't it try 8301 unless explicitly configured?

[ndhanushkodi]

@ishustava Yup, I did change the gossip port on the server to 9301. And the client just used -retry-join 'provider=k8s host_network=true label_selector="app=consul,component=server"' to join the servers. I know its super weird I would expect it to use that code as well!

@lkysow Yup, I would have expected the external client agent fail to join because it would try 8301. However, when trying the exactly the steps here while replacing the retry-join with -retry-join 'provider=k8s host_network=true label_selector="app=consul,component=server"' I observed it joining just fine and on port 9301, which is definitely mind boggling.

I will retry those steps one more time tomorrow morning just to make sure.

[ndhanushkodi]

Update: I ran through the steps with -retry-join 'provider=k8s host_network=true label_selector="app=consul,component=server"' on the external agent without the annotation and it works. In fact, when I do add the annotation, it doesn't accept an integer value, and if I add the port "9301" as a string, the consul clients on k8s itself fail to join saying no consul servers found.

I think for these docs I should remove this annotation, since it doesn't seem to work. wdyt @ishustava @lkysow?

[lkysow]

As discussed lets:

1. Add a warning that if you're using 9301 and your consul servers are on different nodes (which is unlikely because in that case you wouldn't need to use 9301) then you can't use autojoin
2. Create a bug in go-discover about the annotation issue

[ndhanushkodi]

I removed the note about the annotation since its currently broken (we can add it back once the bug is fixed in go-discover), and added the note/warning about default port. I phrased it a bit differently, let me know if you think its clear or not

[ndhanushkodi]

Writing the bug right now. EDIT: hashicorp/go-discover#166

[ishustava]

Looking pretty good! Thanks so much for doing this work and writing the docs; I like that the networking requirements sound much more clear now! I've left a few edits and suggestions, and once those are resolved, it's good to go!

[ishustava]

Hmm that's interesting. Did you change the gossip ports on the servers or just the external client? I definitely see in the code that it's being used.

[lkysow]

This is looking great!

[lkysow]

weird, I don't know how the join could have worked without this? Wouldn't it try 8301 unless explicitly configured?

[ndhanushkodi]

@lkysow @ishustava Thank you so much for the thorough reviews, it really helped. I think I've left open only the stuff worth taking another look at and resolved anything else. Going to re-request your reviews.

[lkysow]

Suggested change

	### Auto-join on via host ports
	### Auto-join via host ports

[lkysow]

💯 looks great! Just comment about not setting -client 0.0.0.0 since that exposes the HTTP API.

[ishustava]

🎉

[hashicorp-ci]

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/307179.