Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpected response code: 500 (... does not have associated TTL) #44

Closed
onyxrev opened this issue Mar 19, 2017 · 2 comments
Closed

Unexpected response code: 500 (... does not have associated TTL) #44

onyxrev opened this issue Mar 19, 2017 · 2 comments

Comments

@onyxrev
Copy link

onyxrev commented Mar 19, 2017 •
edited
Loading

I'm really excited about containerpilot, but I've tried several of the examples, only to generally run into the same problem (on Linux and on OSX).

Essentially it looks like Containerpilot and Consul aren't expecting the same things. The checks are returning a 500 with a message about not having an associated TTL.

I literally cloned the repo, build the containers, and ran docker-compose -p nginx up. I also gave the more recent "microservices" example a try:
https://github.com/autopilotpattern/nodejs-example

... and encountered the same general issue. I also tried newer, older combinations of Consul & Container Pilot.

Here's the log output:

Recreating nginx_consul_1
Recreating nginx_backend_1
Recreating nginx_nginx_1
Attaching to nginx_consul_1, nginx_nginx_1, nginx_backend_1
consul_1   | 2017/03/19 06:59:54     2017-03-19 06:59:54 containerpilot: Updating consul advertise address
consul_1   | ==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
consul_1   | ==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
backend_1  | Hello server listening on port 3001
consul_1   | ==> Starting Consul agent...
consul_1   | ==> Starting Consul agent RPC...
consul_1   | ==> Consul agent running!
consul_1   |            Version: 'v0.7.2'
consul_1   |          Node name: 'd61e3a7e18d5'
consul_1   |         Datacenter: 'dc1'
consul_1   |             Server: true (bootstrap: true)
consul_1   |        Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
consul_1   |       Cluster Addr: 172.17.0.5 (LAN: 8301, WAN: 8302)
consul_1   |     Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
consul_1   |              Atlas: <disabled>
consul_1   |
consul_1   | ==> Log data will now stream in as it occurs:
consul_1   |
consul_1   |     2017/03/19 06:59:54 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:172.17.0.5:8300 Address:172.17.0.5:8300}]
consul_1   |     2017/03/19 06:59:54 [INFO] serf: EventMemberJoin: d61e3a7e18d5 172.17.0.5
consul_1   |     2017/03/19 06:59:54 [INFO] serf: EventMemberJoin: d61e3a7e18d5.dc1 172.17.0.5
consul_1   |     2017/03/19 06:59:54 [INFO] raft: Node at 172.17.0.5:8300 [Follower] entering Follower state (Leader: "")
consul_1   |     2017/03/19 06:59:54 [INFO] serf: Attempting re-join to previously known node: a54894801344: 172.17.0.7:8301
consul_1   |     2017/03/19 06:59:54 [INFO] consul: Adding LAN server d61e3a7e18d5 (Addr: tcp/172.17.0.5:8300) (DC: dc1)
consul_1   |     2017/03/19 06:59:54 [INFO] serf: Attempting re-join to previously known node: e0de3fc32f8d.dc1: 172.17.0.5:8302
consul_1   |     2017/03/19 06:59:55 [INFO] consul: Adding WAN server d61e3a7e18d5.dc1 (Addr: tcp/172.17.0.5:8300) (DC: dc1)
consul_1   |     2017/03/19 06:59:55 [INFO] serf: Re-joined to previously known node: e0de3fc32f8d.dc1: 172.17.0.5:8302
consul_1   |     2017/03/19 06:59:56 [INFO] serf: Attempting re-join to previously known node: c5bc02aa07d1: 172.17.0.6:8301
consul_1   |     2017/03/19 06:59:56 [INFO] serf: Attempting re-join to previously known node: e0de3fc32f8d: 172.17.0.5:8301
consul_1   |     2017/03/19 06:59:56 [INFO] serf: Re-joined to previously known node: e0de3fc32f8d: 172.17.0.5:8301
consul_1   |     2017/03/19 07:00:02 [ERR] agent: failed to sync remote state: No cluster leader
consul_1   |     2017/03/19 07:00:02 [WARN] raft: Heartbeat timeout from "" reached, starting election
consul_1   |     2017/03/19 07:00:02 [INFO] raft: Node at 172.17.0.5:8300 [Candidate] entering Candidate state in term 13
consul_1   |     2017/03/19 07:00:02 [INFO] raft: Election won. Tally: 1
consul_1   |     2017/03/19 07:00:02 [INFO] raft: Node at 172.17.0.5:8300 [Leader] entering Leader state
consul_1   |     2017/03/19 07:00:02 [INFO] consul: cluster leadership acquired
consul_1   |     2017/03/19 07:00:02 [INFO] consul: New leader elected: d61e3a7e18d5
consul_1   |     2017/03/19 07:00:02 [INFO] consul: member 'd61e3a7e18d5' joined, marking health alive
consul_1   |     2017/03/19 07:00:02 [INFO] consul: member 'c5bc02aa07d1' reaped, deregistering
consul_1   |     2017/03/19 07:00:02 [INFO] agent: Synced service 'backend-a80f29353236'
consul_1   |     2017/03/19 07:00:02 [INFO] consul: member 'e0de3fc32f8d' reaped, deregistering
consul_1   |     2017/03/19 07:00:02 [INFO] agent: Synced service 'backend-8fbf988363ca'
nginx_1    | 2017/03/19 07:00:02 telemetry: Listening on 172.17.0.7:9090
nginx_1    | 2017/03/19 07:00:02 ==> Starting Consul agent...
nginx_1    | 2017/03/19 07:00:02 ==> Starting Consul agent RPC...
nginx_1    | 2017/03/19 07:00:02 ==> Consul agent running!
nginx_1    | 2017/03/19 07:00:02            Version: 'v0.7.3'
nginx_1    | 2017/03/19 07:00:02            Node ID: '9459bcfa-0a36-34b2-1d7f-598c21da3344'
nginx_1    | 2017/03/19 07:00:02          Node name: '52068abadcdb'
nginx_1    | 2017/03/19 07:00:02         Datacenter: 'dc1'
nginx_1    | 2017/03/19 07:00:02             Server: false (bootstrap: false)
nginx_1    | 2017/03/19 07:00:02        Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
nginx_1    | 2017/03/19 07:00:02       Cluster Addr: 172.17.0.7 (LAN: 8301, WAN: 8302)
nginx_1    | 2017/03/19 07:00:02     Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
nginx_1    | 2017/03/19 07:00:02              Atlas: <disabled>
nginx_1    | 2017/03/19 07:00:02
nginx_1    | 2017/03/19 07:00:02 ==> Log data will now stream in as it occurs:
nginx_1    | 2017/03/19 07:00:02
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [INFO] agent: Generated unique node ID "9459bcfa-0a36-34b2-1d7f-598c21da3344" for this agent (persisted)
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [INFO] serf: EventMemberJoin: 52068abadcdb 172.17.0.7
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [WARN] manager: No servers available
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [ERR] agent: failed to sync remote state: No known Consul servers
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [INFO] agent: Joining cluster...
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [INFO] agent: (LAN) joining: [consul]
consul_1   |     2017/03/19 07:00:02 [INFO] serf: EventMemberJoin: 52068abadcdb 172.17.0.7
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [INFO] serf: EventMemberJoin: d61e3a7e18d5 172.17.0.5
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [INFO] agent: (LAN) joined: 1 Err: <nil>
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [INFO] agent: Join completed. Synced with 1 initial agents
nginx_1    | 2017/03/19 07:00:02     2017/03/19 07:00:02 [INFO] consul: adding server d61e3a7e18d5 (Addr: tcp/172.17.0.5:8300) (DC: dc1)
consul_1   |     2017/03/19 07:00:02 [INFO] agent: Synced service 'consul'
consul_1   |     2017/03/19 07:00:02 [INFO] consul: member '52068abadcdb' joined, marking health alive
consul_1   |     2017/03/19 07:00:02 [INFO] agent: Synced service 'consul-20d23ac12dbb'
consul_1   |     2017/03/19 07:00:02 [INFO] agent: Synced service 'backend-ead1c884d338'
consul_1   |     2017/03/19 07:00:02 [INFO] agent: Synced service 'backend-a4c7a6e04d61'
consul_1   |     2017/03/19 07:00:02 [INFO] agent: Synced service 'consul-3ab03b1aaf59'
backend_1  | 2017/03/19 07:00:02 Service not registered: Unexpected response code: 500 (CheckID "backend-8fbf988363ca" does not have associated TTL)
consul_1   |     2017/03/19 07:00:02 [INFO] agent: Synced service 'backend-a50b780bc79c'
consul_1   |     2017/03/19 07:00:02 [ERR] http: Request PUT /v1/agent/check/pass/backend-8fbf988363ca?note=ok, error: CheckID "backend-8fbf988363ca" does not have associated TTL from=172.17.0.6:52372
consul_1   |     2017/03/19 07:00:03 [INFO] agent: Synced service 'backend-8fbf988363ca'
consul_1   |     2017/03/19 07:00:03 [INFO] agent: Synced check 'backend-8fbf988363ca'
backend_1  | 2017/03/19 07:00:03 Service registered: backend
consul_1   |     2017/03/19 07:00:03 [INFO] agent: Synced check 'backend-8fbf988363ca'
nginx_1    | 2017/03/19 07:00:03     2017/03/19 07:00:03 [INFO] agent: Synced node info
consul_1   |     2017/03/19 07:00:04 [INFO] agent.rpc: Accepted client: 127.0.0.1:35918
consul_1   | 2017/03/19 07:00:04     2017-03-19 07:00:04 containerpilot: No peers in raft
consul_1   | 2017/03/19 07:00:04 At least one address to join must be specified.
consul_1   | 2017/03/19 07:00:04
consul_1   | 2017/03/19 07:00:04 Usage: consul join [options] address ...
consul_1   | 2017/03/19 07:00:04
consul_1   | 2017/03/19 07:00:04   Tells a running Consul agent (with "consul agent") to join the cluster
consul_1   | 2017/03/19 07:00:04   by specifying at least one existing member.
consul_1   | 2017/03/19 07:00:04
consul_1   | 2017/03/19 07:00:04 Options:
consul_1   | 2017/03/19 07:00:04
consul_1   | 2017/03/19 07:00:04   -rpc-addr=127.0.0.1:8400  RPC address of the Consul agent.
consul_1   | 2017/03/19 07:00:04   -wan                      Joins a server to another server in the WAN pool
consul_1   |     2017/03/19 07:00:04 [ERR] http: Request PUT /v1/agent/check/pass/consul-d61e3a7e18d5?note=ok, error: CheckID "consul-d61e3a7e18d5" does not have associated TTL from=127.0.0.1:60162
consul_1   | 2017/03/19 07:00:04 Unexpected response code: 500 (CheckID "consul-d61e3a7e18d5" does not have associated TTL)
consul_1   | Service not registered, registering...
consul_1   |     2017/03/19 07:00:04 [INFO] agent: Synced service 'consul-d61e3a7e18d5'
consul_1   |     2017/03/19 07:00:04 [WARN] agent: Check 'backend-a4c7a6e04d61' missed TTL, is now critical
consul_1   |     2017/03/19 07:00:04 [WARN] agent: Check 'backend-ead1c884d338' missed TTL, is now critical
consul_1   |     2017/03/19 07:00:04 [WARN] agent: Check 'backend-a50b780bc79c' missed TTL, is now critical
consul_1   |     2017/03/19 07:00:04 [WARN] agent: Check 'backend-a80f29353236' missed TTL, is now critical
consul_1   |     2017/03/19 07:00:05 [INFO] agent: Synced check 'consul-d61e3a7e18d5'
consul_1   |     2017/03/19 07:00:05 [INFO] agent: Synced check 'consul-d61e3a7e18d5'
nginx_1    | 2017/03/19 07:00:07     2017/03/19 07:00:07 [INFO] agent: Synced service 'containerpilot-52068abadcdb'
nginx_1    | 2017/03/19 07:00:07     2017/03/19 07:00:07 [ERR] http: Request PUT /v1/agent/check/pass/containerpilot-52068abadcdb?note=ok, error: CheckID "containerpilot-52068abadcdb" does not have associated TTL from=127.0.0.1:60180
nginx_1    | 2017/03/19 07:00:07 Service not registered: Unexpected response code: 500 (CheckID "containerpilot-52068abadcdb" does not have associated TTL)
nginx_1    | 2017/03/19 07:00:07     2017/03/19 07:00:07 [INFO] agent: Synced service 'containerpilot-52068abadcdb'
nginx_1    | 2017/03/19 07:00:07     2017/03/19 07:00:07 [INFO] agent: Synced check 'containerpilot-52068abadcdb'
nginx_1    | 2017/03/19 07:00:07     2017/03/19 07:00:07 [INFO] agent: Synced check 'containerpilot-52068abadcdb'
nginx_1    | 2017/03/19 07:00:07 Service registered: containerpilot
nginx_1    | 2017/03/19 07:00:09 2017/03/19 07:00:09 [notice] 85#85: signal process started
nginx_1    | 2017/03/19 07:00:12 Service not registered: Unexpected response code: 500 (CheckID "nginx-public-52068abadcdb" does not have associated TTL)
nginx_1    | 2017/03/19 07:00:12     2017/03/19 07:00:12 [ERR] http: Request PUT /v1/agent/check/pass/nginx-public-52068abadcdb?note=ok, error: CheckID "nginx-public-52068abadcdb" does not have associated TTL from=127.0.0.1:60180
nginx_1    | 2017/03/19 07:00:12     2017/03/19 07:00:12 [ERR] http: Request PUT /v1/agent/check/pass/nginx-52068abadcdb?note=ok, error: CheckID "nginx-52068abadcdb" does not have associated TTL from=127.0.0.1:60240
nginx_1    | 2017/03/19 07:00:12 Service not registered: Unexpected response code: 500 (CheckID "nginx-52068abadcdb" does not have associated TTL)
nginx_1    | 2017/03/19 07:00:12     2017/03/19 07:00:12 [INFO] agent: Synced service 'nginx-public-52068abadcdb'
nginx_1    | 2017/03/19 07:00:12     2017/03/19 07:00:12 [INFO] agent: Synced service 'nginx-52068abadcdb'
nginx_1    | 2017/03/19 07:00:12     2017/03/19 07:00:12 [INFO] agent: Synced check 'nginx-public-52068abadcdb'
nginx_1    | 2017/03/19 07:00:12     2017/03/19 07:00:12 [INFO] agent: Synced check 'nginx-52068abadcdb'
nginx_1    | 2017/03/19 07:00:12     2017/03/19 07:00:12 [INFO] agent: Synced check 'nginx-public-52068abadcdb'
nginx_1    | 2017/03/19 07:00:12 Service registered: nginx-public
nginx_1    | 2017/03/19 07:00:13     2017/03/19 07:00:13 [INFO] agent: Synced check 'nginx-52068abadcdb'
nginx_1    | 2017/03/19 07:00:13 Service registered: nginx
consul_1   |     2017/03/19 07:00:14 [INFO] agent.rpc: Accepted client: 127.0.0.1:36006
consul_1   | 2017/03/19 07:00:14     2017-03-19 07:00:14 containerpilot: No peers in raft
consul_1   | 2017/03/19 07:00:14 At least one address to join must be specified.
consul_1   | 2017/03/19 07:00:14
consul_1   | 2017/03/19 07:00:14 Usage: consul join [options] address ...
consul_1   | 2017/03/19 07:00:14
consul_1   | 2017/03/19 07:00:14   Tells a running Consul agent (with "consul agent") to join the cluster
consul_1   | 2017/03/19 07:00:14   by specifying at least one existing member.
consul_1   | 2017/03/19 07:00:14
consul_1   | 2017/03/19 07:00:14 Options:
consul_1   | 2017/03/19 07:00:14
consul_1   | 2017/03/19 07:00:14   -rpc-addr=127.0.0.1:8400  RPC address of the Consul agent.
consul_1   | 2017/03/19 07:00:14   -wan                      Joins a server to another server in the WAN pool
nginx_1    | 2017/03/19 07:00:17 ==> Newer Consul version available: 0.7.5 (currently running: 0.7.3)
consul_1   |     2017/03/19 07:00:19 [WARN] agent: Check 'consul-3ab03b1aaf59' missed TTL, is now critical
consul_1   |     2017/03/19 07:00:19 [WARN] agent: Check 'consul-20d23ac12dbb' missed TTL, is now critical
consul_1   |     2017/03/19 07:00:24 [INFO] agent.rpc: Accepted client: 127.0.0.1:36082
consul_1   | 2017/03/19 07:00:24     2017-03-19 07:00:24 containerpilot: No peers in raft
consul_1   | 2017/03/19 07:00:24 At least one address to join must be specified.
consul_1   | 2017/03/19 07:00:24
consul_1   | 2017/03/19 07:00:24 Usage: consul join [options] address ...
consul_1   | 2017/03/19 07:00:24
consul_1   | 2017/03/19 07:00:24   Tells a running Consul agent (with "consul agent") to join the cluster
consul_1   | 2017/03/19 07:00:24   by specifying at least one existing member.
consul_1   | 2017/03/19 07:00:24
consul_1   | 2017/03/19 07:00:24 Options:
consul_1   | 2017/03/19 07:00:24
consul_1   | 2017/03/19 07:00:24   -rpc-addr=127.0.0.1:8400  RPC address of the Consul agent.
consul_1   | 2017/03/19 07:00:24   -wan                      Joins a server to another server in the WAN pool

OSX:

Docker version 17.03.0-ce, build 60ccb22
docker-compose version 1.11.2, build dfed245

Linux:

Docker version 1.12.6, build 78d1802
docker-compose version 1.5.2, build unknown
@onyxrev
Copy link
Author

onyxrev commented Mar 19, 2017

Actually... I think my containers were mis-matched. After tearing everything down I do still get the 500 error, but it may benign. I can still connect to nginx. The health checks are working.

@onyxrev onyxrev closed this as completed Mar 19, 2017
@tgross
Copy link
Contributor

tgross commented Mar 20, 2017

The error message will appear once for each health check in most versions of ContainerPilot even if it's all working. See TritonDataCenter/containerpilot#264 for some background as to why. This should be fixed in TritonDataCenter/containerpilot#275 (which was released in 2.6.1). But if not I'd love if you'd open a new issue about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@onyxrev @tgross