Error 503 and then token is required when using jupyter/docker-stacks image example

[wenzeslaus]

I'm trying the example from the README for Using jupyter/docker-stacks images but I get error 503 and then token is required. I used these commands:

docker pull jupyter/minimal-notebook
export TOKEN=$( head -c 30 /dev/urandom | xxd -p )
docker run --net=host -d -e CONFIGPROXY_AUTH_TOKEN=$TOKEN --name=proxy jupyter/configurable-http-proxy --default-target http://127.0.0.1:9999
docker run --net=host -d -e CONFIGPROXY_AUTH_TOKEN=$TOKEN --name=tmpnb -v /var/run/docker.sock:/docker.sock jupyter/tmpnb python orchestrate.py --image='jupyter/minimal-notebook' --command="jupyter notebook --NotebookApp.base_url={base_path} --ip=0.0.0.0 --port {port}"

At port 8000 I get first 503:

503: Proxy Target Missing
The upstream service is unavailable
configurable-http-proxy

And then when I reload the page, I get request for password or token:

Token authentication is enabled. You need to open the notebook server with its first-time login token in the URL, or enable a password in order to gain access. The command: 

This docker log proxy:

02:57:13.327 - info: [ConfigProxy] Proxying http://*:8000 to http://127.0.0.1:9999
02:57:13.331 - info: [ConfigProxy] Proxy API at http://localhost:8001/api/routes
02:57:21.257 - error: [ConfigProxy] Proxy error:  code=ECONNREFUSED, errno=ECONNREFUSED, syscall=connect, address=127.0.0.1, port=9999
02:57:21.259 - error: [ConfigProxy] 503 GET /user/aInZOEHNrGSg/api/sessions?_=1483715213616
02:57:21.266 - error: [ConfigProxy] Proxy error:  code=ECONNREFUSED, errno=ECONNREFUSED, syscall=connect, address=127.0.0.1, port=9999
02:57:21.266 - error: [ConfigProxy] 503 GET /user/aInZOEHNrGSg/api/terminals?_=1483715213617
02:57:29.428 - error: [ConfigProxy] Proxy error:  code=ECONNREFUSED, errno=ECONNREFUSED, syscall=connect, address=127.0.0.1, port=9999
02:57:29.428 - error: [ConfigProxy] 503 GET /

This is part of the docker log tmpnb:

[I 170107 02:53:12 spawnpool:148] Performing initial pool cleanup
/usr/local/lib/python3.4/site-packages/docker/utils/utils.py:591: UserWarning: docker.utils.create_host_config() is deprecated. Please use Client.create_host_config() instead.
  'docker.utils.create_host_config() is deprecated. Please use '
[W 170107 02:53:16 dockworker:176] ['Your kernel does not support swap limit capabilities, memory limited without swap.']
[I 170107 02:53:16 dockworker:179] Created container 704e91d2317dfa4168b9e844445e6e2084a0d7a153c56532f0f3af2ccd39de7d
...
[I 170107 02:53:24 spawnpool:305] Proxied path [user/YAkGsKr7V3Bk] to port [32898].
[I 170107 02:53:24 spawnpool:311] Adding container [PooledContainer(id=33cd9996e1782f33551332d1e860ff5cf332754a03e85f440e75495001ddb930, path=user/YAkGsKr7V3Bk)] to the pool.
[I 170107 02:53:24 spawnpool:358] Server [user/eiyu6i1UujHb] at address [127.0.0.1:32901] has booted! Have at it.
[I 170107 02:53:24 spawnpool:305] Proxied path [user/eiyu6i1UujHb] to port [32901].
[I 170107 02:53:24 spawnpool:311] Adding container [PooledContainer(id=26f0c3d8a66f7249704c80b774237821a63982ebad233d445476aea8ae96f738, path=user/eiyu6i1UujHb)] to the pool.
[I 170107 02:53:24 spawnpool:248] Launched [10] new containers.
[I 170107 02:53:24 orchestrate:489] Culling containers unused for 3600 seconds every 600 seconds.
[I 170107 02:53:24 orchestrate:493] Listening on *:9999
[I 170107 02:53:24 orchestrate:497] Admin listening on 127.0.0.1:10000
[I 170107 02:53:52 web:1946] 200 GET / (127.0.0.1) 8.51ms
[I 170107 02:53:52 orchestrate:146] Allocated [user/eiyu6i1UujHb] from the pool.
[I 170107 02:53:52 orchestrate:160] Redirecting [/spawn/] -> [/user/eiyu6i1UujHb/tree].
[I 170107 02:53:52 web:1946] 302 GET /spawn/ (127.0.0.1) 1.19ms

These are running containers:

...
ca074600de6f        jupyter/minimal-notebook          "tini -- /bin/sh -c '"   5 minutes ago       Up 5 minutes                  127.0.0.1:32913->8888/tcp   tmp.jupyterminimal-notebook.uvjxpvn4h7pd
fae3568c8a87        jupyter/minimal-notebook          "tini -- /bin/sh -c '"   5 minutes ago       Up 5 minutes                  127.0.0.1:32912->8888/tcp   tmp.jupyterminimal-notebook.ja8drA246CQG
df56a13cea4a        jupyter/tmpnb                     "python orchestrate.p"   5 minutes ago       Up 5 minutes                                              tmpnb
e1aa82e267da        jupyter/configurable-http-proxy   "configurable-http-pr"   10 minutes ago      Up 10 minutes                                             proxy

I get the same when I use --NotebookApp.trust_xheaders=True as in Using jupyter/docker-stacks images or not as in Launching with your own Docker images:

docker run --net=host -d -e CONFIGPROXY_AUTH_TOKEN=$TOKEN --name=tmpnb -v /var/run/docker.sock:/docker.sock jupyter/tmpnb python orchestrate.py --image='jupyter/minimal-notebook' --command='start-notebook.sh "--NotebookApp.base_url={base_path} --ip=0.0.0.0 --port {port} --NotebookApp.trust_xheaders=True"'

The basic example from README works for me.

[parente]

Try adding --use_tokens=True after orchestrate.py.

Token support was a recent addition to the notebook server, docker stacks, and tmpnb. We just need to update the README here.

[wenzeslaus]

I tried --use_tokens=True but still with the same result. Thanks for the prompt answer anyway.

When I wait a bit, I don't get 503, just the token request. From 8000, I get redirected to:

http://localhost:8000/user/Zi6TwJOKgSVI/login?next=%2Fuser%2FZi6TwJOKgSVI%2Ftree%3Ftoken%3D8f024144ef92336602a153a20227149a2581c6720bb9eb99

This is the docker logs tmpnb with compose.py --log-to-stderr --logging=debug:

[D 170107 04:33:06 curl_httpclient:484] b'Connected to (nil) (127.0.0.1) port 8001 (#0)'
[D 170107 04:33:06 curl_httpclient:484] b'0x1bde640 is at send pipe head!'
[D 170107 04:33:06 curl_httpclient:484] b'STATE: CONNECT => DO handle 0x1c558f0; (connection #0)'
[D 170107 04:33:06 curl_httpclient:487] > b'POST /api/routes/user/VTvKrbYhIXli HTTP/1.1'
[D 170107 04:33:06 curl_httpclient:487] > b'User-Agent: Mozilla/5.0 (compatible; pycurl)'
[D 170107 04:33:06 curl_httpclient:487] > b'Host: 127.0.0.1:8001'
[D 170107 04:33:06 curl_httpclient:487] > b'Accept: */*'
[D 170107 04:33:06 curl_httpclient:487] > b'Accept-Encoding: gzip,deflate'
[D 170107 04:33:06 curl_httpclient:487] > b'Authorization: token e57626fc235dea05d18e98331dd1c40efc694471fc64f676e3c62585961e'
[D 170107 04:33:06 curl_httpclient:487] > b'Content-Length: 120'
[D 170107 04:33:06 curl_httpclient:487] > b'Content-Type: application/x-www-form-urlencoded'
[D 170107 04:33:06 curl_httpclient:487] > b''
[D 170107 04:33:06 curl_httpclient:484] b'STATE: DO => DO_DONE handle 0x1c558f0; (connection #0)'
[D 170107 04:33:06 curl_httpclient:484] b'STATE: DO_DONE => WAITPERFORM handle 0x1c558f0; (connection #0)'
[D 170107 04:33:06 curl_httpclient:484] b'STATE: WAITPERFORM => PERFORM handle 0x1c558f0; (connection #0)'
[D 170107 04:33:06 curl_httpclient:484] b'additional stuff not fine transfer.c:1037: 0 0'
[D 170107 04:33:06 curl_httpclient:489] > b'{"container_id": "7441d5fee6848557eb404c6f7172fe51c7f9e79db5343301c610defedcc3081d", "target": "http://127.0.0.1:32941"}'
[D 170107 04:33:06 curl_httpclient:484] b'additional stuff not fine transfer.c:1037: 0 0'
[D 170107 04:33:06 curl_httpclient:484] b'HTTP 1.1 or later with persistent connection, pipelining supported'
[D 170107 04:33:06 curl_httpclient:487] < b'HTTP/1.1 201 Created'
[D 170107 04:33:06 curl_httpclient:487] < b'Date: Sat, 07 Jan 2017 04:33:06 GMT'
[D 170107 04:33:06 curl_httpclient:487] < b'Connection: keep-alive'
[D 170107 04:33:06 curl_httpclient:487] < b'Transfer-Encoding: chunked'
[D 170107 04:33:06 curl_httpclient:487] < b''
[D 170107 04:33:06 curl_httpclient:484] b'STATE: PERFORM => DONE handle 0x1c558f0; (connection #0)'
...
[I 170107 04:33:06 spawnpool:248] Launched [10] new containers.
[D 170107 04:33:06 spawnpool:256] Heartbeat complete. The pool now includes [10] containers.
[I 170107 04:33:06 orchestrate:489] Culling containers unused for 3600 seconds every 600 seconds.
[I 170107 04:33:06 orchestrate:493] Listening on *:9999
[I 170107 04:33:06 orchestrate:497] Admin listening on 127.0.0.1:10000
[I 170107 04:33:22 web:1946] 200 GET /user/aInZOEHNrGSg/api/sessions?_=1483715213808 (127.0.0.1) 2.13ms
[I 170107 04:33:22 web:1946] 200 GET /user/aInZOEHNrGSg/api/terminals?_=1483715213809 (127.0.0.1) 4.53ms
[I 170107 04:34:08 web:1946] 304 GET / (127.0.0.1) 3.64ms
[I 170107 04:34:08 orchestrate:146] Allocated [user/VTvKrbYhIXli] from the pool.
[I 170107 04:34:08 orchestrate:160] Redirecting [/spawn/] -> [/user/VTvKrbYhIXli/tree?token=95bb1bbd478de9549e7b81bdffd18dcaf1111291af682f86].
[I 170107 04:34:08 web:1946] 302 GET /spawn/ (127.0.0.1) 0.82ms
[I 170107 04:34:22 web:1946] 200 GET /user/aInZOEHNrGSg/api/sessions?_=1483715213810 (127.0.0.1) 2.87ms
[I 170107 04:34:22 web:1946] 200 GET /user/aInZOEHNrGSg/api/terminals?_=1483715213811 (127.0.0.1) 2.58ms
[I 170107 04:35:21 web:1946] 200 GET /user/aInZOEHNrGSg/api/sessions?_=1483715213812 (127.0.0.1) 6.69ms
[I 170107 04:35:21 web:1946] 200 GET /user/aInZOEHNrGSg/api/terminals?_=1483715213813 (127.0.0.1) 2.37ms
[I 170107 04:35:26 web:1946] 304 GET / (127.0.0.1) 2.30ms
[I 170107 04:35:26 orchestrate:146] Allocated [user/Zi6TwJOKgSVI] from the pool.
[I 170107 04:35:26 orchestrate:160] Redirecting [/spawn/] -> [/user/Zi6TwJOKgSVI/tree?token=8f024144ef92336602a153a20227149a2581c6720bb9eb99].
[I 170107 04:35:26 web:1946] 302 GET /spawn/ (127.0.0.1) 1.36ms
...
[I 170107 04:38:21 web:1946] 200 GET /user/aInZOEHNrGSg/api/sessions?_=1483715213818 (127.0.0.1) 2.79ms
[I 170107 04:39:21 web:1946] 200 GET /user/aInZOEHNrGSg/api/sessions?_=1483715213820 (127.0.0.1) 1.83ms
[I 170107 04:39:21 web:1946] 200 GET /user/aInZOEHNrGSg/api/terminals?_=1483715213821 (127.0.0.1) 3.33ms
[I 170107 04:40:21 web:1946] 200 GET /user/aInZOEHNrGSg/api/sessions?_=1483715213822 (127.0.0.1) 2.80ms
[I 170107 04:40:21 web:1946] 200 GET /user/aInZOEHNrGSg/api/terminals?_=1483715213823 (127.0.0.1) 3.38ms

Filling out the token would be actually fine with me if I would know it.

[wenzeslaus]

I see, when I specify the token with --NotebookApp.token= it is the token for all containers, so users need to know the token but nobody else can create the tmp nb.

docker run --net=host -d -e CONFIGPROXY_AUTH_TOKEN=$TOKEN --name=tmpnb -v /var/run/docker.sock:/docker.sock jupyter/tmpnb python orchestrate.py --log-to-stderr --logging=debug --use-tokens=True --image='jupyter/minimal-notebook' --command='start-notebook.sh "--NotebookApp.base_url={base_path} --ip=0.0.0.0 --port {port} --NotebookApp.trust_xheaders=True --NotebookApp.token=22f5b..."'

But as mentioned earlier without specifying the token, it does not work.

[wenzeslaus]

--use-tokens=True and --NotebookApp.token=22f5b together works as expected: When opening a new session, user must enter the token.

I don't know how to not require the token from the user if I would decide to do so.

--use-tokens=False and --NotebookApp.token=22f5b requires user to enter token every time (or almost every time) new page opens. So, for example, when user clicks Download as PDF, token is required, after entering it, PDF is downloaded, but it does not contain the computed cells. I don't know if that's expected or not.

I get the 503 error time to time but it is always just right after start of the "tmpnb" container, so I think at this point is it just because it is not ready yet, so no issue there.