rabbitmq should be a default user

[eveningsteps]

not sure if this is an intended design decision, or pure oversight, but this took me quite a lot of time to narrow down.

consider the following section of Docker Compose, or an equivalent Nomad service definition with Docker task driver:

services:
  rabbitmq:
    image: rabbitmq:3.12-management-alpine
    ports:
      - "15672:15672"
      - "5672:5672"
    volumes:
      - "./rabbitmq/etc/enabled_plugins:/etc/rabbitmq/enabled_plugins"
      - "./rabbitmq/etc/definitions.json:/etc/rabbitmq/definitions.json"
      - "./rabbitmq/etc/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf"
    healthcheck:
      test: rabbitmq-diagnostics ping
      interval: 3s
      timeout: 10s
      retries: 60

the use of rabbitmq-diagnostics in the health check section leads to the container crashing and exiting due to not having access to the access cookie (/var/lib/rabbitmq/.erlang.cookie):

crash

2024-04-16 16:51:45.685285+00:00 [error] <0.148.0> Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>     supervisor: {local,net_sup}
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>     errorContext: start_error
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>     reason: {"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>              [{auth,init_no_setcookie,0,[{file,"auth.erl"},{line,313}]},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>               {auth,init,1,[{file,"auth.erl"},{line,165}]},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>               {gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>               {gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>               {proc_lib,init_p_do_apply,3,
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>                         [{file,"proc_lib.erl"},{line,240}]}]}
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>     offender: [{pid,undefined},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>                {id,auth},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>                {mfargs,{auth,start_link,[]}},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>                {restart_type,permanent},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>                {significant,false},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>                {shutdown,2000},
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0>                {child_type,worker}]
2024-04-16 16:51:45.701306+00:00 [error] <0.146.0> 
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>   crasher:
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     initial call: auth:init/1
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     pid: <0.148.0>
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     registered_name: []
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     exception error: "Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces"
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>       in function  auth:init_no_setcookie/0 (auth.erl, line 313)
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>       in call from auth:init/1 (auth.erl, line 165)
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>       in call from gen_server:init_it/2 (gen_server.erl, line 851)
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>       in call from gen_server:init_it/6 (gen_server.erl, line 814)
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     ancestors: [net_sup,kernel_sup,<0.47.0>]
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     message_queue_len: 0
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     messages: []
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     links: [<0.146.0>]
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     dictionary: []
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     trap_exit: true
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     status: running
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     heap_size: 1598
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     stack_size: 28
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>     reductions: 2279
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0>   neighbours:
2024-04-16 16:51:45.701192+00:00 [error] <0.148.0> 
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0> 
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0> BOOT FAILED
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0> ===========
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0> Exception during startup:
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0> 
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",[{auth,init_no_setcookie,0,[{file,"auth.erl"},{line,313}]},{auth,init,1,[{file,"auth.erl"},{line,165}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{clean_halt => false,name => rabbit_prelaunch_15@localhost,name_domain => shortnames,net_tickintensity => 4,net_ticktime => 60,supervisor => net_sup_dynamic}]},permanent,false,1000,supervisor,[erl_distribution]}}}}
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0> 
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0>     rabbit_prelaunch_dist:duplicate_node_check/1, line 80
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0>     rabbit_prelaunch_dist:setup/1, line 23
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0>     rabbit_prelaunch:do_run/0, line 115
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0>     rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0>     supervisor:do_start_child_i/3, line 420
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0>     supervisor:do_start_child/2, line 406
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0>     supervisor:-start_children/2-fun-0-/3, line 390
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0>     supervisor:children_map/4, line 1256
2024-04-16 16:51:45.711346+00:00 [error] <0.132.0> 
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>     supervisor: {local,rabbit_prelaunch_sup}
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>     errorContext: start_error
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>     reason: {badmatch,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                 {error,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                     {{shutdown,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                          {failed_to_start_child,auth,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                              {"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                               [{auth,init_no_setcookie,0,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                    [{file,"auth.erl"},{line,313}]},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                {auth,init,1,[{file,"auth.erl"},{line,165}]},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                {gen_server,init_it,2,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                    [{file,"gen_server.erl"},{line,851}]},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                {gen_server,init_it,6,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                    [{file,"gen_server.erl"},{line,814}]},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                {proc_lib,init_p_do_apply,3,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                    [{file,"proc_lib.erl"},{line,240}]}]}}},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                      {child,undefined,net_sup_dynamic,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                          {erl_distribution,start_link,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                              [#{clean_halt => false,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                 name => rabbit_prelaunch_15@localhost,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                 name_domain => shortnames,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                 net_tickintensity => 4,net_ticktime => 60,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                                 supervisor => net_sup_dynamic}]},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                          permanent,false,1000,supervisor,
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                          [erl_distribution]}}}}
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>     offender: [{pid,undefined},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                {id,prelaunch},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,[]}},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                {restart_type,transient},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                {significant,false},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                {shutdown,5000},
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0>                {child_type,worker}]
2024-04-16 16:51:46.716641+00:00 [error] <0.132.0> 
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>   crasher:
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     initial call: application_master:init/4
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     pid: <0.130.0>
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     registered_name: []
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     exception exit: {{shutdown,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                       {failed_to_start_child,prelaunch,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                        {badmatch,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                         {error,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                          {{shutdown,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                            {failed_to_start_child,auth,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                             {"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                              [{auth,init_no_setcookie,0,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                                [{file,"auth.erl"},{line,313}]},
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                               {auth,init,1,[{file,"auth.erl"},{line,165}]},
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                               {gen_server,init_it,2,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                                [{file,"gen_server.erl"},{line,851}]},
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                               {gen_server,init_it,6,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                                [{file,"gen_server.erl"},{line,814}]},
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                               {proc_lib,init_p_do_apply,3,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                                [{file,"proc_lib.erl"},{line,240}]}]}}},
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                           {child,undefined,net_sup_dynamic,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                            {erl_distribution,start_link,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                             [#{clean_halt => false,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                                name => rabbit_prelaunch_15@localhost,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                                name_domain => shortnames,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                                net_tickintensity => 4,net_ticktime => 60,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                                supervisor => net_sup_dynamic}]},
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                            permanent,false,1000,supervisor,
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                            [erl_distribution]}}}}}},
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>                      {rabbit_prelaunch_app,start,[normal,[]]}}
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>       in function  application_master:init/4 (application_master.erl, line 142)
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     ancestors: [<0.129.0>]
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     message_queue_len: 1
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     messages: [{'EXIT',<0.131.0>,normal}]
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     links: [<0.129.0>,<0.44.0>]
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     dictionary: []
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     trap_exit: true
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     status: running
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     heap_size: 610
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     stack_size: 28
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>     reductions: 188
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0>   neighbours:
2024-04-16 16:51:46.717109+00:00 [error] <0.130.0> 
2024-04-16 16:51:46.718327+00:00 [notice] <0.44.0> Application rabbitmq_prelaunch exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",[{auth,init_no_setcookie,0,[{file,"auth.erl"},{line,313}]},{auth,init,1,[{file,"auth.erl"},{line,165}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{clean_halt => false,name => rabbit_prelaunch_15@localhost,name_domain => shortnames,net_tickintensity => 4,net_ticktime => 60,supervisor => net_sup_dynamic}]},permanent,false,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{\"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces\",[{auth,init_no_setcookie,0,[{file,\"auth.erl\"},{line,313}]},{auth,init,1,[{file,\"auth.erl\"},{line,165}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,851}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{clean_halt => false,name => rabbit_prelaunch_15@localhost,name_domain => shortnames,net_tickintensity => 4,net_ticktime => 60,supervisor => net_sup_dynamic}]},permanent,false,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}

while adding user: rabbitmq solves the cookie access issue, I'm thinking that the default user should have been pre-built into the container, like, for example, into that of keycloak: https://github.com/keycloak/keycloak/blob/1e3837421e9cea41aa91ef99800af1f82f91089b/quarkus/container/Dockerfile#L31-L34 -- are there any specific reasons to avoid doing that?

[lukebakken]

Thanks for using RabbitMQ.

I'm thinking that the default user should have been pre-built into the container, like, for example, into that of keycloak

The rabbitmq user is added in both of the images we provide:

• https://github.com/docker-library/rabbitmq/blob/master/Dockerfile-alpine.template#L297-L298
• https://github.com/docker-library/rabbitmq/blob/master/Dockerfile-ubuntu.template#L286-L288

TBH I'm not sure why you are seeing that particular crash. The following command works just fine on my Arch Linux system:

docker run --pull always --rm rabbitmq:3.12-management-alpine

Could you please provide a complete docker compose file that I can use to try and reproduce this issue? Ideally, you'd create a git repository so that the files you're mounting can be included.

[eveningsteps]

apologies for being unclear -- I meant that the Dockerfile is missing the USER instruction. what's worse is that, once I have isolated the service, I don't seem to be able to reproduce the exact issue anymore.

I'll keep looking into it and bump the discussion once I get it crashing

[lukebakken]

Hmm, well, this docker image is used by thousands of RabbitMQ users, both OSS users and commercial users who pay for RabbitMQ support... which means, what you're seeing is almost certainly due to your environment.

My guess is that, perhaps, you had an old volume being mounted to /var/lib/rabbitmq that doesn't have the correct permissions, though that doesn't explain the behavior you report.

The USER statement is not needed because the entrypoint script changes to the rabbitmq user upon startup - https://github.com/docker-library/rabbitmq/blob/master/docker-entrypoint.sh#L4-L11

[gmile]

For what it's worth, I am seeing exactly the same problem with reading the cookie:

Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces

Full output

queue-1                 | 2024-04-26 07:55:52.544249+00:00 [error] <0.148.0> Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>     supervisor: {local,net_sup}
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>     errorContext: start_error
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>     reason: {"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>              [{auth,init_no_setcookie,0,[{file,"auth.erl"},{line,313}]},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>               {auth,init,1,[{file,"auth.erl"},{line,165}]},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>               {gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>               {gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>               {proc_lib,init_p_do_apply,3,
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>                         [{file,"proc_lib.erl"},{line,240}]}]}
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>     offender: [{pid,undefined},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>                {id,auth},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>                {mfargs,{auth,start_link,[]}},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>                {restart_type,permanent},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>                {significant,false},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>                {shutdown,2000},
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0>                {child_type,worker}]
queue-1                 | 2024-04-26 07:55:52.569290+00:00 [error] <0.146.0> 
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>   crasher:
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     initial call: auth:init/1
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     pid: <0.148.0>
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     registered_name: []
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     exception error: "Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces"
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>       in function  auth:init_no_setcookie/0 (auth.erl, line 313)
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>       in call from auth:init/1 (auth.erl, line 165)
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>       in call from gen_server:init_it/2 (gen_server.erl, line 851)
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>       in call from gen_server:init_it/6 (gen_server.erl, line 814)
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     ancestors: [net_sup,kernel_sup,<0.47.0>]
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     message_queue_len: 0
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     messages: []
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     links: [<0.146.0>]
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     dictionary: []
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     trap_exit: true
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     status: running
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     heap_size: 1598
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     stack_size: 28
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>     reductions: 2263
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0>   neighbours:
queue-1                 | 2024-04-26 07:55:52.569065+00:00 [error] <0.148.0> 
queue-1                 | 
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0> 
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0> BOOT FAILED
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0> ===========
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0> Exception during startup:
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0> 
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",[{auth,init_no_setcookie,0,[{file,"auth.erl"},{line,313}]},{auth,init,1,[{file,"auth.erl"},{line,165}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{clean_halt => false,name => rabbit_prelaunch_20@localhost,name_domain => shortnames,net_tickintensity => 4,net_ticktime => 60,supervisor => net_sup_dynamic}]},permanent,false,1000,supervisor,[erl_distribution]}}}}
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0> 
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0>     rabbit_prelaunch_dist:duplicate_node_check/1, line 80
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0>     rabbit_prelaunch_dist:setup/1, line 23
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0>     rabbit_prelaunch:do_run/0, line 115
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0>     rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0>     supervisor:do_start_child_i/3, line 420
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0>     supervisor:do_start_child/2, line 406
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0>     supervisor:-start_children/2-fun-0-/3, line 390
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0>     supervisor:children_map/4, line 1256
queue-1                 | 2024-04-26 07:55:52.621630+00:00 [error] <0.132.0> 
queue-1                 | BOOT FAILED
queue-1                 | ===========
queue-1                 | Exception during startup:
queue-1                 | 
queue-1                 | error:{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",[{auth,init_no_setcookie,0,[{file,"auth.erl"},{line,313}]},{auth,init,1,[{file,"auth.erl"},{line,165}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{clean_halt => false,name => rabbit_prelaunch_20@localhost,name_domain => shortnames,net_tickintensity => 4,net_ticktime => 60,supervisor => net_sup_dynamic}]},permanent,false,1000,supervisor,[erl_distribution]}}}}
queue-1                 | 
queue-1                 |     rabbit_prelaunch_dist:duplicate_node_check/1, line 80
queue-1                 |     rabbit_prelaunch_dist:setup/1, line 23
queue-1                 |     rabbit_prelaunch:do_run/0, line 115
queue-1                 |     rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
queue-1                 |     supervisor:do_start_child_i/3, line 420
queue-1                 |     supervisor:do_start_child/2, line 406
queue-1                 |     supervisor:-start_children/2-fun-0-/3, line 390
queue-1                 |     supervisor:children_map/4, line 1256
queue-1                 | 
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>     supervisor: {local,rabbit_prelaunch_sup}
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>     errorContext: start_error
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>     reason: {badmatch,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                 {error,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                     {{shutdown,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                          {failed_to_start_child,auth,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                              {"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                               [{auth,init_no_setcookie,0,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                    [{file,"auth.erl"},{line,313}]},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                {auth,init,1,[{file,"auth.erl"},{line,165}]},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                {gen_server,init_it,2,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                    [{file,"gen_server.erl"},{line,851}]},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                {gen_server,init_it,6,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                    [{file,"gen_server.erl"},{line,814}]},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                {proc_lib,init_p_do_apply,3,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                    [{file,"proc_lib.erl"},{line,240}]}]}}},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                      {child,undefined,net_sup_dynamic,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                          {erl_distribution,start_link,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                              [#{clean_halt => false,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                 name => rabbit_prelaunch_20@localhost,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                 name_domain => shortnames,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                 net_tickintensity => 4,net_ticktime => 60,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                                 supervisor => net_sup_dynamic}]},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                          permanent,false,1000,supervisor,
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                          [erl_distribution]}}}}
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>     offender: [{pid,undefined},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                {id,prelaunch},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,[]}},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                {restart_type,transient},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                {significant,false},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                {shutdown,5000},
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0>                {child_type,worker}]
queue-1                 | 2024-04-26 07:55:53.636390+00:00 [error] <0.132.0> 
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>   crasher:
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     initial call: application_master:init/4
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     pid: <0.130.0>
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     registered_name: []
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     exception exit: {{shutdown,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                       {failed_to_start_child,prelaunch,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                        {badmatch,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                         {error,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                          {{shutdown,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                            {failed_to_start_child,auth,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                             {"Error when reading /var/lib/rabbitmq/.erlang.cookie: eacces",
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                              [{auth,init_no_setcookie,0,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                                [{file,"auth.erl"},{line,313}]},
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                               {auth,init,1,[{file,"auth.erl"},{line,165}]},
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                               {gen_server,init_it,2,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                                [{file,"gen_server.erl"},{line,851}]},
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                               {gen_server,init_it,6,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                                [{file,"gen_server.erl"},{line,814}]},
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                               {proc_lib,init_p_do_apply,3,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                                [{file,"proc_lib.erl"},{line,240}]}]}}},
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                           {child,undefined,net_sup_dynamic,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                            {erl_distribution,start_link,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                             [#{clean_halt => false,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                                name => rabbit_prelaunch_20@localhost,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                                name_domain => shortnames,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                                net_tickintensity => 4,net_ticktime => 60,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                                supervisor => net_sup_dynamic}]},
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                            permanent,false,1000,supervisor,
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                            [erl_distribution]}}}}}},
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>                      {rabbit_prelaunch_app,start,[normal,[]]}}
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>       in function  application_master:init/4 (application_master.erl, line 142)
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     ancestors: [<0.129.0>]
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     message_queue_len: 1
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     messages: [{'EXIT',<0.131.0>,normal}]
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     links: [<0.129.0>,<0.44.0>]
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     dictionary: []
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     trap_exit: true
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     status: running
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     heap_size: 610
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     stack_size: 28
queue-1                 | 2024-04-26 07:55:53.636921+00:00 [error] <0.130.0>     reductions: 188

Based on a shallow GitHub search, this problem is not new. For example, several years ago someone reported a similar problem here: #318.

In my experience, it's reproducible typically when starting RabbitMQ container during GitHub Actions run, maybe once in 5 runs.

[lukebakken]

Thanks for using RabbitMQ.

As I stated in this comment, if this were a common issue, Team RabbitMQ would have heard about it over and over by now.

In my experience, it's reproducible typically when starting RabbitMQ container during GitHub Actions run, maybe once in 5 runs

Nobody has provided steps to reproduce this issue. If you'd like to contribute, provide a GitHub repo with the appropriate action to demonstrate the eacces error.

For what it's worth, I use this docker image all of the time when testing the RabbitMQ .NET Client via GitHub Actions, and have never seen the eacces issue:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/.github/workflows/build-test.yaml#L179-L181
https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/.ci/ubuntu/gha-setup.sh#L64-L82
https://github.com/rabbitmq/rabbitmq-dotnet-client/actions/runs/8841798584/job/24279491156

[jyotty]

I too have only been seeing this with Github Actions and docker compose health checks.

I think what's happening is that the healthcheck command is firing as root, creating the cookie file as root, in between this part of the entrypoint chowning /var/lib/rabbitmq and rabbitmq-server, having dropped permissions with gosu in the entrypoint, creating it on its own. Adjusting the start_interval of the healthcheck to 10 seconds or so resolved the issue.