Cannot stop gateway

[UnderGreen]

I started local cluster on Ubuntu 14.04 with managers, 1 storage and 1 gateway. When I tried to stop gateway I have stucked command(can be interrrupted only with ctrl-c):

ubuntu@leofs1:/usr/local/leofs/1.2.7$ leo_gateway/bin/leo_gateway stop
ok

And status after ctrl-c:

ubuntu@leofs1:/usr/local/leofs/1.2.7$ leofs-adm status
 [System Confiuration]
---------------------------------+----------
 Item                            | Value    
---------------------------------+----------
 Basic/Consistency level
---------------------------------+----------
                  system version | 1.2.7
                      cluster Id | clid
                           DC Id | n1
                  Total replicas | 1
        number of successes of R | 1
        number of successes of W | 1
        number of successes of D | 1
 number of DC-awareness replicas | 0
                       ring size | 2^128
---------------------------------+----------
 Multi DC replication settings
---------------------------------+----------
      max number of joinable DCs | 3
         number of replicas a DC | 1
---------------------------------+----------
 Manager RING hash
---------------------------------+----------
               current ring-hash | 28e8d0fe
              previous ring-hash | 28e8d0fe
---------------------------------+----------

 [State of Node(s)]
-------+-----------------------------+--------------+----------------+----------------+----------------------------
 type  |            node             |    state     |  current ring  |   prev ring    |          updated at         
-------+-----------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@10.54.254.24      | running      | 28e8d0fe       | 28e8d0fe       | 2015-03-11 15:23:57 +0600
  G    | gateway_0@10.54.254.24      | stop         | -1             | -1             | 2015-03-11 16:20:37 +0600
-------+-----------------------------+--------------+----------------+----------------+----------------------------

I tried to start gateway again, but have this error: Node is already running!.

[yosukehara]

It seems LeoFS's gateway failed to stop own process. You can check whether the process still alive or not with the ps command. If the process is still alive, you need to execute the kill command to kill the process.

$ ps aux | grep leo_gateway

We'll check LeoFS gateway's stop command.

[UnderGreen]

Yes, sure. Process is still alive. Don't see in logs info about this problem, maybe it will be able to help you.

[mocchira]

@UnderGreen @yosukehara

Googling erlang init stop hang gave me what happend.

Here are the useful results.

• http://erlang.org/pipermail/erlang-questions/2011-June/059694.html
• http://erlang.org/pipermail/erlang-bugs/2013-May/003587.html
• Riak sometimes does not fully shutdown basho/riak#330

In short,
If there is some applications that forgot to add the dependencies to kernel, stdlib,
There is a possibility that kernel, stdlib got unloaded before unloading its app and result in deadlocked.

As with the Riak,
LeoFS have a dependency to bear which doesn't declare the dependencies to kernel, stdlib.
This is the root cause.

@yosukehara do we apply the same method with Riak? ( fork bear and modify bear.app.src to add dependencies to kernel, stdlib) or PR to boundary?

[mocchira]

So this could happend on leo_(gateway|storage|manager).

We will go forward with the Riak way for now.

[yosukehara]

We've fixed this issue and change the depending library as follows, which will be included in LeoFS v1.2.8.
https://github.com/leo-project/savanna_commons/blob/develop/rebar.config#L28

[yosukehara]

I've found the root cause which is ranch's processes cannot stop w/observer and a script.