[leo_storage] many message queues but no queues of mq-stats result

[ooeyoshinori]

There is no queue which is running,
but there is many message queues on queue directory.

Environment

OS : CentOS 6.3
LeoFS Version : 1.2.18
LeoFS Cluster : 4x Storage, 2x Gateway
Machine : Virtual machine

SPEC

leo_gateway
CPU : 2.40GHz X 2 Cores
Memory : 8GB
Disk : 26GB

leo_storage
CPU : 2.40GHz X 1 Cores
Memory : 4GB
Disk : 296G

mq-stats results

[leofs@leo_manager101 ~/current]$ leofs-adm mq-stats leo_storage101@100.73.xxx.xxx
              id                |    state    | number of msgs | batch of msgs  |    interval    |            description
--------------------------------+-------------+----------------|----------------|----------------|-----------------------------------
 leo_delete_dir_queue           |   idling    | 0              | 1600           | 500            | delete directories
 leo_comp_meta_with_dc_queue    |   idling    | 0              | 1600           | 500            | compare metadata w/remote-node
 leo_sync_obj_with_dc_queue     |   idling    | 0              | 1600           | 500            | sync objs w/remote-node
 leo_recovery_node_queue        |   idling    | 0              | 1600           | 500            | recovery objs of node
 leo_async_deletion_queue       |   idling    | 0              | 1600           | 500            | async deletion of objs
 leo_rebalance_queue            |   idling    | 0              | 1600           | 500            | rebalance objs
 leo_sync_by_vnode_id_queue     |   idling    | 0              | 1600           | 500            | sync objs by vnode-id
 leo_per_object_queue           |   idling    | 0              | 1600           | 500            | recover inconsistent objs

Disk usage result

[leofs@leo_storage101 message0]$ pwd
/usr/local/leofs/1.2.18/leo_storage/work/queue/
[leofs@leo_storage101 queue]$ du -hs ./*
98M     ./1
72K     ./2
72K     ./3
12G     ./4 <<<<---- Here is many queue messages
72K     ./5
72K     ./6
72K     ./7
72K     ./8
140K    ./membership

[leofs@leo_storage101 message0]$ pwd
/usr/local/leofs/1.2.18/leo_storage/work/queue/4/message0
[leofs@leo_storage101 message0]$ ls -ltrh
-rw------- 1 leofs leofs   44 Oct 27 02:25 bitcask.write.lock
-rw-rw-r-- 1 leofs leofs 564M Nov  4 09:47 6.bitcask.hint
-rw-rw-r-- 1 leofs leofs 926M Nov  4 09:47 6.bitcask.data

so many warning messages on error file as below

[W]     leo_storage104@100.73.xxx.xxx   2016-11-04 01:04:34.467496 +0900        1478189074      leo_storage_handler_object:replicate_fun/3      1087    [{cause,"locked obj-conatainer"}]
[W]     leo_storage104@100.73.xxx.xxx   2016-11-04 01:04:34.496388 +0900        1478189074      leo_storage_replicator:replicate_fun/2  232     [{key,<<"photos/baseballking/baseballking.jpg">>},{node,local},{req_id,81511474},{cause,unavailable}]
[W]     leo_storage104@100.73.xxx.xxx   2016-11-04 01:04:34.972646 +0900        1478189074      leo_storage_handler_object:replicate_fun/3      1087    [{cause,"locked obj-conatainer"}]
[W]     leo_storage104@100.73.xxx.xxx   2016-11-04 01:04:35.477565 +0900        1478189075      leo_storage_handler_object:replicate_fun/3      1087    [{cause,"locked obj-conatainer"}]
[W]     leo_storage104@100.73.xxx.xxx   2016-11-04 01:04:35.599180 +0900        1478189075      leo_storage_replicator:replicate_fun/2  232     [{key,<<"photos/baseballking/baseballking.jpg">>},{node,local},{req_id,0},{cause,unavailable}]

erl_crash.dump

(No erl_crash.dump on leo_gateway and leo_storage)

Procedure of the operation

1. execute compaction
2. try to delete file during compaction running.

[ooeyoshinori]

leo_manager_0 setting (Click to expand)

##======================================================================
## LeoFS - Manager Configuration (MASTER)
##
## See: http://leo-project.net/leofs/docs/configuration/configuration_1.html
##======================================================================
## --------------------------------------------------------------------
## SASL
## --------------------------------------------------------------------
## See: http://www.erlang.org/doc/man/sasl_app.html
##
## The following configuration parameters are defined for
## the SASL application. See app(4) for more information
## about configuration parameters

## SASL error log path
## sasl.sasl_error_log = ./log/sasl/sasl-error.log

## Restricts the error logging performed by the specified sasl_error_logger
## to error reports, progress reports, or both.
## errlog_type = [error | progress | all]
## sasl.errlog_type = error

## Specifies in which directory the files are stored.
## If this parameter is undefined or false, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_dir = ./log/sasl

## Specifies how large each individual file can be.
## If this parameter is undefined, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_maxbytes = 10485760

## Specifies how many files are used.
## If this parameter is undefined, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_maxfiles = 5

## --------------------------------------------------------------------
## MANAGER
## --------------------------------------------------------------------
## Partner of manager's alias
manager.partner = leo_manager102@100.67.xxx.xxx

## Manager-console accepatable port number
console.port.cui  = 10010
console.port.json = 10020

## Manager-console's number of acceptors
console.acceptors.cui = 3
console.acceptors.json = 16

## # of histories to display at once
console.histories.num_of_display = 200

## --------------------------------------------------------------------
## MANAGER - System
##     * Only set its configurations to **Manager-master**
## --------------------------------------------------------------------
## DC Id
system.dc_id = dc_1

## Cluster Id
system.cluster_id = leofs_1

## --------------------------------------------------------------------
## MANAGER - Consistency Level
##     * Only set its configurations to **Manager-master**
##     * See: http://www.leofs.org/docs/configuration.html#the-consistency-level
## --------------------------------------------------------------------
## A number of replicas
consistency.num_of_replicas = 2

## A number of replicas needed for a successful WRITE operation
consistency.write = 1

## A number of replicas needed for a successful READ operation
consistency.read = 1

## A number of replicas needed for a successful DELETE operation
consistency.delete = 1

## A number of rack-aware replicas
consistency.rack_aware_replicas = 0


## --------------------------------------------------------------------
## MANAGER - Multi DataCenter Settings
## --------------------------------------------------------------------
## A number of replication targets
## mdc_replication.max_targets = 2

## A number of replicas a DC
## mdc_replication.num_of_replicas_a_dc = 1

## --------------------------------------------------------------------
## MANAGER - Mnesia
##     * Store the info storage-cluster and the info of gateway(s)
##     * Store the RING and the command histories
## --------------------------------------------------------------------
## Mnesia dir
mnesia.dir = ./work/mnesia/127.0.0.1

## The write threshold for transaction log dumps
## as the number of writes to the transaction log
mnesia.dump_log_write_threshold = 50000

## Controls how often disc_copies tables are dumped from memory
mnesia.dc_dump_limit = 40


## --------------------------------------------------------------------
## MANAGER - Log
## --------------------------------------------------------------------
## Log level: [0:debug, 1:info, 2:warn, 3:error]
## log.log_level = 1

## Output log file(s) - Erlang's log
## log.erlang = ./log/erlang

## Output log file(s) - app
## log.app = ./log/app

## Output log file(s) - members of storage-cluster
## log.member_dir = ./log/ring

## Output log file(s) - ring
## log.ring_dir = ./log/ring


## --------------------------------------------------------------------
## MANAGER - Other Directories
## --------------------------------------------------------------------
## Directory of queue for monitoring "RING"
## queue_dir = ./work/queue

## Directory of SNMP agent configuration
## snmp_agent = ./snmp/snmpa_manager_0/LEO-MANAGER


## --------------------------------------------------------------------
## RPC
## --------------------------------------------------------------------
## RPC-Server's acceptors
rpc.server.acceptors = 16

## RPC-Server's listening port number
rpc.server.listen_port = 13075

## RPC-Server's listening timeout
rpc.server.listen_timeout = 5000

## RPC-Client's size of connection pool
rpc.client.connection_pool_size = 16

## RPC-Client's size of connection buffer
rpc.client.connection_buffer_size = 16


## --------------------------------------------------------------------
## Other Libs
## --------------------------------------------------------------------
## Enable profiler - leo_backend_db
## leo_backend_db.profile = false

## Enable profiler - leo_logger
## leo_logger.profile = false

## Enable profiler - leo_mq
## leo_mq.profile = false

## Enable profiler - leo_redundant_manager
## leo_redundant_manager.profile = false

## Enable profiler - leo_statistics
## leo_statistics.profile = false

##======================================================================
## For vm.args
##======================================================================
## Name of the LeoFS's manager node
nodename = leo_manager101@100.67.xxx.xxx

## Cookie for distributed node communication.  All nodes in the same cluster
## should use the same cookie or they will not be able to communicate.
distributed_cookie = 00112233

## Enable kernel poll
erlang.kernel_poll = true

## Number of async threads
erlang.asyc_threads = 32

## Increase number of concurrent ports/sockets
erlang.max_ports = 64000

## Set the location of crash dumps
erlang.crash_dump = ./log/erl_crash.dump

## Raise the ETS table limit
erlang.max_ets_tables = 256000

## Raise the default erlang process limit
process_limit = 1048576

## Path of SNMP-agent configuration
## snmp_conf = ./snmp/snmpa_manager_0/leo_manager_snmp

leo_gateway setting (Click to expand)

##======================================================================
## LeoFS - Gateway Configuration
##
## See: http://leo-project.net/leofs/docs/configuration/configuration_3.html
##======================================================================
## --------------------------------------------------------------------
## SASL
## --------------------------------------------------------------------
## See: http://www.erlang.org/doc/man/sasl_app.html
##
## The following configuration parameters are defined for
## the SASL application. See app(4) for more information
## about configuration parameters

## SASL error log path
## sasl.sasl_error_log = ./log/sasl/sasl-error.log

## Restricts the error logging performed by the specified sasl_error_logger
## to error reports, progress reports, or both.
## errlog_type = [error | progress | all]
## sasl.errlog_type = error

## Specifies in which directory the files are stored.
## If this parameter is undefined or false, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_dir = ./log/sasl

## Specifies how large each individual file can be.
## If this parameter is undefined, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_maxbytes = 10485760

## Specifies how many files are used.
## If this parameter is undefined, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_maxfiles = 5

## --------------------------------------------------------------------
## Manager's Node(s)
## --------------------------------------------------------------------
## Name of Manager node(s)
managers = [leo_manager101@100.67.xxx.xxx, leo_manager102@100.67.xxx.xxx]


## --------------------------------------------------------------------
## GATEWAY Protocol
## --------------------------------------------------------------------
## Gateway Protocol to use: [s3 | rest | embed | nfs]
protocol = s3


## --------------------------------------------------------------------
## GATEWAY - HTTP-related configurations
## --------------------------------------------------------------------
## Port number the Gateway uses for HTTP connections
http.port = 8080

## Numbers of processes listening for connections
http.num_of_acceptors = 128

## Maximum number of requests allowed in a single keep-alive session
http.max_keepalive = 4096

## Maximum number of virtual directories
## http.layer_of_dirs = 12

## Port number the Gateway uses for HTTPS connections
http.ssl_port = 9443

## SSL Certificate file
## http.ssl_certfile = ./etc/server_cert.pem

## SSL key
## http.ssl_keyfile = ./etc/server_key.pem

## HTTP custom header configuration file path
## http.headers_config_file = ./etc/http_custom_header.conf

## HTTP timeout for reading header
## http.timeout_for_header = 5000

## HTTP timeout for reading body
## http.timeout_for_body = 15000

## Synchronized time of a bucket property (second)
bucket_prop_sync_interval = 300

## --------------------------------------------------------------------
## GATEWAY - NFS-related configurations
## --------------------------------------------------------------------
## Mountd's port number
## nfs.mountd.port = 22050

## Mountd's the number of acceptors
## nfs.mountd.acceptors = 128

## NFSd's port number
## nfs.nfsd.port = 2049

## NFSd's the number of acceptors
## nfs.nfsd.acceptors = 128

## --------------------------------------------------------------------
## GATEWAY - Large Object
## --------------------------------------------------------------------
## Total number of chunked objects
large_object.max_chunked_objs = 1000

## Length of a chunked object
large_object.chunked_obj_len = 5242880

## Threshold of length of a chunked object
large_object.threshold_of_chunk_len = 5767168

## Reading length of a chuncked object
##   * If happening timeout when copying a large object,
##     you will solve to set this value as less than 5MB.
##   * default: "large_object.chunked_obj_len" (5242880 - 5MB)
large_object.reading_chunked_obj_len = 5242880


## --------------------------------------------------------------------
## GATEWAY - Cache
## --------------------------------------------------------------------
## If this parameter is 'true', Gateway turns on HTTP-based cache server, like Varnish OR Squid.
## If this parameter is 'false', Stores objects into the Gateway’s memory.
## When operating READ, the Etag of the cache is compared with a backend storage’s Etag.
#cache.http_cache = true


## A number of cache workers
## cache.cache_workers = 16

## Memory cache capacity in bytes
cache.cache_ram_capacity = 10737418240

## Disk cache capacity in bytes
# cache.cache_disc_capacity = 0
cache.cache_disc_capacity = 10737418240

## When the length of the object exceeds this value, store the object on disk
# cache.cache_disc_threshold_len = 1048576
cache.cache_disc_threshold_len = 8388608

## Directory for the disk cache data
cache.cache_disc_dir_data = ./cache/data

## Directory for the disk cache journal
cache.cache_disc_dir_journal = ./cache/journal

## Cache Expire in seconds
cache.cache_expire = 300

## Cache Max Content Length in bytes
#cache.cache_max_content_len = 1048576
cache.cache_max_content_len = 2147483648

## Cache Content Type(s)
## In case of "empty", all objects are cached.
## cache.cachable_content_type =

## Cache Path Pattern(s) (regular expression)
## In case of "empty", all objects are cached.
## cache.cachable_path_pattern =


## --------------------------------------------------------------------
## GATEWAY - Watchdog
## --------------------------------------------------------------------
##
##  Watchdog.REX(RPC)
##
## Is rex -watchdog enabled - default:false
watchdog.rex.is_enabled = true

## rex - watch interval - default:5sec
## watchdog.rex.interval = 5

## Threshold memory capacity of binary for rex(rpc) - default:32MB
watchdog.rex.threshold_mem_capacity = 33554432


##
##  Watchdog.CPU
##
## Is cpu-watchdog enabled - default:false
watchdog.cpu.is_enabled = true

## cpu - raised error times
## watchdog.cpu.raised_error_times = 3

## cpu - watch interval - default:5sec
## watchdog.cpu.interval = 5

## Threshold CPU load avg for 1min/5min - default:5.0
## watchdog.cpu.threshold_cpu_load_avg = 5.0

## Threshold CPU load util - default:100 = "100%"
## watchdog.cpu.threshold_cpu_util = 100


##
##  Watchdog.Erlang IO
##
## Is io-watchdog enabled - default:false
# watchdog.io.is_enabled = true

## io - watch interval - default:1sec
# watchdog.io.interval = 3

## Threshold input size/sec  - default:128MB/sec
## watchdog.io.threshold_input_per_sec = 134217728

## Threshold output size/sec - default:128MB/sec
## watchdog.io.threshold_output_per_sec = 134217728

## --------------------------------------------------------------------
## GATEWAY - Timeout
## --------------------------------------------------------------------
## Timeout value when requesting to a storage
## 0 to 65,535 bytes
## timeout.level_1 = 5000

## 65,535 to 131,071 bytes
## timeout.level_2 = 7000

## 131,072 to 524,287 bytes
## timeout.level_3 = 10000

## 524,288 to 1,048,576 bytes
## timeout.level_4 = 20000

## 1,048,576 bytes and over
## timeout.level_5 = 30000

## Timeout value when requesting a GET to a storage
## timeout.get = 30000

## Timeout value when requesting a LS(find_by_parent_dir) to a storage
## timeout.ls = 30000


## --------------------------------------------------------------------
## GATEWAY - Log
## --------------------------------------------------------------------
##
## Log level: [0:debug, 1:info, 2:warn, 3:error]
## log.log_level = 1

## Is enable access-log [true, false]
log.is_enable_access_log = true

## Output log file(s) - Erlang's log
## log.erlang = ./log/erlang

## Output log file(s) - app
## log.app = ./log/app

## Output log file(s) - members of storage-cluster
## log.member_dir = ./log/ring

## Output log file(s) - ring
## log.ring_dir = ./log/ring

## --------------------------------------------------------------------
## GATEWAY - Other Directories
## --------------------------------------------------------------------
## Directory of queue for monitoring "RING"
## queue_dir = ./work/queue

## Directory of SNMP agent configuration
## snmp_agent = ./snmp/snmpa_gateway_0/LEO-GATEWAY


## --------------------------------------------------------------------
## QoS
## --------------------------------------------------------------------
## Enable QoS for statistics
## qos.stat.is_enable = false

## Enable QoS for notification
## qos.notify.is_enable = false

## Savanna Manager's nodes
## qos.managers = [savanna_leo_manager101@100.67.xxx.xxx, savanna_leo_manager102@100.67.xxx.xxx]


## --------------------------------------------------------------------
## Other Libs
## --------------------------------------------------------------------
## Enable profiler - leo_backend_db
## leo_backend_db.profile = false

## Enable profiler - leo_logger
## leo_logger.profile = false

## Enable profiler - leo_mq
## leo_mq.profile = false

## Enable profiler - leo_redundant_manager
## leo_redundant_manager.profile = false

## Enable profiler - leo_statistics
## leo_statistics.profile = false


##======================================================================
## For vm.args
##======================================================================
## Name of the leofs-gateway node
nodename = leo_gateway102@100.67.xxx.xxx

## Cookie for distributed node communication.  All nodes in the same cluster
## should use the same cookie or they will not be able to communicate.
distributed_cookie = 00112233

## Enable kernel poll
erlang.kernel_poll = true

## Number of async threads
erlang.asyc_threads = 32

## Increase number of concurrent ports/sockets
erlang.max_ports = 64000

## Set the location of crash dumps
erlang.crash_dump = ./log/erl_crash.dump

## Raise the ETS table limit
erlang.max_ets_tables = 256000

## Enable SMP
erlang.smp = enable

## Erlang scheduler's compaction of load
erlang.schedulers.compaction_of_load = true

## Erlang scheduler's balancing of load
erlang.schedulers.utilization_balancing = false

## Sender-side network distribution buffer size
## - default 32MB (32768KB)
erlang.distribution_buffer_size = 32768

## A non-negative integer which indicates
## how many times generational garbage collections
## can be done without forcing a fullsweep collection
erlang.fullsweep_after = 0

## Enable eager check I/O
erlang.secio = true

## Raise the default erlang process limit
process_limit = 1048576

## Path of SNMP-agent configuration
## snmp_conf = ./snmp/snmpa_gateway_0/leo_gateway_snmp

leo_storage setting (Click to expand)

##======================================================================
## LeoFS - Storage Configuration
##
## See: http://leo-project.net/leofs/docs/configuration/configuration_2.html
##======================================================================
## --------------------------------------------------------------------
## SASL
## --------------------------------------------------------------------
## See: http://www.erlang.org/doc/man/sasl_app.html
##
## The following configuration parameters are defined for
## the SASL application. See app(4) for more information
## about configuration parameters

## SASL error log path
## sasl.sasl_error_log = ./log/sasl/sasl-error.log

## Restricts the error logging performed by the specified sasl_error_logger
## to error reports, progress reports, or both.
## errlog_type = [error | progress | all]
## sasl.errlog_type = error

## Specifies in which directory the files are stored.
## If this parameter is undefined or false, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_dir = ./log/sasl

## Specifies how large each individual file can be.
## If this parameter is undefined, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_maxbytes = 10485760

## Specifies how many files are used.
## If this parameter is undefined, the error_logger_mf_h is not installed.
## sasl.error_logger_mf_maxfiles = 5

## --------------------------------------------------------------------
## Manager's Node(s)
## --------------------------------------------------------------------
## Name of Manager node(s)
managers = [leo_manager101@100.67.xxx.xxx, leo_manager102@100.67.xxx.xxx]

## --------------------------------------------------------------------
## STORAGE
## --------------------------------------------------------------------
## Object container
obj_containers.path = [/leofs]
obj_containers.num_of_containers = [64]

## e.g. Case of plural pathes
## obj_containers.path = [/var/leofs/avs/1, /var/leofs/avs/2]
## obj_containers.num_of_containers = [32, 64]

## Metadata Storage: [bitcask, leveldb] - default:leveldb
## obj_containers.metadata_storage = leveldb

## A number of virtual-nodes for the redundant-manager
## num_of_vnodes = 168

## Enable strict check between checksum of a metadata and checksum of an object
## - default:false
## object_storage.is_strict_check = false

## Threshold of slow processing (msec)
## - default:1000
## object_storage.threshold_of_slow_processing

## --------------------------------------------------------------------
## STORAGE - Watchdog
## --------------------------------------------------------------------
##
##  Watchdog.REX(RPC)
##
## Is rex-watchdog enabled - default:false
watchdog.rex.is_enabled = false

## rex - watch interval - default:5sec
## watchdog.rex.interval = 5

## Threshold memory capacity of binary for rex(rpc) - default:32MB
watchdog.rex.threshold_mem_capacity = 33554432

##
##  Watchdog.CPU
##
## Is cpu-watchdog enabled - default:false
watchdog.cpu.is_enabled = true

## cpu - raised error times
## watchdog.cpu.raised_error_times = 3

## cpu - watch interval - default:5sec
watchdog.cpu.interval = 5

## Threshold CPU load avg for 1min/5min - default:5.0
## watchdog.cpu.threshold_cpu_load_avg = 5.0

## Threshold CPU load util - default:100 = "100%"
## watchdog.cpu.threshold_cpu_util = 100


##
##  Watchdog.IO
##
## Is io-watchdog enabled - default:false
watchdog.io.is_enabled = false

## io - watch interval - default:1sec
watchdog.io.interval = 5

## Threshold input size/sec  - default:134217728(B) - 128MB/sec
## watchdog.io.threshold_input_per_sec = 134217728

## Threshold output size/sec - default:134217728(B) - 128MB/sec
## watchdog.io.threshold_output_per_sec = 134217728


##
##  Watchdog.DISK
##
## Is disk-watchdog enabled - default:false
watchdog.disk.is_enabled = false

## disk - raised error times
## watchdog.disk.raised_error_times = 3

## disk - watch interval - default:1sec
watchdog.disk.interval = 5

## Threshold disk use% - defalut:85%
## watchdog.disk.threshold_disk_use = 85

## Threshold disk util% - defalut:90%
## watchdog.disk.threshold_disk_util = 90

## Threshold disk read kb/sec - defalut:98304(KB)
## watchdog.disk.threshold_disk_rkb = 98304

## Threshold disk write kb/sec - defalut:98304(KB)
## watchdog.disk.threshold_disk_wkb = 98304

## disk target devices for checking disk utilization
## watchdog.disk.target_devices = []

##
##  Watchdog.Cluster
##
## Is cluster-watchdog enabled - default:false
watchdog.cluster.is_enabled = true

## cluster - watch interval - default:10sec
## watchdog.cluster.interval = 10

## --------------------------------------------------------------------
## STORAGE - Autonomic Operation
## --------------------------------------------------------------------
## [compaction] enabled compaction? - default:false
autonomic_op.compaction.is_enabled = true

## [compaction] number of parallel procs - default:1
## autonomic_op.compaction.parallel_procs = 1

## [compaction] execution intarval time(sec) - default:300(sec)
## autonomic_op.compaction.interval = 300

## [compaction] warning ratio of active size - default:70%
## autonomic_op.compaction.warn_active_size_ratio = 70

## [compaction] threshold ratio of active size - default:60%
## autonomic_op.compaction.threshold_active_size_ratio = 60


## --------------------------------------------------------------------
## STORAGE - Data Compcation
## --------------------------------------------------------------------
## Limit of a number of procs to execute data-compaction in parallel
## compaction.limit_num_of_compaction_procs = 4

## Minimum value of compaction-proc waiting time/batch-proc(msec)
## compaction.waiting_time_min = 100

## Regular value of compaction-proc waiting time/batch-proc(msec)
## compaction.waiting_time_regular = 300

## Maximum value of compaction-proc waiting time/batch-proc(msec)
## compaction.waiting_time_max = 1000

## Step of compaction-proc waiting time(msec)
## compaction.waiting_time_step = 100


## Minimum compaction batch processes
## compaction.batch_procs_min = 1000

## Regular compaction batch processes
## compaction.batch_procs_regular = 10000

## Maximum compaction batch processes
## compaction.batch_procs_max = 100000

## Step compaction batch processes
## compaction.batch_procs_step = 1000


## --------------------------------------------------------------------
## STORAGE - MQ
## --------------------------------------------------------------------
## MQ backend storage: [bitcask, leveldb] - default:bitcask
## mq.backend_db = bitcask

## A number of mq-server's processes
## mq.num_of_mq_procs = 8

##
## [Number of bach processes of message]
##
## Minimum number of bach processes of message
mq.num_of_batch_process_min = 100

## Maximum number of bach processes of message
mq.num_of_batch_process_max = 10000

## Regular value of bach processes of message
mq.num_of_batch_process_regular = 1000

## Step value of bach processes of message
mq.num_of_batch_process_step = 250


##
## [Interval beween batch-procs]
##
## Minimum value of interval beween batch-procs(msec)
mq.interval_between_batch_procs_min = 10

## Maximum value of interval between batch-procs(msec)
mq.interval_between_batch_procs_max = 1000

## Regular value of interval between batch-procs(msec)
mq.interval_between_batch_procs_regular = 300

## Step value of interval between batch-procs(msec)
mq.interval_between_batch_procs_step = 100


## --------------------------------------------------------------------
## STORAGE - Replication/Recovery object(s)
## --------------------------------------------------------------------
## Rack-id for the rack-awareness replica placement
## replication.rack_awareness.rack_id =

## Size of stacked objects (bytes)
## replication.recovery.size_of_stacked_objs = 33554432

## Stacking timeout (sec)
## replication.recovery.stacking_timeout = 5


## --------------------------------------------------------------------
## STORAGE - MDC Replication
## --------------------------------------------------------------------
## Size of stacked objects (bytes) - default:32MB
mdc_replication.size_of_stacked_objs = 33554432

## Stacking timeout (sec)
mdc_replication.stacking_timeout = 30

## Request timeout (msec)
mdc_replication.req_timeout = 30000

## Number of stacking procs
mdc_replication.stacking_procs = 1


## --------------------------------------------------------------------
## STORAGE - Log
## --------------------------------------------------------------------
## Log level: [0:debug, 1:info, 2:warn, 3:error]
## log.log_level = 1

## Output log file(s) - Erlang's log
## log.erlang = ./log/erlang

## Output log file(s) - app
## log.app = ./log/app

## Output log file(s) - members of storage-cluster
## log.member_dir = ./log/ring

## Output log file(s) - ring
## log.ring_dir = ./log/ring

## Output data-diagnosis log
log.is_enable_diagnosis_log = true

## --------------------------------------------------------------------
## STORAGE - Other Directories
## --------------------------------------------------------------------
## Directory of queue for monitoring "RING"
## queue_dir  = ./work/queue

## Directory of SNMP agent configuration
## snmp_agent = ./snmp/snmpa_storage_0/LEO-STORAGE


## --------------------------------------------------------------------
## Other Libs
## --------------------------------------------------------------------
## Send after interval
## leo_ordning_reda.send_after_interval = 100

## Temporary directory of stacked objects
## leo_ordning_reda.temp_stacked_dir = "work/ord_reda/"

## --------------------------------------------------------------------
## RPC
## --------------------------------------------------------------------
## RPC-Server's acceptors
## this value must be determinted by following logic
## rpc.server.acceptor need to be larger than
## rpc.client.connection_pool(buffer)_size * "# of storage nodes + # of manager nodes in your cluster"
## The default value is suitable for less than 16 nodes in a cluster
## rpc.server.acceptors = 128

## RPC-Server's listening port number
## rpc.server.listen_port = 13077

## RPC-Server's listening timeout
## rpc.server.listen_timeout = 30000

## RPC-Client's size of connection pool
## rpc.client.connection_pool_size = 8

## RPC-Client's size of connection buffer
## rpc.client.connection_buffer_size = 8

## RPC-Client's max number of requests for reconnection to a remote-node
## * 1..<integer>: specialized value
## rpc.client.max_requests_for_reconnection = 64


## --------------------------------------------------------------------
## MANAGER - Mnesia
##     * Store the info storage-cluster and the info of gateway(s)
##     * Store the RING and the command histories
## --------------------------------------------------------------------
## The write threshold for transaction log dumps
## as the number of writes to the transaction log
mnesia.dump_log_write_threshold = 50000

## Controls how often disc_copies tables are dumped from memory
mnesia.dc_dump_limit = 40

## --------------------------------------------------------------------
## Profiling
## --------------------------------------------------------------------
## Enable profiler - leo_backend_db
## leo_backend_db.profile = false

## Enable profiler - leo_logger
## leo_logger.profile = false

## Enable profiler - leo_mq
## leo_mq.profile = false

## Enable profiler - leo_object_storage
## leo_object_storage.profile = false

## Enable profiler - leo_ordning_reda
## leo_ordning_reda.profile = false

## Enable profiler - leo_redundant_manager
## leo_redundant_manager.profile = false

## Enable profiler - leo_rpc
## leo_rpc.profile = false

## Enable profiler - leo_statistics
## leo_statistics.profile = false

##======================================================================
## For vm.args
##======================================================================
## Name of the leofs-storage node
nodename =  leo_storage102@100.67.xxx.xxx
## Cookie for distributed node communication.  All nodes in the same cluster
## should use the same cookie or they will not be able to communicate.
distributed_cookie = 00112233

## Enable kernel poll
erlang.kernel_poll = true

## Number of async threads
erlang.async_threads = 64

## Increase number of concurrent ports/sockets
erlang.max_ports = 64000

## Set the location of crash dumps
erlang.crash_dump = ./log/erl_crash.dump

## Raise the ETS table limit
erlang.max_ets_tables = 256000

## Enable SMP
erlang.smp = enable

## Erlang scheduler's compaction of load
erlang.schedulers.compaction_of_load = true

## Erlang scheduler's balancing of load
erlang.schedulers.utilization_balancing = false

## Sender-side network distribution buffer size
## - default 32MB (32768KB)
erlang.distribution_buffer_size = 32768

## A non-negative integer which indicates
## how many times generational garbage collections
## can be done without forcing a fullsweep collection
erlang.fullsweep_after = 0

## Enable/Disable eager check I/O (Erlang 17.4/erts-6.3-, ref:OTP-12117)
erlang.secio = true

## Raise the default erlang process limit
process_limit = 1048576

## Path of SNMP-agent configuration
## snmp_conf = ./snmp/snmpa_storage_0/leo_storage_snmp

[yosukehara]

@ooeyoshinori Thanks for your report.

leo_storage restricts write/delete operation during the data-compaction, so {cause,"locked obj-conatainer"} is happened.

We improved the mq's feature with #437 (1.2.18) and #448 (1.2.20). If you update your leo_straoge configuration of mq.backend_db from bitcask to leveldb, and migrate your LeoFS from 1.2.18 to the latest version, message data is reduced.

In addition, the default configuration of mq.backend_db of v1.2.18 is bitcask.

[ooeyoshinori]

@yosukehara Thank you for your direction.
Let me ask a question about the mq.backend_db.
As you mentioned on #437 (1.2.18) , it is OK to use "leveldb" option on LeoFS ver.1.2.18, right?

[yosukehara]

@ooeyoshinori

it is OK to use "leveldb" option on LeoFS ver.1.2.18, right?

Right, and v1.2.20 or later is better than v1.2.18 because of #448 .

[ooeyoshinori]

@yosukehara
I have configured mq.backend_db from "bitcask" to "leveldb" on LeoFS v1.2.18. And I watch the efficiency for a while.
Is it much better to use "leveldb" option on v1.2.18 even to using v1.2.20 to resolve disk usage situation?

==Better==
"leveldb" on v1.2.20 or later
"leveldb" on v1.2.18 <<<<---- I'm here.
"bitcask" on v1.2.18
==Worse==

[yosukehara]

@ooeyoshinori The disk usage situation was fixed with v1.2.20.

[ooeyoshinori]

@yosukehara Now it works well with "leveldb" setting for mq.backend_db.
Thank you for your suggestion.