Skip to content

Latest commit

 

History

History
3092 lines (2769 loc) · 209 KB

CHANGELOG.md

File metadata and controls

3092 lines (2769 loc) · 209 KB
Raw

CHANGELOG

1.3.8 (Nov 22, 2017)

Fixed Bugs

LeoGateway

  • #799 s3cmd 2.0 hangs itself and slows down storage nodes
  • #845 Errors about multipart object parts on storages during upload
  • #848 Gateway returns object that did not change instead of 304 (Not Modified)
  • #851 Unable to remove gateway using DNS name through leofs-adm
  • #854 Create bucket and put object tests fail
  • #884 Omit Content-Length when responding 204
  • #903 Correctly uploaded file was removed during multipart upload
  • #907 Parallel access to same file locked
  • #926 Upload ID can conflict in case multiple clients try to upload a same file at once
  • #937 Content-Range Header is not set in range request reply

LeoManager

  • #914 Old Cluster cannot join cluster from 1.3.3, incompatible system_conf sent

LeoStorage

  • #874 Recover-file doesn't work at all
  • #880 Recover-node fails to recover all data on storage node
  • #881 Read-repairer does not fix all objects

Others

  • #729 leo_logger Tons of logs can get LeoFS overloaded
  • #835 leo_gateway leo_storage LeoFS v1.3.7 - MDC not replicating custom headers

Improvements

  • #841 leo_object_storage Take much time to open with lots of AVS files
  • #858 leo_gateway Special URL to check gateway's health in S3 mode for load balancers
  • #883 leo_storage Increase default open files limit (on Linux)
  • #912 leo_storage Watchdog for slow processing/timeout always enabled

Libraries

Leo Project

Others

1.3.7 (Sep 12, 2017)

Fixed Bugs

  • #592 leo_storage data-compaction Avoids write operation before leo_storage is not able to execute data-compaction
  • #816 leo_manager leo_storage mdc-replication LeoFS 1.3.6 - MDC not replicating data
  • #817 leo_manager leofs-adm "version all" and "status" hang when node is offline
  • #821 leo_storage PR #802 break listing objects
  • #827 leofs_package Nodes fail to start on Ubuntu 16.04 (different sudo setup)

Libraries

Leo Project

Others

1.3.6 (Aug 30, 2017)

Fixed Bugs

  • #776 leo_watchdog leo_storage Continuous warnings caused by disk usage make leo_mq stop
  • #803 leo_storage Much higher amount of messages during deletion of two buckets at once compared to deleting them one after another
  • #804 leo_storage delete-bucket The state change from enqueuing to monitoring could take much time

Libraries

Leo Project

Others

1.3.5 (Aug 10, 2017)

Add two leofs-adm commands which relate to LeoFS' bucket operation

LeoGateway

NFS

  • #608 Set FSF_HOMOGENEOUS bit to the properties
  • #609 Set ffiles and afiles in the response to fsstat to MAX

Others

  • #730 Respond 403 when a multipart upload complete request failed
  • #735 Abort multipart upload can respond 204 even if the operation failed
  • #769 Abort multipart upload may not delete parts objects
  • #770 Remove the code putting a zero byte object when handling abort MPU
  • #784 The source object get deleted when x-amz-metadata-directive is set to REPLACE

LeoStorage

  • #725 Deleting bucket eventually fails and makes delete queues stuck
    • Some #725's issues remains to be done with v1.4.0
  • #732 leo_mq v1.3.3 Items in QUEUE_ID_ASYNC_DELETION can keep existing
  • #754 delete-bucket can not work in case the objects were created on LeoFS <= v1.3.2.1
  • #783 Consistency Problem with asynchrpnous deletion

LeoManager

  • #150 s3-api Implement a robust delete_bucket
  • #556 mnesia Restarting slave may fail while master is down

Commons

  • #653 Use timer:send_after/3 for the periodic action in gen_server
  • #731 leo_backend_db The number mq-stats displays can be different from the number leo_backend_db actually stores #731
  • #744 leo_mq Timeout in return values are not needed at handle_call, handle_cast in leo_mq_server
  • #746 leo_mq Cache the result of leo_backend_db_api:status/0 to reduce call times
  • #751 leo_watchdog LeoWatchdog for disk could make leo_backend_db overloaded
  • #761 leo_backend_db Revision of dependency libraries
  • #774 elarm records in ets may leak

Libraries

Leo Project

Others

1.3.4 (May 3, 2017)

LeoStorage

  • #644 [Bug] Fixed "badarg" from watchdog on storage node in error log
    • Fixes for #644 did not include in v1.3.3 because leo_wachdog's version was wrong.

Libraries

Leo Project

Others

1.3.3 (Apr 28, 2017)

LeoStorage

  • #573 [Improve] To be configurable the data synchronization
  • #590 [Improve] API to break a data block in AVS
  • #690 [Improve] Track Slow I/O on leo_compact_fsm_worker
  • #593 [Bug] Fixed that reduce not_found logs
  • #624 [Bug] Fixed that messages in a queue can get lost
  • #645 [Bug][MDC-Replication] read-repair, data-compaction, data-recovery feature may replicate objects more than mdc_replication.num_of_replicas_a_dc
  • #641 [Bug] Fixed that MDC replicated objects are missing HTTP headers after upgrading from 1.3.0 to 1.3.2.1
  • #644 [Bug] Fixed "badarg" from watchdog on storage node in error log
  • #668 [Bug] Fixed that leo_storage can be rebooted when lots of badarg from eleveldb happened
  • #704 [Bug] Fixed that possibility of MQ's data loss
  • #713 [Bug] Fixed that replicator/5 can receive messages generated by gen_server:call/cast
  • #660 [Maintenance] Needs to deliver moving AVS directory after launching

LeoGateway

NFS

  • #327 [Improve] Return file attributes on every operation
  • #598 [Improve] Make the response to fsinfo configurable
  • #613 [Improve] nfs_fh3 is not needed to be true random
  • #614 [Improve] Avoid concating list in leo_nfs_file_handler:list_dir
  • #615 [Improve] large_obj_delete_chunks should keep iterating even some error happened
  • #596 [Bug] Fixed that the total file size can be wrong when writing the same file in parallel
  • #599 [Bug] Fixed that some responses don't comply rfc1813
  • #604 [Bug] Fixed that NFS_DUMMY_FILE4S3DIR should be replaced with one including reserved characters
  • #611 [Bug] Fixed that miss error handling in nfsproc3_rmdir_3
  • #612 [Bug] Fixed that readdir entries can leak when NFS client crash

S3-API

  • #483 [Bug] Fixed that S3 Sync feature does not synchronize directories properly
  • #642 [Bug] Fixed that failed to fetch a bucket info while both managers are down after bucket_prop_sync_interval passed
  • #647 [Bug] Fixed that make find_bucket_by_name fail-safe as #642

LeoManager

  • #695 [Bug] Fixed that degrading the get-bucket command
  • #606 [Improve] Able to create a user with access-key and secret-access-key
  • #607 [Improve] Make a user password more secure

Commons

  • #626 [Bug] Fixed that missing error handlings of leo_ordning_reda
  • #643 [Bug] Fixed that a way to remotely check system_version on nodes with the status command
  • #712 [Bug] Fixed that iterator-handler can be leaked
  • #667 [Improve] Set ERL_CRASH_DUMP_SECONDS to a negative value
  • #671 [Improve][leo_backend_db] Implement status callback for eleveldb
  • #617 [Package] /usr/local/bin/leofs-adm should be removed
  • #638 [Package] Running packaged version as non-privileged user

Libraries

Leo Project

Others

1.3.2 (Jan 30, 2017)

1.3.1 (Dec 28, 2016)

1.3.0 (Aug 9, 2016)

1.2.22 (May 20, 2016)

1.2.21 (March 29, 2016)

1.2.20 (March 23, 2016)

  • Improvements
    • #389 erlang To be able to build LeoFS with Erlang 18
    • #442 leo_storage When removing a large size object, a high-latency issue can be happened
    • #445 leo_backend_db May take too much time when restarting leo_storage
    • #446 leo_object_storage Unnecessary a file:position call when opening a AVS file
    • #447 There are some possibilities that some supervisours don't restart their child as we expect
    • #448 leo_storage,leo_mq Reduce queue messages
    • Improve the LeoStorage's read performance
      • #450 leo_object_storage Improve concurrency
      • #467 leo_object_storage Improve concurrency of read operations
    • leo_manager Implement the command history feature to be able to confirm command histories with it
      • It is stored under log/app directory
## command history file:
$ tree -vhD /path/to/leofs/leofs-1.2.20/leo_manager_0/log/app/
./package/leo_manager_0/log/app/
├── [ 103 Mar 17 16:40]  cmd_history -> /path/to/leofs/leofs-1.2.20/leo_manager_0/log/app/cmd_history.20160317.16.1
├── [ 230 Mar 17 16:41]  cmd_history.20160317.16.1
├── [  97 Mar 17 16:40]  error -> /path/to/leofs/leofs-1.2.20/leo_manager_0/log/app/error.20160317.16.1
├── [ 256 Mar 17 16:41]  error.20160317.16.1
├── [  96 Mar 17 16:40]  info -> /path/to/leofs/leofs-1.2.20/leo_manager_0/log/app/info.20160317.16.1
└── [1.6K Mar 17 16:40]  info.20160317.16.1

## command history format:
## | <datetime> | <unixtime> | <command> |
$ tail -f /path/to/leofs/leofs-1.2.20/leo_manager_0/log/app/cmd_history
2016-03-17 16:40:42.685451 +0900    1452670842685495    start
2016-03-17 16:41:22.732225 +0900    1452670882732267    update-log-level storage_0@127.0.0.1 debug
2016-03-17 16:41:36.745973 +0900    1452670896746008    update-consistency-level 1 1 1

1.2.18 (January 7, 2016)

1.2.16 (Octorber 9, 2015)

1.2.14 (September 11, 2015)

1.2.12 (July 10, 2015)

1.2.11 (June 30, 2015)

1.2.10 (May 22, 2015)

1.2.9 (May 15, 2015)

1.2.8 (Apr 14, 2015)

1.2.7 (Mar 6, 2015)

1.2.6 (Feb 20, 2015)

1.2.5 (Jan 29, 2015)

1.2.4 (Jan 23, 2015)

1.2.3 (Jan 20, 2015)

1.2.2 (Dec 17, 2014)

  • New Features
    • #117 leo_storage Implemented the auto-compaction mechanism
      • The auto-compaction configuration file for LeoFS Storage, name of which is leo_storage.conf
      • The auto-compaction configuration is as follows:
      ## [compaction] enabled compaction? - default:false
autonomic_op.compaction.is_enabled = true

## [compaction] number of parallel procs - default:1
autonomic_op.compaction.parallel_procs = 1

## [compaction] warning ratio of active size - default:70%
autonomic_op.compaction.warn_active_size_ratio = 70

## [compaction] threshold ratio of active size - default:60%
autonomic_op.compaction.threshold_active_size_ratio = 60
  • Improvements
    • #270 Supported leo_watchdog for FreeBSD and SmartOS
    • #281 leo_gateway Made timeout configurable for the get and the find_by_parent_dir
    • The watchdog mechnism to production status
      • The watchdog configuration file for LeoFS Storage and LeoFS Gateway, name of which are leo_storage.conf and leo_gateway.conf
      • The watchdog configuration for CPU and Disk as follows:
      ##  Watchdog.CPU
##
## Is cpu-watchdog enabled - default:false
watchdog.cpu.is_enabled = true

## cpu - raised error times
watchdog.cpu.raised_error_times = 3

## cpu - watch interval - default:5sec
watchdog.cpu.interval = 5

## Threshold CPU load avg for 1min/5min - default:5.0
watchdog.cpu.threshold_cpu_load_avg = 5.0

## Threshold CPU load util - default:100 = "100%"
watchdog.cpu.threshold_cpu_util = 100

##  Watchdog.DISK
##
## Is disk-watchdog enabled - default:false
watchdog.disk.is_enabled = true

## disk - raised error times
watchdog.disk.raised_error_times = 3

## disk - watch interval - default:1sec
watchdog.disk.interval = 1

## Threshold disk use% - defalut:85%
watchdog.disk.threshold_disk_use = 85

## Threshold disk util% - defalut:100%
watchdog.disk.threshold_disk_util = 100

## Threshold disk read kb/sec - defalut:262144(KB)
watchdog.disk.threshold_disk_rkb = 262144

## Threshold disk write kb/sec - defalut:262144(KB)
watchdog.disk.threshold_disk_wkb = 262144

## disk target devices for checking disk utilization
watchdog.disk.target_devices = []
      • Improved MQ's confurations for LeoFS Stroage in order to be able to control system load
      ## [Number of bach processes of message]
##
## Minimum number of bach processes of message
mq.num_of_batch_process_min = 100

## Maxmim  number of bach processes of message
mq.num_of_batch_process_max = 10000

## Maxmim  number of bach processes of message
mq.num_of_batch_process_regular = 1000

## Maxmim  number of bach processes of message
mq.num_of_batch_process_step = 250

## [Interval beween batch-procs]
##
## Minimum value of interval beween batch-procs(msec)
mq.interval_between_batch_procs_min = 10

## Maximum value of interval between batch-procs(msec)
mq.interval_between_batch_procs_max = 1000

## Regular value of interval between batch-procs(msec)
mq.interval_between_batch_procs_regular = 300

## Step value of interval between batch-procs(msec)
mq.interval_between_batch_procs_step = 100
      • Improved the mq-status command of leofs-adm
      $ leofs-adm mq-stats storage_1@127.0.0.1
              id                |    state    | number of msgs | batch of msgs  |    interval    |            description
--------------------------------+-------------+----------------|----------------|----------------|-----------------------------------
leo_delete_dir_queue            |   idling    | 0              | 1000           | 100            | delete directories
leo_comp_meta_with_dc_queue     |   idling    | 0              | 1000           | 100            | compare metadata w/remote-node
leo_sync_obj_with_dc_queue      |   idling    | 0              | 1000           | 100            | sync objs w/remote-node
leo_recovery_node_queue         |   idling    | 0              | 1000           | 100            | recovery objs of node
leo_async_deletion_queue        |   idling    | 0              | 1000           | 100            | async deletion of objs
leo_rebalance_queue             |   running   | 2167           | 1400           | 10             | rebalance objs
leo_sync_by_vnode_id_queue      |   idling    | 0              | 1000           | 100            | sync objs by vnode-id
leo_per_object_queue            |   idling    | 0              | 1000           | 100            | recover inconsistent objs
  • Fixed Bugs
    • #273 leo_gateway NFS Max file size was hardcoded as 4GB
    • #274 leo_manager leo_storage leo_gateway Crashed a manager node by snmp-related issue with Erlang-17
    • #275 leo_gateawy An error message was output if there is no the http custom header file
    • #277 leo_gateway In REST mode, Putting a large object failed
    • #279 leo_object_storage Compaction status can be inconsistent when an error occured while preparing compaction
    • #282 leo_gateway Gateway's timeout configuration has been ignored
  • Used libraries

1.2.1 (Nov 11, 2014)

1.2.0 (Nov 7, 2014)

1.1.5 (Oct 15, 2014)

1.1.4 (Oct 3, 2014)

1.1.3 (Sep 12, 2014)

1.1.2 (Sep 4, 2014)

1.1.1 (Aug 21, 2014)

1.1.0 (Aug 6, 2014)

1.0.2 (May 9, 2014)

1.0.1 (May 8, 2014)

1.0.0 (Apr 2, 2014)

1.0.0-pre3 (Feb 20, 2014)

1.0.0-pre2 (Jan 22, 2014)

1.0.0-pre1 (Dec 27, 2013)

0.16.8 (Dec 12, 2013)

0.16.5 (Nov 26, 2013)

0.16.0 (Oct 25, 2013)

0.14.9 (Sep 27, 2013)

0.14.8 (Sep 12, 2013)

0.14.7 (Aug 23, 2013)

0.14.6 (July 11, 2013)

0.14.5 (July 8, 2013)

0.14.4 (June 28, 2013)

0.14.3 (June 21, 2013)

0.14.2 (June 7, 2013)

0.14.1-1 (May 8, 2013)

  • Improved
    • Gateway
      • Upgrade Ranch and Cowboy to latest version
  • Fix Bugs
    • Gateway
      • Reply empty response unintentionally when using some s3 clients
        • "leo_gateway" stored a "0-byte length object" into the cache when put an large-object(over 5MB) from clients. So We supported to remove a registered object from the cache.

0.14.1 (Apr 15, 2013)

0.14.0 (Mar 20, 2013)

0.14.0-RC2 (Mar 14, 2013)

0.14.0-RC1 (Mar 5, 2013)

0.12.7 (Dec 26, 2012)

0.12.5 (Dec 7, 2012)

0.12.4 (Nov 21, 2012)

0.12.3 (Nov 9, 2012)

0.12.2 (Nov 2, 2012)

0.12.1 (Oct 25, 2012)

0.12.0 (Oct 20, 2012)

0.10.2 (Sep 25, 2012)

  • Improve performances
    • NOT used "proplists:get_value" function
      • Replace from "proplists:get_value/2,3" to "lists:keyfind/2"
    • Related libs:
      • leo_gateway
      • leo_storage
      • leo_manager
      • leo_logger
      • leo_mq
      • leo_object_storage
      • leo_ordning_reda
      • leo_redundant_manager
      • leo_s3_libs
      • leo_statistics
  • Improve leo_manager
    • Support output of json-format
    • Support multi-ports TCP-server
      • for CUI console
      • for Application (JSON-Format)
  • Improve leo_storage
    • Support plural devices
    • Reduced rebalance/recover costs
      • Support compression of stacked objects and decompresson of received objects (Using snappy)
  • Fix bugs
    • leo-gateway related (S3-related):
      • Create bucket from 'Dragon Disk'
      • When using cowboy can send bad values(not iodata)

0.10.1 (Sep 12, 2012)

  • Improve leo_hex performances
    • "binary_to_hex" function
    • Related libs:
      • leo_gateway
      • leo_object_storage
      • leo_redundant_manager
    • By this correspondence, LeoFS's performance improved 20% up.
  • Improve leo_manager
    • Format of output from manager-console
      • Commands:
        • "status"
        • "whereis"
  • Improve performance of "storage-stats" in manager-console
  • Fix bugs
    • A handling error in S3-libs

0.10.0 (Aug 30, 2012)

  • Improve S3-API's compatibility
    • Add S3-authentication
    • Add S3-bucket
  • Add S3-related command in LeoFS's manager
    • "s3-gen-key" : Generate a S3 key pair(AccessKeyID and SecretAccessKey)
    • "s3-set-endpoint" : Register a new S3 Endpoint
    • "s3-delete-endpoint" : Delete a S3 Endpoint
    • "s3-get-endpoints" : Retrieve all of S3 Endpoints registered
    • "s3-get-buckets" : Retrieve all of Buckets registered
  • Improve order of system launch
    • Remove "attach command" in manager - After Storage launched, the node's state is automatically changed to "attached"
  • Improve rebalance-function's performance which is about 5 times compare with v0.9.1
  • Improve compact-function. Restrain storage's load when compact objects.
  • Fix bugs
    • Deletion of Zero bytes in Storage
    • Behavior after the restart of Manager
    • Re-register procs into the Manager's monitor

0.9.1 (Jul 13, 2012)

  • Remove "apps" directory in leofs - Modified "reltool.config"
  • Fix 'shadow vars'
  • Remove ZMQ log-appender and AMQP log-appender for leo_logger, They will be provided LeoFS's sub projects.
  • Improve - In order to be able to extend "LeoFS's Object Container's file format".

0.9.0 (Jul 4, 2012)

  • Initial release