Skip to content

expiry_beta1

Jump to bottom
Matthew Von-Maszewski edited this page Jul 5, 2016 · 16 revisions

Expiry Beta 1 - Fundamental Features

Status

  • beta period ended -
  • announced and available -
  • preparation - July 5, 2016

Feature Set

This beta release contains the most basic, most rudimentary set of object expiry features. The release is for not for production. There exists the possibility that this new code could unintentionally delete data records that need to not be deleted. Basho is making this code available for experimental work and preproduction feedback.

Features:

  • global leveldb expiry of key/value objects based upon the last update of the object (write-time expiry)

  • option to erase entire file of expired objects instead of selected removal of expired objects during leveldb compactions (whole file expiry)

  • explicit timestamp expiry that is individualized to each key/value object (explicit-time expiry)

  • leveldb API support for non-Riak applications to set and read both write-time expiry and explicit timestamp expiry information

  • Riak specific: the expiry options apply to all leveldb instances in a multi-backend configuration

  • Riak specific: key/value objects transferred via handoff, multi-datacenter replication, or rebuilt by read repair/AAE have their write-time expiry restarted.

Future Features

  • Riak specific: selection of expiry options based upon Riak Buckets and Bucket Types

  • Riak specific: Riak client access to write time and explicit timestamp metadata supporting expiry

  • Riak specific: logging and/or archiving of expired objects and/or whole files expired

  • Riak specific: write time and explicit expiry metadata propagated by handoff, replication, and repair operations

Known Bugs / Limitations

  • Google's original block cache code does not actively release cached file blocks for closed or deleted .sst table files. This behavior makes leveldb appear to have a memory leak as files quietly expire but memory footprint increases to its configured maximum (configured in leveldb::Options).

Example configurations

  • leveldb C/C++ API users:
    #include "leveldb/options.h"
    #include "leveldb_os/expiry_os.h"

    leveldb::Options opts;
    leveldb::ExpiryPtr_t expiry;  // reference counted pointer

    expiry.assign(new leveldb::ExpiryModuleOS);

    expiry->expiry_enabled=true;
    expiry->expiry_minutes=30;
    expiry->whole_file_expiry=true;

    opts.expiry_module.assign(expiry);
  • Riak users' advanced.config: Note: for beta, advanced.config contains the expiry options. Future releases will have the options within riak.conf like other eleveldb/leveldb options.
    [
        {eleveldb, [
           {expiry_enabled, true},
           {expiry_minutes, 21},
           {whole_file_expiry, true}
        ]}
    ].

Other resources

Prebuilt patches for Riak eleveldb.so

If a package upgrade is not possible in your environment, the LevelDB shared library can be patched.

Do not apply the eleveldb.so patch to Riak TS, it will prevent it functioning correctly.

Installation and Removal Instructions

This patch contains natively compiled code. The eleveldb.so file must be installed to the eleveldb priv directory and cannot be added to basho-patches.

By default, this directory is in the following locations per OS:

  • RHEL/CentOS - /usr/lib64/riak/lib/eleveldb-2.1.10-0-g0537ca9/priv/,
  • Debian/Ubuntu - /usr/lib/riak/lib/eleveldb-2.1.10-0-g0537ca9/priv/,
  • Solaris - /opt/riak/lib/eleveldb-2.1.10-0-g0537ca9/priv/,
  • SmartOS - /opt/local/lib/riak/lib/eleveldb-2.1.10-0-g0537ca9/priv/,
  • FreeBSD - /usr/local/lib/riak/lib/eleveldb-2.1.10-0-g0537ca9/priv/,
  • On other platforms it may be /usr/lib/riak/lib/eleveldb-2.1.10-0-g0537ca9/priv/.

To install this patch, on each node in the cluster you must:

  1. Stop the node: riak stop.
  2. Change to the eleveldb priv directory (similar to /usr/lib64/riak/lib/eleveldb-2.1.10-0-g0537ca9/priv/)
  3. Rename the original leveldb library ( eleveldb.so) to eleveldb.so.orig.
  4. Copy the provided eleveldb.so to the directory and verify correct permissions.
  5. If possible, verify that the md5 sum of the eleveldb.so located in the eleveldb priv directory is correct.
  6. Start the node: riak start.

To back out of this patch, on each node in the cluster you must:

  1. Stop the node: riak stop.
  2. Remove the patched eleveldb.so file from the eleveldb priv directory.
  3. Rename eleveldb.so.orig to eleveldb.so.
  4. Start the node: riak start.

Verifying the Patch Installation

When the patch is installed, the LevelDB LOG files will report that version expiry_beta1 is installed. The LOG files for each running vnode will have a log line similar to the following:

    2016/07/05-18:42:50.544293 7ffaaf3b1700             Version: expiry_beta1
Clone this wiki locally