Skip to content

expiry_beta1

Jump to bottom
Matthew Von-Maszewski edited this page Jul 5, 2016 · 16 revisions

Expiry Beta 1 - Fundamental Features

Status

  • beta period ended -
  • announced and available -
  • preparation - July 5, 2016

Feature Set

This beta release contains the most basic, most rudimentary set of object expiry features. The release is for not for production. There exists the possibility that this new code could unintentionally delete data records that need to not be deleted. Basho is making this code available for experimental work and preproduction feedback.

Features:

  • global leveldb expiry of key/value objects based upon the last update of the object (write-time expiry)

  • option to erase entire file of expired objects instead of selected removal of expired objects during leveldb compactions (whole file expiry)

  • explicit timestamp expiry that is individualized to each key/value object (explicit-time expiry)

  • leveldb API support for non-Riak applications to set and read both write-time expiry and explicit timestamp expiry information

  • Riak specific: the expiry options apply to all leveldb instances in a multi-backend configuration

  • Riak specific: key/value objects transferred via handoff, multi-datacenter replication, or rebuilt by read repair/AAE have their write-time expiry restarted.

Future Features

  • Riak specific: selection of expiry options based upon Riak Buckets and Bucket Types

  • Riak specific: Riak client access to write time and explicit timestamp metadata supporting expiry

  • Riak specific: logging and/or archiving of expired objects and/or whole files expired

  • Riak specific: write time and explicit expiry metadata propagated by handoff, replication, and repair operations

Known Bugs / Limitations

  • Google's original block cache code does not actively release cached file blocks for closed or deleted .sst table files. This behavior makes leveldb appear to have a memory leak as files quietly expire but memory footprint increases to its configured maximum (configured in leveldb::Options).

Example configurations

  • leveldb C/C++ API users:
    #include "leveldb/options.h"
    #include "leveldb_os/expiry_os.h"

    leveldb::Options opts;
    leveldb::ExpiryPtr_t expiry;  // reference counted pointer

    expiry.assign(new leveldb::ExpiryModuleOS);

    expiry->expiry_enabled=true;
    expiry->expiry_minutes=30;
    expiry->whole_file_expiry=true;

    opts.expiry_module.assign(expiry);
  • Riak users' advanced.config: Note: for beta, advanced.config contains the expiry options. Future releases will have the options within riak.conf like other eleveldb/leveldb options.
    [
        {eleveldb, [
           {expiry_enabled, true},
           {expiry_minutes, 21},
           {whole_file_expiry, true}
        ]}
    ].

Other resources

Clone this wiki locally