Skip to content

Basic Operations

Jump to bottom
Siying Dong edited this page Nov 29, 2018 · 72 revisions

Basic operations

The rocksdb library provides a persistent key value store. Keys and values are arbitrary byte arrays. The keys are ordered within the key value store according to a user-specified comparator function.

Opening A Database

A rocksdb database has a name which corresponds to a file system directory. All of the contents of database are stored in this directory. The following example shows how to open a database, creating it if necessary:

  #include <cassert>
  #include "rocksdb/db.h"

  rocksdb::DB* db;
  rocksdb::Options options;
  options.create_if_missing = true;
  rocksdb::Status status = rocksdb::DB::Open(options, "/tmp/testdb", &db);
  assert(status.ok());
  ...

If you want to raise an error if the database already exists, add the following line before the rocksdb::DB::Open call:

  options.error_if_exists = true;

If you are porting code from leveldb to rocksdb, you can convert your leveldb::Options object to a rocksdb::Options object using rocksdb::LevelDBOptions, which has the same functionality as leveldb::Options:

  #include "rocksdb/utilities/leveldb_options.h"

  rocksdb::LevelDBOptions leveldb_options;
  leveldb_options.option1 = value1;
  leveldb_options.option2 = value2;
  ...
  rocksdb::Options options = rocksdb::ConvertOptions(leveldb_options);

RocksDB Options

Users can choose to always set options fields explicitly in code, as shown above. Alternatively, you can also set it though a string to string map, or an option string. See Option String and Option Map

Some options can be changed dynamically while DB is running. For example:

rocksdb::Status s;
s = db->SetOptions({{"write_buffer_size", "131072"}});
assert(s.ok());
s = db->SetDBOptions({{"max_background_flushes", "2"}});
assert(s.ok());

RocksDB automatically keeps options used in the database in OPTIONS-xxxx files under the DB directory. Users can choose to preserve the option values after DB restart by extracting options from these option files. See RocksDB Options File.

Status

You may have noticed the rocksdb::Status type above. Values of this type are returned by most functions in rocksdb that may encounter an error. You can check if such a result is ok, and also print an associated error message:

   rocksdb::Status s = ...;
   if (!s.ok()) cerr << s.ToString() << endl;

Closing A Database

When you are done with a database, just delete the database object.

Example:

  ... open the db as described above ...
  ... do something with db ...
  delete db;

Reads And Writes

The database provides Put, Delete, and Get methods to modify/query the database. For example, the following code moves the value stored under key1 to key2.

  std::string value;
  rocksdb::Status s = db->Get(rocksdb::ReadOptions(), key1, &value);
  if (s.ok()) s = db->Put(rocksdb::WriteOptions(), key2, value);
  if (s.ok()) s = db->Delete(rocksdb::WriteOptions(), key1);

Right now, value size must be smaller than 4GB. RocksDB also allows Single Delete which is useful in some special cases.

Each Get results into at least a memcpy from the source to the value string. If the source is in the block cache, you can avoid the extra copy by using a PinnableSlice.

  PinnableSlice pinnable_val;
  rocksdb::Status s = db->Get(rocksdb::ReadOptions(), key1, &pinnable_val);

The source will be released once pinnable_val is destructed or ::Reset is invoked on it. Read more here.

Atomic Updates

Note that if the process dies after the Put of key2 but before the delete of key1, the same value may be left stored under multiple keys. Such problems can be avoided by using the WriteBatch class to atomically apply a set of updates:

  #include "rocksdb/write_batch.h"
  ...
  std::string value;
  rocksdb::Status s = db->Get(rocksdb::ReadOptions(), key1, &value);
  if (s.ok()) {
    rocksdb::WriteBatch batch;
    batch.Delete(key1);
    batch.Put(key2, value);
    s = db->Write(rocksdb::WriteOptions(), &batch);
  }

The WriteBatch holds a sequence of edits to be made to the database, and these edits within the batch are applied in order. Note that we called Delete before Put so that if key1 is identical to key2, we do not end up erroneously dropping the value entirely.

Apart from its atomicity benefits, WriteBatch may also be used to speed up bulk updates by placing lots of individual mutations into the same batch.

Synchronous Writes

By default, each write to rocksdb is asynchronous: it returns after pushing the write from the process into the operating system. The transfer from operating system memory to the underlying persistent storage happens asynchronously. The sync flag can be turned on for a particular write to make the write operation not return until the data being written has been pushed all the way to persistent storage. (On Posix systems, this is implemented by calling either fsync(...) or fdatasync(...) or msync(..., MS_SYNC) before the write operation returns.)

  rocksdb::WriteOptions write_options;
  write_options.sync = true;
  db->Put(write_options, ...);

Asynchronous Writes

Asynchronous writes are often more than a thousand times as fast as synchronous writes. The downside of asynchronous writes is that a crash of the machine may cause the last few updates to be lost. Note that a crash of just the writing process (i.e., not a reboot) will not cause any loss since even when sync is false, an update is pushed from the process memory into the operating system before it is considered done.

Asynchronous writes can often be used safely. For example, when loading a large amount of data into the database you can handle lost updates by restarting the bulk load after a crash. A hybrid scheme is also possible where every Nth write is synchronous, and in the event of a crash, the bulk load is restarted just after the last synchronous write finished by the previous run. (The synchronous write can update a marker that describes where to restart on a crash.)

WriteBatch provides an alternative to asynchronous writes. Multiple updates may be placed in the same WriteBatch and applied together using a synchronous write (i.e., write_options.sync is set to true). The extra cost of the synchronous write will be amortized across all of the writes in the batch.

We also provide a way to completely disable Write Ahead Log for a particular write. If you set write_option.disableWAL to true, the write will not go to the log at all and may be lost in an event of process crash.

RocksDB by default uses faster fdatasync() to sync files. If you want to use fsync(), you can set Options::use_fsync to true. You should set this to true on filesystems like ext3 that can lose files after a reboot.

Concurrency

A database may only be opened by one process at a time. The rocksdb implementation acquires a lock from the operating system to prevent misuse. Within a single process, the same rocksdb::DB object may be safely shared by multiple concurrent threads. I.e., different threads may write into or fetch iterators or call Get on the same database without any external synchronization (the rocksdb implementation will automatically do the required synchronization). However other objects (like Iterator and WriteBatch) may require external synchronization. If two threads share such an object, they must protect access to it using their own locking protocol. More details are available in the public header files.

Merge operators

Merge operators provide efficient support for read-modify-write operation. More on the interface and implementation can be found on:

Iteration

The following example demonstrates how to print all (key, value) pairs in a database.

  rocksdb::Iterator* it = db->NewIterator(rocksdb::ReadOptions());
  for (it->SeekToFirst(); it->Valid(); it->Next()) {
    cout << it->key().ToString() << ": " << it->value().ToString() << endl;
  }
  assert(it->status().ok()); // Check for any errors found during the scan
  delete it;

The following variation shows how to process just the keys in the range [start, limit):

  for (it->Seek(start);
       it->Valid() && it->key().ToString() < limit;
       it->Next()) {
    ...
  }
  assert(it->status().ok()); // Check for any errors found during the scan

You can also process entries in reverse order. (Caveat: reverse iteration may be somewhat slower than forward iteration.)

  for (it->SeekToLast(); it->Valid(); it->Prev()) {
    ...
  }
  assert(it->status().ok()); // Check for any errors found during the scan

This is an example of processing entries in range (limit, start] in reverse order from one specific key:

  for (it->SeekForPrev(start);
       it->Valid() && it->key().ToString() > limit;
       it->Prev()) {
    ...
  }
  assert(it->status().ok()); // Check for any errors found during the scan

See SeekForPrev.

For explanation of error handling, different iterating options and best practice, see Iterator.

To know about implementation details, see Iterator's Implementation

Snapshots

Snapshots provide consistent read-only views over the entire state of the key-value store. ReadOptions::snapshot may be non-NULL to indicate that a read should operate on a particular version of the DB state.

If ReadOptions::snapshot is NULL, the read will operate on an implicit snapshot of the current state.

Snapshots are created by the DB::GetSnapshot() method:

  rocksdb::ReadOptions options;
  options.snapshot = db->GetSnapshot();
  ... apply some updates to db ...
  rocksdb::Iterator* iter = db->NewIterator(options);
  ... read using iter to view the state when the snapshot was created ...
  delete iter;
  db->ReleaseSnapshot(options.snapshot);

Note that when a snapshot is no longer needed, it should be released using the DB::ReleaseSnapshot interface. This allows the implementation to get rid of state that was being maintained just to support reading as of that snapshot.

Slice

The return value of the it->key() and it->value() calls above are instances of the rocksdb::Slice type. Slice is a simple structure that contains a length and a pointer to an external byte array. Returning a Slice is a cheaper alternative to returning a std::string since we do not need to copy potentially large keys and values. In addition, rocksdb methods do not return null-terminated C-style strings since rocksdb keys and values are allowed to contain '\0' bytes.

C++ strings and null-terminated C-style strings can be easily converted to a Slice:

   rocksdb::Slice s1 = "hello";

   std::string str("world");
   rocksdb::Slice s2 = str;

A Slice can be easily converted back to a C++ string:

   std::string str = s1.ToString();
   assert(str == std::string("hello"));

Be careful when using Slices since it is up to the caller to ensure that the external byte array into which the Slice points remains live while the Slice is in use. For example, the following is buggy:

   rocksdb::Slice slice;
   if (...) {
     std::string str = ...;
     slice = str;
   }
   Use(slice);

When the if statement goes out of scope, str will be destroyed and the backing storage for slice will disappear.

Transactions

RocksDB now supports multi-operation transactions. See Transactions

Comparators

The preceding examples used the default ordering function for key, which orders bytes lexicographically. You can however supply a custom comparator when opening a database. For example, suppose each database key consists of two numbers and we should sort by the firstnumber, breaking ties by the second number. First, define a proper subclass of rocksdb::Comparator that expresses these rules:

  class TwoPartComparator : public rocksdb::Comparator {
   public:
    // Three-way comparison function:
    // if a < b: negative result
    // if a > b: positive result
    // else: zero result
    int Compare(const rocksdb::Slice& a, const rocksdb::Slice& b) const {
      int a1, a2, b1, b2;
      ParseKey(a, &a1, &a2);
      ParseKey(b, &b1, &b2);
      if (a1 < b1) return -1;
      if (a1 > b1) return +1;
      if (a2 < b2) return -1;
      if (a2 > b2) return +1;
      return 0;
    }

    // Ignore the following methods for now:
    const char* Name() const { return "TwoPartComparator"; }
    void FindShortestSeparator(std::string*, const rocksdb::Slice&) const { }
    void FindShortSuccessor(std::string*) const { }
  };

Now create a database using this custom comparator:

  TwoPartComparator cmp;
  rocksdb::DB* db;
  rocksdb::Options options;
  options.create_if_missing = true;
  options.comparator = &cmp;
  rocksdb::Status status = rocksdb::DB::Open(options, "/tmp/testdb", &db);
  ...

Column Families

Column Families provide a way to logically partition the database. Users can provide atomic writes of multiple keys across multiple column families and read a consistent view from them.

Bulk Load

You can Creating and Ingesting SST files to bulk load a large amount of data directly into DB with minimum impacts on the live traffic.

Backup and Checkpoint

Backup allows users to create periodic incremental backups in a remote file system (think about HDFS or S3) and recover from any of them.

Checkpoints provides the ability to take a snapshot of a running RocksDB database in a separate directory. Files are hardlinked, rather than copied, if possible, so it is a relatively lightweight operation.

I/O

By default, RocksDB's I/O go through operating system's page cache. Setting Rate Limiter can limit the speed that RocksDB issues file writes, to make room for read I/Os.

Users can also choose to by pass by page cache, using Direct I/O.

See IO for more details.

Backwards compatibility

The result of the comparator's Name method is attached to the database when it is created, and is checked on every subsequent database open. If the name changes, the rocksdb::DB::Open call will fail. Therefore, change the name if and only if the new key format and comparison function are incompatible with existing databases, and it is ok to discard the contents of all existing databases.

You can however still gradually evolve your key format over time with a little bit of pre-planning. For example, you could store a version number at the end of each key (one byte should suffice for most uses). When you wish to switch to a new key format (e.g., adding an optional third part to the keys processed by TwoPartComparator), (a) keep the same comparator name (b) increment the version number for new keys (c) change the comparator function so it uses the version numbers found in the keys to decide how to interpret them.

MemTable and Table factories

By default, we keep the data in memory in skiplist memtable and the data on disk in a table format described here: RocksDB Table Format.

Since one of the goals of RocksDB is to have different parts of the system easily pluggable, we support different implementations of both memtable and table format. You can supply your own memtable factory by setting Options::memtable_factory and your own table factory by setting Options::table_factory. For available memtable factories, please refer to rocksdb/memtablerep.h and for table factories to rocksdb/table.h. These features are both in active development and please be wary of any API changes that might break your application going forward.

You can also read more about memtables here and here.

Performance

Start with Set Up Options. For more information about RocksDB performance, see the "Performance" section in the sidebar in the right side.

Block size

rocksdb groups adjacent keys together into the same block and such a block is the unit of transfer to and from persistent storage. The default block size is approximately 4096 uncompressed bytes. Applications that mostly do bulk scans over the contents of the database may wish to increase this size. Applications that do a lot of point reads of small values may wish to switch to a smaller block size if performance measurements indicate an improvement. There isn't much benefit in using blocks smaller than one kilobyte, or larger than a few megabytes. Also note that compression will be more effective with larger block sizes. To change block size parameter, use Options::block_size.

Write buffer

Options::write_buffer_size specifies the amount of data to build up in memory before converting to a sorted on-disk file. Larger values increase performance, especially during bulk loads. Up to max_write_buffer_number write buffers may be held in memory at the same time, so you may wish to adjust this parameter to control memory usage. Also, a larger write buffer will result in a longer recovery time the next time the database is opened.

Related option is Options::max_write_buffer_number, which is maximum number of write buffers that are built up in memory. The default is 2, so that when 1 write buffer is being flushed to storage, new writes can continue to the other write buffer. The flush operation is executed in a Thread Pool.

Options::min_write_buffer_number_to_merge is the minimum number of write buffers that will be merged together before writing to storage. If set to 1, then all write buffers are flushed to L0 as individual files and this increases read amplification because a get request has to check in all of these files. Also, an in-memory merge may result in writing lesser data to storage if there are duplicate records in each of these individual write buffers. Default: 1

Compression

Each block is individually compressed before being written to persistent storage. Compression is on by default since the default compression method is very fast, and is automatically disabled for uncompressible data. In rare cases, applications may want to disable compression entirely, but should only do so if benchmarks show a performance improvement:

  rocksdb::Options options;
  options.compression = rocksdb::kNoCompression;
  ... rocksdb::DB::Open(options, name, ...) ....

Also Dictionary Compression is also available.

Cache

The contents of the database are stored in a set of files in the filesystem and each file stores a sequence of compressed blocks. If options.block_cache is non-NULL, it is used to cache frequently used uncompressed block contents. We use operating systems file cache to cache our raw data, which is compressed. So file cache acts as a cache for compressed data.

  #include "rocksdb/cache.h"
  rocksdb::BlockBasedTableOptions table_options;
  table_options.block_cache = rocksdb::NewLRUCache(100 * 1048576); // 100MB uncompressed cache

  rocksdb::Options options;
  options.table_factory.reset(rocksdb::NewBlockBasedTableFactory(table_options));
  rocksdb::DB* db;
  rocksdb::DB::Open(options, name, &db);
  ... use the db ...
  delete db

When performing a bulk read, the application may wish to disable caching so that the data processed by the bulk read does not end up displacing most of the cached contents. A per-iterator option can be used to achieve this:

  rocksdb::ReadOptions options;
  options.fill_cache = false;
  rocksdb::Iterator* it = db->NewIterator(options);
  for (it->SeekToFirst(); it->Valid(); it->Next()) {
    ...
  }

You can also disable block cache by setting options.no_block_cache to true.

See Block Cache for more details.

Key Layout

Note that the unit of disk transfer and caching is a block. Adjacent keys (according to the database sort order) will usually be placed in the same block. Therefore the application can improve its performance by placing keys that are accessed together near each other and placing infrequently used keys in a separate region of the key space.

For example, suppose we are implementing a simple file system on top of rocksdb. The types of entries we might wish to store are:

   filename -> permission-bits, length, list of file_block_ids
   file_block_id -> data

We might want to prefix filename keys with one letter (say '/') and the file_block_id keys with a different letter (say '0') so that scans over just the metadata do not force us to fetch and cache bulky file contents.

Filters

Because of the way rocksdb data is organized on disk, a single Get() call may involve multiple reads from disk. The optional FilterPolicy mechanism can be used to reduce the number of disk reads substantially.

   rocksdb::Options options;
   rocksdb::BlockBasedTableOptions bbto;
   bbto.filter_policy.reset(rocksdb::NewBloomFilterPolicy(
       10 /* bits_per_key */,
       false /* use_block_based_builder*/));
   options.table_factory.reset(rocksdb::NewBlockBasedTableFactory(bbto));
   rocksdb::DB* db;
   rocksdb::DB::Open(options, "/tmp/testdb", &db);
   ... use the database ...
   delete db;
   delete options.filter_policy;

The preceding code associates a Bloom Filter based filtering policy with the database. Bloom filter based filtering relies on keeping some number of bits of data in memory per key (in this case 10 bits per key since that is the argument we passed to NewBloomFilter). This filter will reduce the number of unnecessary disk reads needed for Get() calls by a factor of approximately a 100. Increasing the bits per key will lead to a larger reduction at the cost of more memory usage. We recommend that applications whose working set does not fit in memory and that do a lot of random reads set a filter policy.

If you are using a custom comparator, you should ensure that the filter policy you are using is compatible with your comparator. For example, consider a comparator that ignores trailing spaces when comparing keys. NewBloomFilter must not be used with such a comparator. Instead, the application should provide a custom filter policy that also ignores trailing spaces.

For example:

  class CustomFilterPolicy : public rocksdb::FilterPolicy {
   private:
    FilterPolicy* builtin_policy_;
   public:
    CustomFilterPolicy() : builtin_policy_(NewBloomFilter(10, false)) { }
    ~CustomFilterPolicy() { delete builtin_policy_; }

    const char* Name() const { return "IgnoreTrailingSpacesFilter"; }

    void CreateFilter(const Slice* keys, int n, std::string* dst) const {
      // Use builtin bloom filter code after removing trailing spaces
      std::vector<Slice> trimmed(n);
      for (int i = 0; i < n; i++) {
        trimmed[i] = RemoveTrailingSpaces(keys[i]);
      }
      return builtin_policy_->CreateFilter(&trimmed[i], n, dst);
    }

    bool KeyMayMatch(const Slice& key, const Slice& filter) const {
      // Use builtin bloom filter code after removing trailing spaces
      return builtin_policy_->KeyMayMatch(RemoveTrailingSpaces(key), filter);
    }
  };

Advanced applications may provide a filter policy that does not use a bloom filter but uses some other mechanism for summarizing a set of keys. See rocksdb/filter_policy.h for detail.

Checksums

rocksdb associates checksums with all data it stores in the file system. There are two separate controls provided over how aggressively these checksums are verified:

  • ReadOptions::verify_checksums forces checksum verification of all data that is read from the file system on behalf of a particular read. This is on by default.
  • Options::paranoid_checks may be set to true before opening a database to make the database implementation raise an error as soon as it detects an internal corruption. Depending on which portion of the database has been corrupted, the error may be raised when the database is opened, or later by another database operation. By default, paranoid checking is off so that the database can be used even if parts of its persistent storage have been corrupted.

    If a database is corrupted (perhaps it cannot be opened when paranoid checking is turned on), the rocksdb::RepairDB function may be used to recover as much of the data as possible.

Compaction

RocksDB keeps rewriting existing data files. This is to clean stale versions of keys, and to keep the data structure optimal for reads.

The information about compaction has been moved to Compaction. Users don't have to know internal of compactions before operating RocksDB.

Approximate Sizes

The GetApproximateSizes method can used to get the approximate number of bytes of file system space used by one or more key ranges.

   rocksdb::Range ranges[2];
   ranges[0] = rocksdb::Range("a", "c");
   ranges[1] = rocksdb::Range("x", "z");
   uint64_t sizes[2];
   rocksdb::Status s = db->GetApproximateSizes(ranges, 2, sizes);

The preceding call will set sizes[0] to the approximate number of bytes of file system space used by the key range [a..c) and sizes[1] to the approximate number of bytes used by the key range [x..z).

Environment

All file operations (and other operating system calls) issued by the rocksdb implementation are routed through a rocksdb::Env object. Sophisticated clients may wish to provide their own Env implementation to get better control. For example, an application may introduce artificial delays in the file IO paths to limit the impact of rocksdb on other activities in the system.

  class SlowEnv : public rocksdb::Env {
    .. implementation of the Env interface ...
  };

  SlowEnv env;
  rocksdb::Options options;
  options.env = &env;
  Status s = rocksdb::DB::Open(options, ...);

Porting

rocksdb may be ported to a new platform by providing platform specific implementations of the types/methods/functions exported by rocksdb/port/port.h. See rocksdb/port/port_example.h for more details.

In addition, the new platform may need a new default rocksdb::Env implementation. See rocksdb/util/env_posix.h for an example.

Manageability

To be able to efficiently tune your application, it is always helpful if you have access to usage statistics. You can collect those statistics by setting Options::table_properties_collectors or Options::statistics. For more information, refer to rocksdb/table_properties.h and rocksdb/statistics.h. These should not add significant overhead to your application and we recommend exporting them to other monitoring tools. See Statistics. You can also profile single requests using Perf Context and IO Stats Context. Users can register EventListener for callbacks for some internal events.

Purging WAL files

By default, old write-ahead logs are deleted automatically when they fall out of scope and application doesn't need them anymore. There are options that enable the user to archive the logs and then delete them lazily, either in TTL fashion or based on size limit.

The options are Options::WAL_ttl_seconds and Options::WAL_size_limit_MB. Here is how they can be used:

  • If both set to 0, logs will be deleted asap and will never get into the archive.

  • If WAL_ttl_seconds is 0 and WAL_size_limit_MB is not 0, WAL files will be checked every 10 min and if total size is greater then WAL_size_limit_MB, they will be deleted starting with the earliest until size_limit is met. All empty files will be deleted.

  • If WAL_ttl_seconds is not 0 and WAL_size_limit_MB is 0, then WAL files will be checked every WAL_ttl_seconds / 2 and those that are older than WAL_ttl_seconds will be deleted.

  • If both are not 0, WAL files will be checked every 10 min and both checks will be performed with ttl being first.

Other Information

To set up RocksDB options:

Details about the rocksdb implementation may be found in the following documents:

Contents

Clone this wiki locally