Skip to content

Home

Jump to bottom
engelsanchez edited this page Feb 19, 2013 · 32 revisions

Riak Core is the distributed systems framework upon which Riak, Basho's distributed Key/Value datastore, is built. This Wiki is meant to help you understand the riak core concepts and how they are connected to the source code modules. There is also more detailed information about the riak_core modules and functions at http://basho.github.com/riak_core, the edoc pages generated straight from the source code. For more information, check out:

Core is written in Erlang, using its OTP framework. You will need to understand Erlang and OTP to understand how Riak Core works. The Erlang/OTP in a Nutshell page is a good starting point.

Contents

In a nutshell

Riak Core handles the distribution of ownership among a group of nodes in a cluster. An abstract space is split into a given number of partitions, each of which is owned by an entity called a virtual node. We normally refer to this space as a Ring. The virtual nodes are spread among all the nodes in the cluster. The number of partitions is fixed, so adding and removing nodes would cause the virtual nodes to be re-distributed. For example, your application may start with 64 partitions in a single node, and hence 64 virtual nodes running in that single node. When another node is added ownership would be split, and you would end up with 2 nodes each one running 32 virtual nodes, and so on.

A typical riak_core application would handle incoming requests by determining the partition they correspond to and routing them to the corresponding virtual node. Requests could also span many or all partitions. In core, that is handled by coverage queries.

Main concepts

Ring

Information about existing partitions and the virtual nodes responsible to handle them is stored in a data structure called the Ring. Each node in the cluster has a copy of this ring. The number of partitions in the ring is decided at installation time and stays fixed. There is current work being done to allow it to change size. We also store data about buckets in it. This might change in the future as it limits scalability. Information about ongoing ownership transitions are also stored in the ring, as well as the designated Claimant node. The ring structure is defined in riak_core_ring (it's the chstate_v2 record). Also see riak_core_ring_manager.

Virtual nodes

Each virtual node (or vnode) is a worker responsible for a given partition in the ring. A riak_core node will host any number of vnodes, depending on the size of the ring and the number of nodes in the cluster. Basically, around (size of ring) / (number of nodes). What vnodes do will vary per application. In general, they handle commands that need to be executed for the partition they own. See more about their API in the vnode page

Buckets

Buckets are nothing but namespaces with configuration properties. Your application may use only one, or whatever number works best. See riak_core_bucket

Gossip

Changes to the ring need to be propagated throughout the cluster. This is what the gossip protocol is for. The main module involved is riak_core_gossip.

Handoffs

Ownership of a partition may be transferred from one virtual node to another in a different node when nodes join or are removed from the cluster, and under certain failure scenarios to guarantee high availability. If a node goes down unexpectedly, the partitions owned by the virtual nodes it contained will be temporarily handled by virtual nodes in other physical nodes. If the original node comes back up, ownership will eventually be transferred back to the original owners, also called primary virtual nodes. The virtual nodes that took over ownership in that scenario are called secondary virtual nodes. The process by which this ownership is negotiated and any relevant data is transferred to accomplish that is what we call a handoff. Transfer of ownership may also occur when adding or removing physical nodes to the cluster. See all the gory details in the handoff page

Claims

At any given point, one node from the cluster is assigned the task of handling ownership changes when other nodes are added or removed from the cluster. This process also involves keeping the cluster nicely balanced with partitions well spread out among all nodes.

Capabilities

Sometimes not all nodes in a cluster are the same. The most common scenario is a rolling upgrade. Nodes may be taken down, upgraded to a new version of the software and put back in the cluster. The new software may contain new features that the old version can not understand, so they should not be activated. But they should be used as soon as all nodes are upgraded. This is resolved with the capabilities negotiation mechanism.

See riak_core_capability.

Supervisor hierarchy

At the top, we have the riak_core_sup module. Underneath that supervisor, we have the following process hierarchy:

Clone this wiki locally