Skip to content

Commit

Permalink
doc UPDATE datastore plugins MONGO DS and REDIS DS
Browse files Browse the repository at this point in the history
  • Loading branch information
Humblesaw committed Apr 16, 2024
1 parent cd1fe42 commit 75ce990
Show file tree
Hide file tree
Showing 2 changed files with 73 additions and 3 deletions.
45 changes: 42 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,12 +82,12 @@ other users.
#### Optional

* pkg-config & libsystemd (to support `sysrepo-plugind` systemd service)
* mongodb-org; libmongoc >= 1.26.2; libbson >= 1.26.2 (for MONGO DS datastore plugin)
* redis-stack-server; hiredis (for REDIS DS datastore plugin)
* doxygen (for generating documentation)
* cmocka >= 1.0.1 (for tests only, see [Tests](#Tests))
* valgrind (for enhanced testing)
* gcov (for code coverage)
* lcov (for code coverage)
* genhtml (for code coverage)
* gcov; lcov; genhtml (for code coverage)

## Building

Expand Down Expand Up @@ -275,6 +275,45 @@ an internal subscription to the `/ietf-factory-default:factory-reset` RPC which
able to subscribe to it with higher or lower priority and perform any other tasks required for a device
to be rest to its factory settings.

## Datastore plugins

In sysrepo there are three internal datastore plugins (`JSON DS file`, `MONGO DS` and `REDIS DS`). The default datastore
plugin is `JSON DS file` which stores all the data to JSON files. `MONGO DS` and `REDIS DS` store data to a database and are used after setting a few CMake environment
variables. For every datastore a different datastore plugin can be set. For example:

`cmake -DDEFAULT_STARTUP_DS_PLG="MONGO DS" -DDEFAULT_RUNNING_DS_PLG="MONGO DS" -DDEFAULT_CANDIDATE_DS_PLG="REDIS DS" -DDEFAULT_OPERATIONAL_DS_PLG="JSON DS file" -DDEFAULT_FACTORY_DEFAULT_DS_PLG="JSON DS file" ..`

The shared memory prefix set by `SYSREPO_SHM_PREFIX` is used by each plugin differently.
`JSON DS file` includes it in the name of every file it creates, whereas `MONGO DS` includes it
in the name of every collection and lastly `REDIS DS` includes it in the name of every key as a part of the prefix.
For more information about plugins, see [plugin documentation](doc/sr_plugins.dox).

### MONGO DS

To use `MONGO DS` datastore plugin, **libmongoc** and **libbson** libraries have to be present
on the system (see [Official libmongoc documentation](https://mongoc.org/libmongoc/current/installing.html)). Additionally
a running MongoDB server has to be available to the system (see [Official MongoDB documentation](https://www.mongodb.com/docs/manual/installation/)). By default
sysrepo assumes that the server is available at the loopback address `127.0.0.1` and port `27017` with
no authentication needed. For different IP address and port, set `MONGO_HOST` and `MONGO_PORT` CMake
variables. For an authentication via username and password, set `MONGO_USERNAME` and `MONGO_PASSWORD`
CMake variables. Please note that for sysrepo to correctly authenticate, an existing user with sufficient rights
and with the configured username and password has to be available
on the server. Also if the user is created on a different database than `admin`, provide the correct database name on which the user was created
via the `MONGO_AUTHSOURCE` CMake variable. Lastly authentication has to be enabled in the server configuration (see [Official MongoDB documentation](https://www.mongodb.com/docs/manual/administration/security-checklist/#std-label-checklist-auth)).
For more information on how the plugin works, please refer to the [plugin documentation](doc/sr_plugins.dox).

### REDIS DS

To use `REDIS DS` datastore plugin, **libhiredis** library has to be present on the system (see [Official Hiredis documentation](https://github.com/redis/hiredis)). Additionally a running
Redis Stack server has to be available to the system (see [Official Redis documentation](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)).
By default sysrepo assumes that the server is available at the loopback address `127.0.0.1` and port `6379` with
no authentication needed. For different IP address and port, set `REDIS_HOST` and `REDIS_PORT` CMake variables.
For an authentication via username and password, set `REDIS_USERNAME` and `REDIS_PASSWORD` CMake variables.
Please note that for sysrepo to correctly authenticate, an existing user with sufficient rights and with the same
username and password provided via `REDIS_USERNAME` and `REDIS_PASSWORD` has to be available on the server.
Additionally authentication has to be enforced by the server (see [Official Redis documentation](https://redis.io/docs/latest/commands/auth/)).
For more information on how the plugin works, please refer to the [plugin documentation](doc/sr_plugins.dox).

## Examples

See [examples](examples) directory, which contains an example for basic API functions.
Expand Down
31 changes: 31 additions & 0 deletions doc/sr_plugins.dox
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,35 @@ implementation.

@ref ntfplg_api

@section sr_ds_plg_mongo MONGO DS plugin

`MONGO DS` plugin works as a database client. It connects and sends necessary commands directly to the MongoDB database
server. It connects via the **libmongoc** library API and uses its functions to further communicate with the server.
In MongoDB database there is a hierarchy of data. All data are stored in records which are then stored in collections
which are then stored in databases. Each datastore for which `MONGO DS` plugin was deployed has its own database.
The name of the database consists of a `sr_` prefix and the name of the datastore, e.g. `sr_factory-default`. Data of each
particular YANG module are then stored in a collection within the particular database. In order to avoid data
conflicts the databases with the names `sr_startup`, `sr_running`, `sr_candidate`, `sr_operational` and `sr_factory-default`
can only be used by the sysrepo library.

In order for the authentication (see README) to work the configured user has to have a read and a write permission on
following databases: `sr_startup`, `sr_running`, `sr_candidate`, `sr_operational` and `sr_factory-default`.

@section sr_ds_plg_redis REDIS DS plugin

`REDIS DS` plugin first connects to the Redis Stack database server via **hiredis** library API and then sends
commands to the server using its functions. In Redis database all data are stored together. In order to differentiate between them,
the keys have to be prefixed. The prefix for all sysrepo data is `sr:` and in order to avoid data conflicts the keys and indexes
starting with a `sr:` prefix can only be used by the sysrepo library.

In Redis database there are certain limits with respect to data management which cannot be set to unlimited. The maximum amount
of data which can be loaded from the database at once is set to one trillion. Some of these limits are enforced via the server
configuration and since the sysrepo library does not have an access to the server's configuration file, it sets these limits
via commands (more specifically the options MAXAGGREGATERESULTS and MAXEXPANSIONS).

In order for the authentication (see README) to work the user has to have the right to operate on all keys and indexes with the prefix `sr:` and
at least the right to use these following commands: `AUTH`, `FT.CONFIG SET`, `FT.AGGREGATE`, `FT.CURSOR READ`, `FT.CREATE`,
`FT.DROPINDEX`, `HSET`, `HGET`, `SET`, `GET`, `DEL`, `COPY`. Redis supports categories of commands, so to enable all of these commands
at once you only have to enable a category containing all of them, e.g. `@all`.

*/

0 comments on commit 75ce990

Please sign in to comment.