From 75ce9904b3afcbc871d7011c66ef99124c4072b0 Mon Sep 17 00:00:00 2001 From: Ondrej Kusnirik Date: Tue, 16 Apr 2024 15:59:29 +0200 Subject: [PATCH] doc UPDATE datastore plugins MONGO DS and REDIS DS --- README.md | 45 ++++++++++++++++++++++++++++++++++++++++++--- doc/sr_plugins.dox | 31 +++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d1c9c7b82..39b899fff 100644 --- a/README.md +++ b/README.md @@ -82,12 +82,12 @@ other users. #### Optional * pkg-config & libsystemd (to support `sysrepo-plugind` systemd service) +* mongodb-org; libmongoc >= 1.26.2; libbson >= 1.26.2 (for MONGO DS datastore plugin) +* redis-stack-server; hiredis (for REDIS DS datastore plugin) * doxygen (for generating documentation) * cmocka >= 1.0.1 (for tests only, see [Tests](#Tests)) * valgrind (for enhanced testing) -* gcov (for code coverage) -* lcov (for code coverage) -* genhtml (for code coverage) +* gcov; lcov; genhtml (for code coverage) ## Building @@ -275,6 +275,45 @@ an internal subscription to the `/ietf-factory-default:factory-reset` RPC which able to subscribe to it with higher or lower priority and perform any other tasks required for a device to be rest to its factory settings. +## Datastore plugins + +In sysrepo there are three internal datastore plugins (`JSON DS file`, `MONGO DS` and `REDIS DS`). The default datastore +plugin is `JSON DS file` which stores all the data to JSON files. `MONGO DS` and `REDIS DS` store data to a database and are used after setting a few CMake environment +variables. For every datastore a different datastore plugin can be set. For example: + +`cmake -DDEFAULT_STARTUP_DS_PLG="MONGO DS" -DDEFAULT_RUNNING_DS_PLG="MONGO DS" -DDEFAULT_CANDIDATE_DS_PLG="REDIS DS" -DDEFAULT_OPERATIONAL_DS_PLG="JSON DS file" -DDEFAULT_FACTORY_DEFAULT_DS_PLG="JSON DS file" ..` + +The shared memory prefix set by `SYSREPO_SHM_PREFIX` is used by each plugin differently. +`JSON DS file` includes it in the name of every file it creates, whereas `MONGO DS` includes it +in the name of every collection and lastly `REDIS DS` includes it in the name of every key as a part of the prefix. +For more information about plugins, see [plugin documentation](doc/sr_plugins.dox). + +### MONGO DS + +To use `MONGO DS` datastore plugin, **libmongoc** and **libbson** libraries have to be present +on the system (see [Official libmongoc documentation](https://mongoc.org/libmongoc/current/installing.html)). Additionally +a running MongoDB server has to be available to the system (see [Official MongoDB documentation](https://www.mongodb.com/docs/manual/installation/)). By default +sysrepo assumes that the server is available at the loopback address `127.0.0.1` and port `27017` with +no authentication needed. For different IP address and port, set `MONGO_HOST` and `MONGO_PORT` CMake +variables. For an authentication via username and password, set `MONGO_USERNAME` and `MONGO_PASSWORD` +CMake variables. Please note that for sysrepo to correctly authenticate, an existing user with sufficient rights +and with the configured username and password has to be available +on the server. Also if the user is created on a different database than `admin`, provide the correct database name on which the user was created +via the `MONGO_AUTHSOURCE` CMake variable. Lastly authentication has to be enabled in the server configuration (see [Official MongoDB documentation](https://www.mongodb.com/docs/manual/administration/security-checklist/#std-label-checklist-auth)). +For more information on how the plugin works, please refer to the [plugin documentation](doc/sr_plugins.dox). + +### REDIS DS + +To use `REDIS DS` datastore plugin, **libhiredis** library has to be present on the system (see [Official Hiredis documentation](https://github.com/redis/hiredis)). Additionally a running +Redis Stack server has to be available to the system (see [Official Redis documentation](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)). +By default sysrepo assumes that the server is available at the loopback address `127.0.0.1` and port `6379` with +no authentication needed. For different IP address and port, set `REDIS_HOST` and `REDIS_PORT` CMake variables. +For an authentication via username and password, set `REDIS_USERNAME` and `REDIS_PASSWORD` CMake variables. +Please note that for sysrepo to correctly authenticate, an existing user with sufficient rights and with the same +username and password provided via `REDIS_USERNAME` and `REDIS_PASSWORD` has to be available on the server. +Additionally authentication has to be enforced by the server (see [Official Redis documentation](https://redis.io/docs/latest/commands/auth/)). +For more information on how the plugin works, please refer to the [plugin documentation](doc/sr_plugins.dox). + ## Examples See [examples](examples) directory, which contains an example for basic API functions. diff --git a/doc/sr_plugins.dox b/doc/sr_plugins.dox index 28d91a5ef..0306c62ba 100644 --- a/doc/sr_plugins.dox +++ b/doc/sr_plugins.dox @@ -28,4 +28,35 @@ implementation. @ref ntfplg_api +@section sr_ds_plg_mongo MONGO DS plugin + +`MONGO DS` plugin works as a database client. It connects and sends necessary commands directly to the MongoDB database +server. It connects via the **libmongoc** library API and uses its functions to further communicate with the server. +In MongoDB database there is a hierarchy of data. All data are stored in records which are then stored in collections +which are then stored in databases. Each datastore for which `MONGO DS` plugin was deployed has its own database. +The name of the database consists of a `sr_` prefix and the name of the datastore, e.g. `sr_factory-default`. Data of each +particular YANG module are then stored in a collection within the particular database. In order to avoid data +conflicts the databases with the names `sr_startup`, `sr_running`, `sr_candidate`, `sr_operational` and `sr_factory-default` +can only be used by the sysrepo library. + +In order for the authentication (see README) to work the configured user has to have a read and a write permission on +following databases: `sr_startup`, `sr_running`, `sr_candidate`, `sr_operational` and `sr_factory-default`. + +@section sr_ds_plg_redis REDIS DS plugin + +`REDIS DS` plugin first connects to the Redis Stack database server via **hiredis** library API and then sends +commands to the server using its functions. In Redis database all data are stored together. In order to differentiate between them, +the keys have to be prefixed. The prefix for all sysrepo data is `sr:` and in order to avoid data conflicts the keys and indexes +starting with a `sr:` prefix can only be used by the sysrepo library. + +In Redis database there are certain limits with respect to data management which cannot be set to unlimited. The maximum amount +of data which can be loaded from the database at once is set to one trillion. Some of these limits are enforced via the server +configuration and since the sysrepo library does not have an access to the server's configuration file, it sets these limits +via commands (more specifically the options MAXAGGREGATERESULTS and MAXEXPANSIONS). + +In order for the authentication (see README) to work the user has to have the right to operate on all keys and indexes with the prefix `sr:` and +at least the right to use these following commands: `AUTH`, `FT.CONFIG SET`, `FT.AGGREGATE`, `FT.CURSOR READ`, `FT.CREATE`, +`FT.DROPINDEX`, `HSET`, `HGET`, `SET`, `GET`, `DEL`, `COPY`. Redis supports categories of commands, so to enable all of these commands +at once you only have to enable a category containing all of them, e.g. `@all`. + */