Redis is an in-memory key-value store. Redis can be operated as a simple store for managing (notably) volatile persistent data. Redis can operate in serveral modes, ranging from a single server to clusters organised in several different ways to provide high availability, resilience and replicating the data close to the involved servers. In addition to being a key-value store, Redis enables additional communication between clients such as publish/subscribe to message channels, streams, etc.
These features can be used to connect micro services, both for sharing state, notifications and distributing tasks.
The connection between the redis client and server uses a stream pair. Although SWI-Prolog stream I/O is thread-safe, having multiple threads using this same connection will mixup writes and their replies.
At the moment, the following locking is in place.
- Connections created using redis_connect/3 are not locked. This implies the connection handle may be used from a single thread only, or redis/3 requests must be protected using with_mutex/2.
- Redis/3 request using a server name established using redis_server/3 are locked using a mutex with the same name as the server name.
If SWI-Prolog includes the ssl library, the Redis client can connect
to the server using TLS (SSL). Connecting requires the same three
files as redis-cli requires: the root certificate file, a client
certificate and the private key of the client certificate. Below is
an example call to redis_server/3:
:- redis_server(swish, localhost:6379,
[ user(bob),
password("topsecret"),
version(3),
tls(true),
cacert('ca.crt'),
key('client.key'),
cert('client.cert')
]).
Redis sentinels is one of the two options to create a high
availability service. It consists of minimally three Redis servers
and mininally three sentinel servers. The sentinel servers monitor
the Redis servers and will initiate a fail-over when the master
becomes disfunctional and certain safety constraints are satisfied. A
client needs to be aware of this setup. It is given an initial list
with (a subset of) the known sentinels. The client attempts to
connect to one of the sentinels and ask it for the current Redis
master server. Details are described in Sentinel client
spec. The
SWI-Prolog client maintains the actual list of sentinels dynamically
after successful discovery of the first sentinel. Below is an example
redis_server/3 to connect to a sentinel network. The Address specification
sentinel(swish) tells the library we want to connect to a sentinel
network that is monitored under the name swish.
:- redis_server(swish_sentinel, sentinel(swish),
[ user(janbob),
password("topsecret"),
version(3),
sentinels([ host1:26379,
host2:26379,
host3:26379
])
]).
The current stable version of Redis is 6. Many Linux distros still ship with version 5. Both talk protocol version 2. Version 6 also supports protocol version 3. The main differences are:
- The version 3 protocol has several improvements that notably improvement passing large objects using a streaming protocol.
- Hashes (maps) in the version 3 protocol are exchanged as lists
of pairs (
Name-Value), while version 2 exchanges hashes as a list of alternating names and values. This is visible to the user. High level predicates such as redis_get_hash/3 deal with both representations. - The version 3 protocol supports push messages to deal with monitor and subscribe events on the same connection as used for handling normal requests.
New projects are encouraged to use Redis version 6 with the version 3 protocol. See redis_server/3.
Starting with Redis 5, redis supports streams. A stream is a list of messages. Streams can be used as a reliable alternative to the older Redis PUB/SUB (Publish Subscribe) mechanism that has no memory, i.e., if a node is down when a message arrives the message is missed. In addition, they can be used to have each message processed by a consumer that belongs to a consumer group. Both facilities are supported by library(redis_streams)
Redis streams provide all the low-level primitives to realise message brokering. Putting it all together is non-trivial though. Notably:
-
We must take care of messages that have been sent to some consumer but the consumer fails to process the message and (thus) ACK it is processed. This is handled by xlisten_group/5 using several options. Good defaults for these options are hard to give as it depends on the required processing time for a message, how common failures are and an acceptable delay time in case of a failure, what to do in case of a persistent failure, etc.
-
Streams are independent from consumer groups and acknowledged messages remain in the stream. xstream_set/3 can be used to limit the length of the stream, discarding the oldest messages. However, it is hard to give a sensible default. The required queue length depends on the the size of the messages, whether messages come in more or less randomly or in bursts (that cause the stream to grow for a while), available memory, how bad it is if some messages get lost, etc.
The directory doc/packages/examples/redis in the installation provides
an example using streams and consumer groups to realise one or more
clients connected to one or more compute nodes.
This module is based on the gpredis.pl by Sean Charles for GNU-Prolog.
This file greatly helped me understanding what had to be done, although,
eventually, not much of the original interface is left. The main
difference to the original client are:
- Replies are not wrapped by type in a compound term.
- String replies use the SWI-Prolog string type.
- Values can be specified as
Value as prolog, after which they are returns as a (copy of) Value. This prefixes the value using "\u0000T\u0000". - Strings are in UTF-8 encoding to support full Unicode.
- Using redis_server/3, actual connections are established lazily and when a connection is lost it is automatically restarted.
- This library allows for using the Redis publish/subscribe interface. Messages are propagated using broadcast/1.