Skip to content

Commit

Permalink
docs: Updated jdbc options
Browse files Browse the repository at this point in the history
  • Loading branch information
jruaux committed Nov 1, 2024
1 parent 7bf804a commit 42c5428
Show file tree
Hide file tree
Showing 5 changed files with 90 additions and 52 deletions.
1 change: 0 additions & 1 deletion docs/guide/src/docs/asciidoc/_links.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
:link_redis_crdb: link:https://redis.com/redis-enterprise/technology/active-active-geo-distribution/[Redis Enterprise CRDB]
:link_redis_bigkeys: link:https://developer.redis.com/operate/redis-at-scale/observability/identifying-issues/#scanning-keys[Big keys]
:link_lettuce_api: link:https://lettuce.io/core/release/api/io/lettuce/core/api/sync/RedisCommands.html[Lettuce API]
:link_lettuce_uri: link:https://github.com/lettuce-io/lettuce-core/wiki/Redis-URI-and-connection-details#uri-syntax[Redis URI Syntax]
:link_lettuce_readfrom: link:https://github.com/lettuce-io/lettuce-core/wiki/ReadFrom-Settings#read-from-settings[Read-From Settings]
:link_spel: link:https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#expressions[SpEL]
:link_java_dateformat: link:https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html[Java DateFormat]
Expand Down
39 changes: 39 additions & 0 deletions docs/guide/src/docs/asciidoc/concepts.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,45 @@

image::architecture.svg[]

[[_concepts_redis_uri]]
== Redis URI

{project-title} follows the https://github.com/redis/redis-specifications/tree/master/uri[Redis URI] specification, which supports standalone, sentinel and cluster Redis deployments with plain, SSL, TLS and unix domain socket connections.

TIP: You can use the `host:port` short hand for `redis://host:port`.

TIP: You can provide the database, password and timeouts within the Redis URI.

Redis Standalone::
`redis :// [[username :] password@] host [:port][/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName]
[&libraryName=libraryName] [&libraryVersion=libraryVersion] ]`
Redis Standalone (SSL)::
`rediss :// [[username :] password@] host [: port][/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName]
[&libraryName=libraryName] [&libraryVersion=libraryVersion] ]`
Redis Standalone (Unix Domain Sockets)::
`redis-socket :// [[username :] password@]path
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&database=database]
[&clientName=clientName] [&libraryName=libraryName]
[&libraryVersion=libraryVersion] ]`
Redis Sentinel::
`redis-sentinel :// [[username :] password@] host1[:port1] [, host2[:port2]] [, hostN[:portN]] [/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&sentinelMasterId=sentinelMasterId]
[&clientName=clientName] [&libraryName=libraryName]
[&libraryVersion=libraryVersion] ]`

.Timeout Units
[horizontal]
`d`:: Days
`h`:: Hours
`m`:: Minutes
`s`:: Seconds
`ms`:: Milliseconds
`us`:: Microseconds
`ns`:: Nanoseconds


[[_concepts_batching]]
== Batching

Expand Down
5 changes: 2 additions & 3 deletions docs/guide/src/docs/asciidoc/databases.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ See the <<_db_drivers,Drivers>> section for more details.

[source,console]
----
riot db-import -h <redis host> -p <redis port> --url <jdbc url> SQL [REDIS COMMAND...]
riot db-import --jdbc-url <jdbc url> SQL [REDIS COMMAND...]
----

To show the full usage, run:
Expand All @@ -70,7 +70,6 @@ riot db-import "SELECT * FROM customers" hset -h myredis.com -p 6380
----
====


The keys that will be written are constructed from input records by concatenating the keyspace prefix and key fields.

image::mapping.svg[]
Expand Down Expand Up @@ -104,7 +103,7 @@ See the <<_db_drivers,Drivers>> section for more details.
The general usage is:
[source,console]
----
riot db-export -h <redis host> -p <redis port> --url <jdbc url> SQL
riot db-export --jdbc-url <jdbc url> SQL
----

To show the full usage, run:
Expand Down
48 changes: 17 additions & 31 deletions docs/guide/src/docs/asciidoc/faq.adoc
Original file line number Diff line number Diff line change
@@ -1,44 +1,30 @@
[[_faq]]
= FAQ

[[_faq_logs_cutoff]]
[qanda]
Logs are cut off or missing::
This could be due to concurrency issues in the terminal when refreshing the progress bar and displaying logs.
Try running with job option `--progress log`.

[[_faq_unknown_options]]
This could be due to concurrency issues in the terminal when refreshing the progress bar and displaying logs.
Try running with job option `--progress log`.
Unknown options: '--keyspace', '--key'::
You must specify one or more Redis commands with import commands (<<_file_import,`file-import`>>, <<_datagen_faker,`faker`>>, <<_db_import,`db-import`>>).

[[_faq_dump_version]]
You must specify one or more Redis commands with import commands (<<_file_import,`file-import`>>, <<_datagen_faker,`faker`>>, <<_db_import,`db-import`>>).
ERR DUMP payload version or checksum are wrong::
Redis 7 DUMP format is not backwards compatible with previous versions.
To replicate between different Redis versions, use <<_replication_type_struct,Type-Based Replication>>.

[[_faq_replication_timeout]]
Redis 7 DUMP format is not backwards compatible with previous versions.
To replicate between different Redis versions, use <<_replication_type_struct,Type-Based Replication>>.
Process gets stuck during replication and eventually times out::
This could be due to big keys clogging the replication pipes.
In these cases it might be hard to catch the offending key(s).
Try running the same command with `--info` and `--progress log` so that all errors are reported.
Check the database with `redis-cli` {link_redis_bigkeys} and/or use reader options to filter these keys out.

[[_faq_noauth]]
This could be due to big keys clogging the replication pipes.
In these cases it might be hard to catch the offending key(s).
Try running the same command with `--info` and `--progress log` so that all errors are reported.
Check the database with `redis-cli` {link_redis_bigkeys} and/or use reader options to filter these keys out.
NOAUTH Authentication required::
This issue occurs when you fail to supply the `--pass <password>` parameter.

[[_faq_stream_id]]
This issue occurs when you fail to supply the `--pass <password>` parameter.
ERR The ID argument cannot be a complete ID because xadd-id-uniqueness-mode is strict::
This usually happens in Active/Active (CRDB) setups where stream message IDs cannot be copied over to the target database.
Use the `--no-stream-id` option to disable ID propagation.

[[_faq_script]]
This usually happens in Active/Active (CRDB) setups where stream message IDs cannot be copied over to the target database.
Use the `--no-stream-id` option to disable ID propagation.
ERR Error running script... This Redis command is not allowed from scripts::
This can happen with Active/Active (CRDB) databases because the `MEMORY USAGE` command is not allowed to be run from a LUA script.
Use the `--mem-limit -1` option to disable memory usage.

[[_faq_outofmemory]]
This can happen with Active/Active (CRDB) databases because the `MEMORY USAGE` command is not allowed to be run from a LUA script.
Use the `--mem-limit -1` option to disable memory usage.
java.lang.OutOfMemoryError: Java heap space::
The JVM running RIOT ran out of memory.
Either increase max JVM heap size (`export JAVA_OPTS="-Xmx8g"`) or reduce RIOT memory usage by lowering `threads`, `batch`, `read-batch` and `read-queue`.
The RIOT JVM ran out of memory.
Either increase max JVM heap size (`export JAVA_OPTS="-Xmx8g"`) or reduce RIOT memory usage by lowering `threads`, `batch`, `read-batch` and `read-queue`.


49 changes: 32 additions & 17 deletions docs/guide/src/docs/asciidoc/replication.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,54 @@

The `replicate` command reads data from a *source* Redis database and writes to a *target* Redis database.

The basic usage is:

[source,console]
----
riot replicate SOURCE TARGET [--mode <scan|live|liveonly>] [--struct] [OPTIONS]
riot replicate [OPTIONS] SOURCE TARGET
----

where SOURCE and TARGET are <<_concepts_redis_uri,Redis URIs>>.

For the full usage, run:
[source,console]
----
riot replicate --help
----

TIP: To replicate a Redis logical database other than the default (`0`), specify the database in the source Redis URI.
For example `riot replicate redis://source:6379/1 redis://target:6379` replicates database `1`.

A status bar shows progress with a percentage of keys that have been replicated.
The total number of keys is estimated when the process starts and it can change by the time it is finished, for example if keys are deleted or added during replication.

[[_replication_mode]]
== Replication Modes

The `--mode` option allows you to specify how {project-title} identifies keys to be replicated.
Possible values: `SCAN` (default), `LIVE`, `LIVEONLY`.

[[_replication_mode_scan]]
=== Scan
=== `SCAN`
This mode uses the `SCAN` command to identify keys to propagate to the target Redis database.

.Scan replication example
[source,console]
----
include::{testdir}/replicate[]
----

[[_replication_mode_live]]
=== Live
=== `LIVE`
This mode builds upon scan replication by also listening for changes on the source Redis database.
Whenever a key is modified its corresponding value is read and propagated to the target Redis database.

.Live replication example
[source,console]
----
include::{testdir}/replicate-live[]
----

Live replication relies on keyspace notifications for capturing these changes.

**Make sure the source database has keyspace notifications enabled** using:
Expand All @@ -53,11 +75,16 @@ If you need assistance please contact your Redis account team.
====

[[_replication_mode_liveonly]]
=== Live Only
=== `LIVEONLY`
In this mode only keyspace notifications are used.

== Replication Types

{project-title} offers two different mechanisms for reading and writing keys:

* Dump & restore (default)
* Data structure replication (`--struct`)

[[_replication_type_dump]]
=== Dump & Restore

Expand All @@ -78,7 +105,7 @@ There are situations where Dump & Restore cannot be used, for example:
* The target Redis database does not support the RESTORE command ({link_redis_crdb})
* Incompatible DUMP formats between source and target ({link_redis_7})

In those cases you can use another replication strategy that is data structure-specific: each key is introspected to determine its type and then use the corresponding read/write commands:
In those cases you can use another replication strategy that is data structure-specific: each key is introspected to determine its type and then use the corresponding read/write commands.

[%header,cols="h,1,1"]
|=========================================================
Expand All @@ -98,17 +125,6 @@ In those cases you can use another replication strategy that is data structure-s
WARNING: This replication strategy is more intensive in terms of CPU, memory, and network for all the machines involved (source Redis, target Redis, and {project-title} machines).
Adjust number of threads, batch and queue sizes accordingly.

.Snapshot replication example
[source,console]
----
include::{testdir}/replicate[]
----

.Live replication example
[source,console]
----
include::{testdir}/replicate-live[]
----

.Type-based replication example
[source,console]
Expand Down Expand Up @@ -136,7 +152,6 @@ Verification can also be run on-demand using the `compare` command:
riot compare SOURCE TARGET [OPTIONS]
----


The output looks like this:

----
Expand Down

0 comments on commit 42c5428

Please sign in to comment.