From 6ed3a4951ccad6a914b3ec65cead2e07eaa96f1d Mon Sep 17 00:00:00 2001 From: Matteo Merli Date: Sat, 3 Feb 2018 18:44:13 -0800 Subject: [PATCH] Some clarifications on Retention documentation (#1141) --- site/docs/latest/advanced/RetentionExpiry.md | 38 ++++++++++++++++---- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/site/docs/latest/advanced/RetentionExpiry.md b/site/docs/latest/advanced/RetentionExpiry.md index 1df5ac3b9a095..68448c031c277 100644 --- a/site/docs/latest/advanced/RetentionExpiry.md +++ b/site/docs/latest/advanced/RetentionExpiry.md @@ -27,18 +27,21 @@ tags: [admin, expiry, retention, backlog] Pulsar {% popover brokers %} are responsible for handling messages that pass through Pulsar, including [persistent storage](../../getting-started/ConceptsAndArchitecture#persistent-storage) of messages. By default, brokers: -* immediately delete all messages that have been {% popover acknowledged %} by a {% popover consumer %}, and +* immediately delete all messages that have been {% popover acknowledged %} on every {% popover subscription %}, and * persistently store all {% popover unacknowledged %} messages in a [backlog](#backlog-quotas). In Pulsar, you can override both of these default behaviors, at the {% popover namespace %} level, in two ways: -* You can persistently store messages that have already been {% popover acknowledged %} by a {% popover consumer %} by setting [retention policies](#retention-policies). -* You can delete messages that have not been acknowledged within a specified timeframe using [time to live](#time-to-live-ttl) (TTL). +* You can persistently store messages that have already been consumed and {% popover acknowledged %} for a minimum time by setting [retention policies](#retention-policies). +* Messages that are not acknowledged within a specified timeframe, can be automatically marked as consumed, by specifying the [time to live](#time-to-live-ttl) (TTL). Pulsar's [admin interface](../../admin-api/overview) enables you to manage both retention policies and TTL at the {% popover namespace %} level (and thus within a specific {% popover property %} and either on a specific {% popover cluster %} or in the [`global`](../../getting-started/ConceptsAndArchitecture#global-cluster) cluster). -{% include admonition.html type="warning" title="Don't use retention and TTL at the same time" content=" -Message retention policies and TTL fulfill similar purposes and should not be used in conjunction. For any given namespace, use one or the other. +{% include admonition.html type="info" title="Retention and TTL are solving two different problems" content=" + * Message retention: Keep the data for at least X hours (even if acknowledged) + * Time-to-live: Discard data after some time (by automatically acknowledging) + +In most cases, applications will want to use either one or the other (or none). " %} ## Retention policies @@ -47,9 +50,12 @@ By default, when a Pulsar message arrives at a {% popover broker %} it will be s When you set a size limit of, say, 10 gigabytes, then messages in all topics in the namespace, *even acknowledged messages*, will be retained until the size limit for the topic is reached; if you set a time limit of, say, 1 day, then messages for all topics in the namespace will be retained for 24 hours. +It is also possible to set *infinite* retention time or size, by setting `-1` for either time or +size retention. + ### Defaults -There are two configuration parameters that you can use to set {% popover instance %}-wide defaults for message retention: [`defaultRetentionTimeInMinutes`](../../reference/Configuration#broker-defaultRetentionTimeInMinutes) and [`defaultRetentionSizeInMB`](../../reference/Configuration#broker-defaultRetentionSizeInMB). +There are two configuration parameters that you can use to set {% popover instance %}-wide defaults for message retention: [`defaultRetentionTimeInMinutes=0`](../../reference/Configuration#broker-defaultRetentionTimeInMinutes) and [`defaultRetentionSizeInMB=0`](../../reference/Configuration#broker-defaultRetentionSizeInMB). Both of these parameters are in the [`broker.conf`](../../reference/Configuration#broker) configuration file. @@ -71,6 +77,24 @@ $ pulsar-admin namespaces set-retention my-prop/my-cluster/my-ns \ --time 3h ``` +To set retention with infinite time and a size limit: + +```shell +$ pulsar-admin namespaces set-retention my-prop/my-cluster/my-ns \ + --size 1T \ + --time -1 +``` + +Similarly, even the size can be to unlimited: + +```shell +$ pulsar-admin namespaces set-retention my-prop/my-cluster/my-ns \ + --size -1 \ + --time -1 +``` + + + #### REST API {% endpoint POST /admin/namespaces/:property/:cluster/:namespace/retention %} @@ -240,7 +264,7 @@ By default, you will be prompted to ensure that you really want to clear the bac ## Time to live (TTL) -By default, Pulsar stores all unacknowledged messages forever. This can lead to heavy disk space usage in cases where a lot of messages are going unacknowledged. If disk space is a concern, you can set a time to live (TTL) that determines how long unacknowledged messages will +By default, Pulsar stores all unacknowledged messages forever. This can lead to heavy disk space usage in cases where a lot of messages are going unacknowledged. If disk space is a concern, you can set a time to live (TTL) that determines how long unacknowledged messages will be retained. ### Set the TTL for a namespace