Skip to content

Latest commit

 

History

History
172 lines (117 loc) · 9.02 KB

ephemeraldata.adoc

File metadata and controls

172 lines (117 loc) · 9.02 KB
Raw

Ephemeral Data

There are 3 main types of ephemeral data stored in LDAP during normal operation of {product-name}.

  • Last Logon Time Stamps (zimbraLastLogonTimestamp)

  • Auth Tokens (zimbraAuthTokens)

  • CSRF Tokens (zimbraCsrfTokenData)

On small systems, storage of these types of ephemeral data may be done in the LDAP Server. However, mail systems with large numbers of active users have been found to overload LDAP for short-lived data storage. Therefore, the preferred option is to store this ephemeral data using an external server.

Note
 This document does not cover how to install and maintain the ephemeral storage server.

Configuring the storage location of ephemeral data is done through the following LDAP attribute:

Attribute

Format

Description

zimbraEphemeralBackendURL

[backend name]:[params]

Ephemeral Backend URL Configuration

The two currently supported Ephemeral Data backends are:

Backend

Format

Description

LDAP

ldap://default

Default configuration

SSDB

ssdb:127.0.0.1:8888

SSDB server and port

Frequent authentication requests place a high load on Ephemeral Data storage backend. See the following Zimbra wiki pages for results of authentication-based load tests:

Configuring a Running {product-name} to Use SSDB

Configuring an already running {product-name} installation to utilize SSDB instead of LDAP for short-lived data storage is done through the following process:

  1. Install SSDB and note the IP address and port configured since you will need this data for the next steps. Refer to ssdb-configuration-options.adoc for more information.

  2. Migrate any existing short-lived data to SSDB using the /opt/zimbra/bin/zmmigrateattrs command.

  3. Configure {product-name} to utilize SSDB.

Migration Procedure

  1. Access the command prompt on one of the machines in the installation.

  1. Migrate existing ephemeral data to the SSDB backend using the zmmigrateattrs utility

sudo su - zimbra
/opt/zimbra/bin/zmmigrateattrs ssdb:<ip address|hostname>:port # substituting your server values

You may use either an IP address or a hostname for the host portion of the destination URL. Either way, you will need to ensure it resolves to the proper IP address on all machines in the cluster. If the provided SSDB address does not resolve to a functioning backend, the migration process will terminate.

  1. Configure {product-name} to use SSDB:

sudo su - zimbra
zmprov mcf zimbraEphemeralBackendURL ssdb:<ip address|hostname>:port # substituting your server values

As with migration, the host and port must resolve to a functioning SSDB backend. Otherwise, the value of zimbraEphemeralBackendURL will not be changed.

Migration Details

Migration Info

Information about the latest migration process can be viewed by running the command zmmigrateattrs --status. If the migration is currently in progress, this command may have to be run from a new terminal window. This command will output three pieces of information:

  1. The status of the migration: one of IN_PROGRESS, COMPLETED or FAILED

  2. The URL of the SSDB backend acting as the destination

  3. A timestamp of when the migration process was initiated

The migration info can be reset with the command zmmigrateattrs --clear. This should only be done if the status does not reflect the true state of the system.

Changing the Ephemeral Backend URL

When the value of zimbraEphemeralBackendURL is modified, {product-name} checks the status of the last known migration. This can result in one of several scenarios:

  1. If the migration is completed and the URL of this migration matches the newly provided value, zimbraEphemeralBackendURL is changed to the new value and the migration info is reset. This is the expected use case.

  2. If a migration is currently in progress, zimbraEphemeralBackendURL will not be changed.

  3. If no migration info is available, the migration has failed, or the new URL does not match the migration URL, zimbraEphemeralBackendURL will be changed; however, a warning will be logged stating that data is not guaranteed to have been migrated.

Forwarding Ephemeral Data

During the migration process, and until the backend URL is changed, {product-name} will store new ephemeral data both in LDAP and SSDB; this keeps the two backends from getting out of sync. If the new value of zimbraEphemeralBackendURL is changed to match the migration URL, migration info is reset and the forwarding mechanism is turned off. If the values do not match, migration info is not reset, and forwarding remains in place. Note that this means that migration only needs to be run once, even if there is a gap between the initial migration and URL change. As long as the target backend is never taken offline, it will stay up-to-date. However, if SSDB is taken offline between the end of the migration and the backend URL change, migration will need to be re-run.

These scenarios are demonstrated below:

ephemeral data migration

Advanced Migration Options

The zmmigrateattrs tool provides several migration options, to be used with careful consideration:

  • The -r or --dry-run option outputs the changes to be made for each account to the console, without actually performing the migration.

  • The -n or --num-threads option specifies how many threads will be used for migration. Omitting this will result in migration happening synchronously.

  • The -a or --account option allows for migration of a comma-separated list of specific accounts. This should be used only for testing or debugging.

  • The -d or --debug option enables debug logging.

If no attribute names are explicitly passed in as arguments, migration will occur for all known ephemeral attributes, as in the example above.

Migration Limitations

Ephemeral data migration is a one-way process. The zmmigrateattrs script does not support migrating data from SSDB back into LDAP, nor does it support migrating data between different instances of SSDB. This means that if the value of zimbraEphemeralBackendURL is reverted back to LDAP after migration, prior authentication data will become inaccessible, and all user sessions will be invalidated. If migration to a new SSDB backend becomes necessary, the data should be replicated to the new location prior to changing the value of zimbraEphemeralBackendURL.

There is one exception to this is: the backend can be safely reverted back to LDAP immediately after the switch to SSDB with minimal loss of data. This is because the original values are retained in LDAP during migration; switching the backend to SSDB leaves a "snapshot" of ephemeral data in LDAP at the time of the switch. The migration utility does not currently provide a way to delete this data to free up space; however, it allows for the backend to be reverted. The more time passes between the initial change and the reversion, the less the LDAP snapshot will reflect the true state of ephemeral data.

Changes to zmprov

Due to changes in the way multi-valued ephemeral data is stored, the attributes zimbraAuthTokens and zimbraCsrfTokenData are no longer returned as part of the zmprov ga <account> response. The value of zimbraLastLogonTimestamp is returned as before, although only if the -l flag is not used, as adding the -l flag will restrict the server to accessing attributes in LDAP only. It is still possible to modify these attributes using the zmprov ma <account> command, regardless of the ephemeral backend. In order to do this, the provided attribute value must match its LDAP format: tokenId|expiration|serverVersion for auth tokens; data:crumb:expiration for CSRF tokens.

Migration CSV Output

Each run of zmmigrateattrs generates a CSV file in the /opt/zimbra/data/tmp/ folder. The file contains migration info for every migrated account, such as the number of attributes migrated. Note that it is possible for this to be zero, which can happen if all ephemeral data for an account is already present in the destination store.

If any migrations fail, a cutdown CSV file report detailing only the errors is also created in the same directory. The name(s) of the file(s) are logged at the end of the run.

Account Deletion Behavior

Ephemeral data deletion behavior differs slightly between SSDB and LDAP backends. With SSDB as the backend, account deletion results in the zimbraLastLogonTimestamp attribute being explicitly deleted from SSDB. zimbraAuthTokens and zimbraCsrfTokenData, however, are left to be expired by SSDB when the token lifetimes are reached (default of 2 days). Conversely, ephemeral data in LDAP is wiped immediately as part of the account deletion process.