Shared DB use cases

[cam-schultz]

Background

awm-relayer supports being used in a distributed deployment in which many instances access a single shared DB. This was done as a way to share infrastructure costs for an application with very lightweight persisted state. Each awm-relayer instance accesses a set of keys pertaining to the logical "application relayers" derived from its config. This supports use cases in which any given awm-relayer worker can pick up where another instance left off for the same application relayer.

Since the DB only stored the latest checkpointed block for source chains, the only important parameters that define an application relayer and its associated DB key are:

1. The sending address (i.e. the address that called TeleporterMessenger)
2. The source blockchain

The sending address may be either an address directly specified in the allowed-origin-sender-addresses config, or a "wildcard" address if that config is empty. In practice, this wildcard is the zero address.

Motivation

Enumerate any use cases, the current level of support for them, and any modifications needed to better support them. Use cases that are 1) not well supported and 2) whose modifications to better support don't further complicate the application configuration will be considered for implementation.

Use case 1: Start from the last block processed by an application relayer

Consider the following scenario:

1. Addresses A and B are added to allowed-origin-sender-addresses (note: either A or B could be the "wildcard" address to allow all addresses, which is the case when no addresses are specified)
2. The relayer runs and relays all messages for A and B up through block 100.
3. Address A removed from allowed-origin-sender-addresses.
4. The relayer runs and relays all messages for B up through block 200.
5. Address B is removed from allowed-origin-sender-addresses.

On restart, the application relayer for Address A will process blocks 100-200, since that's what's at that application relayer's DB key. If a message was was sent from Address A at block 150, it would have been explicitly blocked in step 4, but processed in step 5.

Support

This is currently supported by specifying process-missed-block=true.

Use case 2: Start from the last block processed by any application relayer

Consider the same scenario as Use Case 1, but in step 5, the Address A application relayer resumes from the latest seen block for all application relayers for that source chain. As a result, the message that was blocked in step 4 is not processed in step 5. This use case is relevant for users who wish to always process from the last seen block on a source chain, but who don't want to disable catch-up altogether.

Support

This use case is not well supported at present, and requires an additional restart. To do so, do the following in before step 5 in Use Case 1

1. Restart the relayer with process-missed-blocks=false. This will force the DB to update to the chain tip
2. Restart the relayer again with process-missed-blocks=true, and any changes to allowed-origin-sender-addresses

Alternative

The application can directly support this use case by writing to an application relayer key that only includes the source chain, and does not include the sender address. This key would be accessed by all application relayers that correspond to that source chain, meaning that key would store the latest height that has been processed by any application relayer.

A new configuration option start-from-last-seen would cause application relayers to read from that new key, rather than its distinct key, on startup. This does, however, risk gaps in processing if a separate awm-relayer instance writes to that key during downtime.