Mediator Software Architecture

Current Mediator Software Architecture

current mediator architecture

In OpenLI 1.1.0, the mediator moved from a single-threaded, event loop driven architecture to a multi-threaded structure. This allows us to make better use of multiple cores on the mediator host, as well as minimise the impact of a slow handover for one agency on any other agencies that you are dealing with. The other benefit of the new architecture is that we now use RabbitMQ internally to buffer intercept records received from the collectors which we haven't yet been able to send to the agencies, so these records will no longer be lost if the mediator has to restart for some reason.

The core principles of the new architecture can be summarised as follows:

every collector that connects to the mediator is handled by a separate "receiving" thread (called "Col Recv" in the diagram). These threads simply receive ETSI records (either via a TCP connection, or by reading from a RabbitMQ instance running on the collector) and write them into the internal RabbitMQ instance running on the mediator.
each intercept will have two queues defined in the internal RabbitMQ: one for CCs and one for IRIs -- in the diagram, for the sake of brevity, I've represented each PAIR of queues as a single "ladder" named after the intercept target. In practice, we use the unique LIID for naming queues.
a collector thread that receives CCs for the "Roger" intercept will therefore write those CCs into the "Roger-CC" queue in RabbitMQ. IRIs for the "Sharon" intercept are written to the "Sharon-IRI" queue, etc.
every LEA that the mediator is connecting to also is handled by a separate "sending" thread (called "LEA Send" in the diagram). These threads set up and maintain the HI2 and HI3 handovers to their respective agency.
The main thread tells each LEA thread which intercepts it is responsible for, e.g. LEA X might be the intended recipient for the "Roger", "Peter" and "Lisa" intercepts.
Each LEA thread then subscribes to the corresponding RabbitMQ queues for the intercepts that they are responsible for, both CCs and IRIs.
The LEA thread continuously checks if the HI2 and HI3 handovers are able to receive data. If so, the thread will grab the next IRI or CC (depending on which handover is available) from its RabbitMQ subscription and forwards it on via the handover. If successful, the IRI/CC is popped from its corresponding RabbitMQ queue.
The pcap writer thread is a special case, which I will explain further down the page.

In short, collector receiver threads consume ETSI records from the collectors and put them into a RabbitMQ queue. LEA send threads then subscribe to the queues for the intercepts that they need to deliver and forward ETSI records from those queues to their agency as long as the handover is available. If a handover is unavailable for any length of time, the records simply queue up inside RabbitMQ and will persist on disk even if the mediator is halted.

Main Thread

Key source files:

src/mediator/mediator.c
src/mediator/mediator_prov.c

The main thread's role is to orchestrate the rest of the mediator threads. It runs a continuous event-driven epoll loop, which checks for file descriptor events and acts accordingly.

The key events include:

a new instruction from the provisioner
a connection attempt by a collector
the expiry of a timer indicating that we should try to reconnect to the provisioner (after the connection has failed)
the arrival of a signal that needs to be handled safely

Since these events are all likely to occur occasionally, the main thread is typically not very active.

The main thread maintains a communication channel to each of the collector receive and LEA send threads, so that it can send them further instructions (e.g. if there is a configuration or provisioning change that would affect them).

If a new collector connection attempt is observed, the main thread will accept the connection and then spawn a new collector receive thread for that collector.

If the provisioner announces a new agency, the main thread will spawn a new LEA send thread for that agency.

If the provisioner announces a new or modified LIID-to-agency mapping, the main thread will inform the affected LEA send threads about the mapping change.

If the provisioner sends an HI1 notification, the main thread will determine which intercept the notification is from and then pass it on to the corresponding LEA send thread.

Collector Receive Threads

Key source files:

src/mediator/coll_recv_thread.c
src/mediator/mediator_rmq.c

A collector receive thread begins with a setup phase that establishes a connection to the channel by which the collector is going to provide us with ETSI records. The activities in this phase will depend on whether the collector is delivering records via a RabbitMQ instance (in which case, the thread will connect to RabbitMQ) or via a TCP session (in which case, the thread will use the file descriptor created when the main thread accepted the collector's connection attempt). For TCP sessions, there will be further differences depending on whether the session is configured to use TLS.

Once the setup is complete, the receive thread will run a continuous event-driven epoll loop, similar to that operated by the main thread but a lot busier.

The events that occur in the collector receive epoll loop include:

the arrival of encoded ETSI records, either via RabbitMQ or the TCP session with the collector.
the arrival of instructions from the main thread.
the expiry of a timer that triggers the expiry of local state for any inactive intercepts.
the continuation of a partially completed TLS handshake for the TCP session with the collector.

Encoded ETSI records received by a collector thread are published into the corresponding internal RabbitMQ queue based on the LIID for the record and whether the record is a CC or IRI record.

LEA Send Threads

Key source files:

src/mediator/lea_send_thread.c
src/mediator/handover.c
src/mediator/mediator_rmq.c
src/mediator/liidmapping.c

Handovers

A handover is simply a TCP session with an LEA that is used for transmitting the ETSI-encoded records for an intercept to the requesting agency. There are two types of handovers: HI2, for sending IRIs, and HI3, for sending CCs. Each agency will require both handovers.

One requirement in the ETSI standards is that the mediator must regularly send keep-alive messages on each handover so that the LEA can confirm that the session is still up. The time between keep-alives is configurable on a per-agency basis in OpenLI, and keep-alives can also be disabled if required.

The standards also expect that agencies will send keep-alive responses back to the mediator to confirm that the reverse path is functioning as well. An OpenLI mediator can be configured to drop a handover session if a keep-alive response is not seen in a timely manner, but it is our experience that some LEA equipment may not send responses so this feature may need to be disabled for some agencies.

Pcap Thread

Key source files:

src/mediator/pcapthread.c
src/mediator/mediator_rmq.c

In some circumstances, it may be necessary to deliver the intercept as a pcap file rather than using the standard ETSI handovers. The pcap thread is a special type of LEA send thread that performs this role for intercepts that have been assigned to the special agency pcapdisk in their OpenLI configuration.

Instead of maintaining TCP handover sessions, the pcap thread has writable file handles for each intercept that it is responsible for. Whenever an intercept record is available in the RabbitMQ queue for one of those intercepts, the pcap thread will consume the record from the queue, convert the payload portion of the record to a pcap packet and write that packet to the corresponding file handle. The pcap files for each intercept are rotated at a user-specified frequency and completed files can be delivered to the agency as soon as they have been rotated out.

The pcap thread receives the same types of instructions from the main thread as any other LEA send thread, so changes to intercept mappings will also be passed on to the pcap thread.