Status: Beta
Note: this document requires a simplification pass to reduce the scope, size and complexity.
Table of Contents
- Introduction
- Communication Model
- Operation
- Status Reporting
- Connection Settings Management
- OpAMP Connection Setting Offer Flow
- Trust On First Use
- Registration On First Use
- Revoking Access
- Certificate Generation
- Connection Settings for "Other" Destinations
- ConnectionSettingsOffers Message
- OpAMPConnectionSettings
- TelemetryConnectionSettings
- OtherConnectionSettings
- Headers Message
- TLSCertificate Message
- Own Telemetry Reporting
- Configuration
- Packages
- Connection Management
- Security
- Interoperability
- Performance and Scale
- FAQ for Reviewers
- Future Possibilities
- References
Open Agent Management Protocol (OpAMP) is a network protocol for remote management of large fleets of data collection Agents.
OpAMP allows Agents to report their status to and receive configuration from a Server and to receive Agent installation package updates from the Server. The protocol is vendor-agnostic, so the Server can remotely monitor and manage a fleet of different Agents that implement OpAMP, including a fleet of mixed Agents from different vendors.
OpAMP supports the following functionality:
- Remote configuration of the Agents.
- Status reporting. The protocol allows the Agent to report the properties of the Agent such as its type and version or the operating system type and version it runs on. The status reporting also allows the management Server to tailor the remote configuration to individual Agents or types of Agents.
- Agent's own telemetry reporting to an OTLP-compatible backend to monitor Agent's process metrics such as CPU or RAM usage, as well as Agent-specific metrics such as rate of data processing.
- Management of downloadable Agent-specific packages.
- Secure auto-updating capabilities (both upgrading and downgrading of the Agents).
- Connection credentials management, including client-side TLS certificate revocation and rotation.
The functionality listed above enables a 'single pane of glass' management view of a large fleet of mixed Agents (e.g. OpenTelemetry Collector, Fluentd, etc).
The OpAMP Server manages Agents that provide a client-side implementation of OpAMP protocol, further referred to as OpAMP Client or simply Client. OpAMP does not assume any particular relationship between the Agent and the Client. The Client can be run as a separate process with a different lifecycle than the Agent, a sidecar, a plugin, or be fully integrated into the Agent code.
The Agents can optionally send their own telemetry to an OTLP destination when directed so by the OpAMP Server. The Agents likely also connect to other destinations, where they send the data they collect:
┌────────────┬────────┐ ┌─────────┐
│ │ OpAMP │ OpAMP │ OpAMP │
│ │ ├──────────►│ │
│ │ Client │ │ Server │
│ └────────┤ └─────────┘
│ │
│ ┌────────┤ ┌─────────┐
│ │OTLP │ OTLP/HTTP │ OTLP │
│ Agent │ ├──────────►│ │
│ │Exporter│ │ Receiver│
│ └────────┤ └─────────┘
│ │
│ ┌────────┤
│ │Other ├──────────► Other
│ │Clients ├──────────► Destinations
└────────────┴────────┘
This specification defines the OpAMP network protocol and the expected behavior for OpAMP Agents and Servers. The OTLP/HTTP protocol is specified here. The protocols used by the Agent to connect to other destinations are Agent type-specific and are outside the scope of this specification.
OpAMP protocol works over one of the 2 supported transports: plain HTTP connections and WebSocket connections. Server implementations SHOULD accept both plain HTTP connections and WebSocket connections. OpAMP Client implementations may choose to support either plain HTTP or WebSocket transport, depending on their needs.
OpAMP Clients connect to OpAMP Server on behalf of the Agents. Typically a single Server accepts connections from many Clients. Agents are identified by self-assigned or Server-assigned globally unique instance identifiers (or instance_uid for short). The instance_uid is recorded in each message sent from the Agent (via the Client) to the Server and from the Server to the Agent.
The default URL path for the connection is /v1/opamp. The URL path MAY be configurable on the Client and on the Server.
One of the typical ways to implement OpAMP on the Agent side is by having a helping Supervisor process, which controls the Agent process. The Supervisor is also typically responsible for communicating with the OpAMP Server:
┌───────────────────┐
│ Supervisor │
│ │
│ ┌────────┤ ┌────────┐
│ │ OpAMP │ OpAMP │ OpAMP │
│ │ ├──────────►│ │
│ │ Client │ │ Server │
└────┬─────┴────────┘ └────────┘
│
│
▼
┌──────────┐
│ │
│ Agent │
│ │
└──────────┘
The OpAMP specification does not require using a Supervisor and does not define how the Supervisor and the Agent communicate.
In the rest of the OpAMP specification the term Agent is used to refer to the entity which implements the client portion of the OpAMP, regardless of whether or not a Supervisor is part of that entity.
One of the supported transports for OpAMP protocol is
WebSocket. The OpAMP Client
is a WebSocket client and the Server is a WebSocket Server. The Client and the Server
communicate using binary data WebSocket messages. The content of each WebSocket
message is an encoded header, followed by a
binary encoded Protobuf
message data (see WebSocket Message Format).
On behalf of the Agent, the Client sends AgentToServer message data and the Server sends ServerToAgent Protobuf message data:
┌────────────\ \────────┐ ┌──────────────┐
│ / / │ Data:AgentToServer │ │
│ \ \ OpAmp ├───────────────────────►│ │
│ Agent / / │ │ Server │
│ \ \ Client │ Data:ServerToAgent │ │
│ / / │◄───────────────────────┤ │
└────────────\ \────────┘ └──────────────┘
The format of each WebSocket message is the following:
┌────────────┬────────────────────────────────────────┬───────────────────┐
│ header │ Varint encoded unsigned 64 bit integer │ 1-10 bytes │
├────────────┼────────────────────────────────────────┼───────────────────┤
│ data │ Encoded Protobuf message, │ 0 or more bytes │
│ │ either AgentToServer or ServerToAgent │ │
└────────────┴────────────────────────────────────────┴───────────────────┘
The unencoded header is a 64 bit unsigned integer. In the WebSocket message the 64 bit
unencoded header value is encoded into bytes using Base 128 Varint format. The
number of the bytes that the encoded header uses depends on the value of unencoded
header and can be anything between 1 and 10 bytes.
The value of the unencoded header is set equal to 0 in this version of the specification.
All other header values are reserved for future use. Such values will be defined in
future versions of OpAMP specification. OpAMP WebSocket message decoders that are
compliant with this specification SHOULD check that the value of the header is equal
to 0 and if it is not SHOULD assume that the WebSocket message is malformed.
The data field contains the bytes that represent the AgentToServer or ServerToAgent
message encoded in Protobuf binary wire format.
Note that both header and data fields contain a variable number of bytes.
The decoding Base 128 Varint algorithm for the header knows when to stop based on the
bytes it reads.
To decode the data field using Protobuf decoding logic the implementation needs
to know the number of the bytes of the data field. To calculate this the implementation
MUST deduct the size of the header in bytes from the size of the WebSocket message
in bytes.
Note that due to the way Protobuf wire format is designed the size of the data in
bytes can be 0 if the encoded AgentToServer or ServerToAgent message is empty (i.e. all
fields are unset). This is a valid situation.
OpAMP over WebSocket is an asynchronous, full-duplex message exchange protocol. The order and sequence of messages exchanged by the OpAMP Client and the Server is defined for each particular capability in the corresponding section of this specification.
The sequence is normally started by an initiating message triggered by some external event. For example after the connection is established the Client sends an AgentToServer message. In this case the "connection established" is the triggering event and the AgentToServer is the initiating message.
Both the Client and the Server may begin a sequence by sending an initiating message.
The initiating message may trigger the recipient to send one or more messages back, which in turn may trigger messages in the opposite direction and so on. This exchange of messages in both directions continues until the sequence is over because the goal of the exchange is achieved or the sequence failed with an error.
Note that the same message may in some cases be the initiating message of the sequence and in some other cases it may be triggered in response to receiving some other message. Unlike other protocols in OpAMP there is no strict separation between "request" and "response" messages types. The role of the message depends on how the sequence is triggered.
For example the AgentToServer message may be the initiating message sent by the Client when the Client connects to the Server for the first time. The AgentToServer message may also be sent by the Client in response to the Server making a remote configuration offer to the Agent and Agent reporting that it accepted the configuration.
See sections under the Operation section for the details of the message sequences.
The WebSocket transport is typically used when it is necessary to have instant communication ability from the Server to the Agent without waiting for the Client to poll the Server like it is done when using the HTTP transport (see below).
The second supported transport for OpAMP protocol is plain HTTP connection. The OpAMP Client is an HTTP client and the Server is an HTTP server. The Client makes POST requests to the Server. The body of the POST request and response is a binary serialized Protobuf message. The Client sends AgentToServer Protobuf messages in the request body and the Server sends ServerToAgent Protobuf messages in the response body.
OpAMP over HTTP is a synchronous, half-duplex message exchange protocol. The Client initiates an HTTP request when it has an AgentToServer message to deliver. The Server responds to each HTTP request with a ServerToAgent message it wants to deliver to the Agent. If the Agent has nothing to deliver to the Server the Client MUST periodically poll the Server by sending an AgentToServer message where only instance_uid field is set. This gives the Server an opportunity to send back in the response any messages that the Server wants to deliver to the Agent (such as for example a new remote configuration).
The default polling interval when the Agent does not have anything to deliver is 30 seconds. This polling interval SHOULD be configurable on the Client.
When using HTTP transport the sequence of messages is exactly the same as it is when using the WebSocket transport. The only difference is in the timing:
- When the Server wants to send a message to the Agent, the Server needs to wait for the Client to poll the Server and establish an HTTP request over which the Server's message can be sent back as an HTTP response.
- When the Agent wants to send a message to the Server and the Agent has previously sent a request to the Server that is not yet responded, the Client MUST wait until the response is received before a new request can be made. Note that the new request in this case can be made immediately after the previous response is received, the Client does not need to wait for the polling period between requests.
The Client MUST set "Content-Type: application/x-protobuf" request header when using plain HTTP transport. When the Server receives an HTTP request with this header set it SHOULD assume this is a plain HTTP transport request, otherwise it SHOULD assume this is a WebSocket transport initiation.
The Client MAY compress the request body using gzip method and MUST specify "Content-Encoding: gzip" in that case. Server implementations MUST honour the "Content-Encoding" header and MUST support gzipped or uncompressed request bodies.
The Server SHOULD compress the response if the Client indicated it can accept compressed response via the "Accept-Encoding" header.
The body of the OpAMP WebSocket message or HTTP body of the request is a binary serialized Protobuf message AgentToServer as defined below (all messages in this document are specified in Protobuf 3 language):
message AgentToServer {
string instance_uid = 1;
uint64 sequence_num = 2;
AgentDescription agent_description = 3;
uint64 capabilities = 4;
AgentHealth health = 5;
EffectiveConfig effective_config = 6;
RemoteConfigStatus remote_config_status = 7;
PackageStatuses package_statuses = 8;
AgentDisconnect agent_disconnect = 9;
uint64 flags = 10;
}The Server should process each field as it is described in the corresponding Operation section.
The instance_uid field is a globally unique identifier of the running instance of the Agent. The Agent SHOULD self-generate this identifier and make the best effort to avoid creating an identifier that may conflict with identifiers created by other Agents. The instance_uid SHOULD remain unchanged for the lifetime of the Agent process. The instance_uid MUST be a ULID formatted as a 26 character string in canonical representation.
In case the Agent wants to use an identifier generated by the Server, the field SHOULD be set with a temporary value and RequestInstanceUid flag MUST be set.
The sequence number is incremented by 1 for every AgentToServer message sent by the Client. This allows the Server to detect that it missed a message when it notices that the sequence_num is not exactly by 1 greater than the previously received one. See Agent Status Compression for more details.
Data that describes the Agent, its type, where it runs, etc. See AgentDescription message for details. This field SHOULD be unset if this information is unchanged since the last AgentToServer message.
Bitmask of flags defined by AgentCapabilities enum. All bits that are not defined in AgentCapabilities enum MUST be set to 0 by the Client. This allows extending the protocol and the AgentCapabilities enum in the future such that old Agents automatically report that they don't support the new capability. This field MUST be always set.
enum AgentCapabilities {
// The capabilities field is unspecified.
UnspecifiedAgentCapability = 0;
// The Agent can report status. This bit MUST be set, since all Agents MUST
// report status.
ReportsStatus = 0x00000001;
// The Agent can accept remote configuration from the Server.
AcceptsRemoteConfig = 0x00000002;
// The Agent will report EffectiveConfig in AgentToServer.
ReportsEffectiveConfig = 0x00000004;
// The Agent can accept package offers.
AcceptsPackages = 0x00000008;
// The Agent can report package status.
ReportsPackageStatuses = 0x00000010;
// The Agent can report own trace to the destination specified by
// the Server via ConnectionSettingsOffers.own_traces field.
ReportsOwnTraces = 0x00000020;
// The Agent can report own metrics to the destination specified by
// the Server via ConnectionSettingsOffers.own_metrics field.
ReportsOwnMetrics = 0x00000040;
// The Agent can report own logs to the destination specified by
// the Server via ConnectionSettingsOffers.own_logs field.
ReportsOwnLogs = 0x00000080;
// The can accept connections settings for OpAMP via
// ConnectionSettingsOffers.opamp field.
AcceptsOpAMPConnectionSettings = 0x00000100;
// The can accept connections settings for other destinations via
// ConnectionSettingsOffers.other_connections field.
AcceptsOtherConnectionSettings = 0x00000200;
// The Agent can accept restart requests.
AcceptsRestartCommand = 0x00000400;
// The Agent will report Health via AgentToServer.health field.
ReportsHealth = 0x00000800;
// The Agent will report RemoteConfig status via AgentToServer.remote_config_status field.
ReportsRemoteConfig = 0x00001000;
// Add new capabilities here, continuing with the least significant unused bit.
}The current health of the Agent. See AgentHealth message. May be omitted if nothing changed since last AgentToServer message.
The current effective configuration of the Agent. The effective configuration is the one that is currently used by the Agent. The effective configuration may be different from the remote configuration received from the Server earlier, e.g. because the Agent uses a local configuration instead (or in addition). See EffectiveConfig message for details. This field SHOULD be unset if this information is unchanged since the last AgentToServer message.
The status of the remote config that was previously received from the Server. See RemoteConfigStatus message for details. This field SHOULD be unset if this information is unchanged since the last AgentToServer message.
The list of the Agent packages, including package statuses. This field SHOULD be unset if this information is unchanged since the last AgentToServer message.
AgentDisconnect MUST be set in the last AgentToServer message sent from the Client to the Server.
Bit flags as defined by AgentToServerFlags bit masks.
enum AgentToServerFlags {
FlagsUnspecified = 0;
// Flags is a bit mask. Values below define individual bits.
// The Agent requests Server go generate a new instance_uid, which will
// be sent back in ServerToAgent message
RequestInstanceUid = 0x00000001;
}The body of the WebSocket message or HTTP response body is a binary serialized Protobuf message ServerToAgent.
ServerToAgent message is sent from the Server to the Agent either in response to the AgentToServer message or when the Server has data to deliver to the Agent.
If the Server receives an AgentToServer message and the Server has no data to send back to the Agent then ServerToAgent message will still be sent, but all fields except instance_uid will be unset (in that case ServerToAgent serves simply as an acknowledgement of receipt).
Upon receiving a ServerToAgent message the Agent MUST process it. The processing that needs to be performed depends on what fields in the message are set. For details see links to the corresponding sections of this specification from the field descriptions below.
As a result of this processing the Agent may need to send status reports to the Server. The Agent is free to perform all the processing of the ServerToAgent message completely and then send one status report or it may send multiple status reports as it processes the portions of ServerToAgent message to indicate the progress (see e.g. Downloading Packages). Multiple status reports may be desirable when processing takes a long time, in which case the status reports allow the Server to stay informed.
Note that the Server will reply to each status report with a ServerToAgent message (or with an ServerErrorResponse if something goes wrong). These ServerToAgent messages may have the same content as the one received earlier or the content may be different if the situation on the Server has changed. The Agent SHOULD be ready to process these additional ServerToAgent messages as they arrive.
The Client SHOULD NOT send any status reports at all if the status of the Agent did not change as a result of processing.
The ServerToAgent message has the following structure:
message ServerToAgent {
string instance_uid = 1;
ServerErrorResponse error_response = 2;
AgentRemoteConfig remote_config = 3;
ConnectionSettingsOffers connection_settings = 4;
PackagesAvailable packages_available = 5;
uint64 flags = 6;
uint64 capabilities = 7;
AgentIdentification agent_identification = 8;
ServerToAgentCommand command = 9;
}The Agent instance identifier. MUST match the instance_uid field previously received in the AgentToServer message. When communication with multiple Agents is multiplexed into one WebSocket connection (for example when a terminating proxy is used) the instance_uid field allows to distinguish which Agent the ServerToAgent message is addressed to.
Note: the value can be overriden by Server by sending a new one in the AgentIdentification field. When this happens then Agent MUST update its instance_uid to the value provided and use it for all further communication.
error_response is set if the Server wants to indicate that something went wrong during processing of an AgentToServer message. If error_response is set then all other fields below must be unset and vice versa, if any of the fields below is set then error_response must be unset.
This field is set when the Server has a remote config offer for the Agent. See Configuration for details.
This field is set when the Server wants the Agent to change one or more of its client connection settings (destination, headers, certificate, etc). See Connection Settings Management for details.
This field is set when the Server has packages to offer to the Agent. See Packages for details.
Bit flags as defined by ServerToAgentFlags bit masks.
Report* flags can be used by the Server if the Client did not include the
particular portion of the data in the last AgentToServer message (which is an allowed
compression approach) but the Server does not have that data, e.g. the Server was
restarted and lost the agent status (see the details in
this section).
enum Flags {
FlagsUnspecified = 0;
// Flags is a bit mask. Values below define individual bits.
// ReportFullState flag can be used by the Server if the Client did not include
// some sub-message in the last AgentToServer message (which is an allowed
// optimization) but the Server detects that it does not have it (e.g. was
// restarted and lost state). The detection happens using
// AgentToServer.sequence_num values.
// The Server asks the Agent to report the full status again by sending
// a new, full AgentToServer message.
ReportFullState = 0x00000001;
}Bitmask of flags defined by ServerCapabilities enum. All bits that are not defined in ServerCapabilities enum MUST be set to 0 by the Server. This allows extending the protocol and the ServerCapabilities enum in the future such that old Servers automatically report that they don't support the new capability. This field MUST be set in the first ServerToAgent sent by the Server and MAY be omitted in subsequent ServerToAgent messages by setting it to UnspecifiedServerCapability value.
enum ServerCapabilities {
// The capabilities field is unspecified.
UnspecifiedServerCapability = 0;
// The Server can accept status reports. This bit MUST be set, since all Server
// MUST be able to accept status reports.
AcceptsStatus = 0x00000001;
// The Server can offer remote configuration to the Agent.
OffersRemoteConfig = 0x00000002;
// The Server can accept EffectiveConfig in AgentToServer.
AcceptsEffectiveConfig = 0x00000004;
// The Server can offer Packages.
OffersPackages = 0x00000008;
// The Server can accept Packages status.
AcceptsPackagesStatus = 0x00000010;
// The Server can offer connection settings.
OffersConnectionSettings = 0x00000020;
// Add new capabilities here, continuing with the least significant unused bit.
}Properties related to identification of the Agent, which can be overriden by the Server if needed. When new_instance_uid is set, Agent MUST update instance_uid to the value provided and use it for all further communication. The new_instance_uid MUST be a ULID formatted as a 26 character string in canonical representation.
message AgentIdentification {
string new_instance_uid = 1;
}This field is set when the Server wants the Agent to perform a restart. This field must not be set with other fields besides instance_uid or capabilities. All other fields will be ignored and the Agent will execute the command. See ServerToAgentCommand Message for details.
The message has the following structure:
message ServerErrorResponse {
enum Type {
UNKNOWN = 0;
BAD_REQUEST = 1;
UNAVAILABLE = 2
}
Type type = 1;
string error_message = 2;
oneof Details {
RetryInfo retry_info = 3;
}
}This field defines the type of the error that the Server encountered when trying to process the Agent's request. Possible values are:
UNKNOWN: Unknown error. Something went wrong, but it is not known what exactly. The error_message field may contain a description of the problem.
BAD_REQUEST: Only sent as a response to a previously received AgentToServer message and indicates that the AgentToServer message was malformed. See Bad Request processing.
UNAVAILABLE: The Server is overloaded and unable to process the request. See Throttling.
Error message, typically human readable.
Additional RetryInfo message about retrying if type==UNAVAILABLE.
The message has the following structure:
// ServerToAgentCommand is sent from the Server to the Agent to request that the Agent
// perform a command.
message ServerToAgentCommand {
enum CommandType {
// The Agent should restart. This request will be ignored if the Agent does not
// support restart.
Restart = 0;
}
CommandType type = 1;
}The ServerToAgentCommand message is sent when the Server wants the Agent to restart. This message must only contain the command, instance_uid, and capabilities fields. All other fields will be ignored.
The Client MUST send a status report:
- First time immediately after connecting to the Server. The status report MUST be the first message sent by the Client.
- Subsequently, every time the status of the Agent changes.
The status report is sent as an AgentToServer message. The following fields in the message can be set to reflect the corresponding part of the status: agent_description, capabilities, health, effective_config, remote_config_status, package_statuses.
The Server MUST respond to the AgentToServer message by sending a ServerToAgent message.
If the status report processing failed then the error_response field MUST be set to ServerErrorResponse message.
If the status report is processed successfully by the Server then the error_response field MUST be unset and the other fields can be populated as necessary.
Here is the sequence diagram that shows how status reporting works (assuming server-side processing is successful):
Client Server
│ │
│ │
│ WebSocket Connect │
├──────────────────────────────────────►│
│ │
│ AgentToServer │ ┌─────────┐
├──────────────────────────────────────►├──►│ │
│ │ │ Process │
│ ServerToAgent │ │ Status │
│◄──────────────────────────────────────┤◄──┤ │
│ │ └─────────┘
. ... .
│ AgentToServer │ ┌─────────┐
├──────────────────────────────────────►├──►│ │
│ │ │ Process │
│ ServerToAgent │ │ Status │
│◄──────────────────────────────────────┤◄──┤ │
│ │ └─────────┘
│ │
Note that the status of the Agent may change as a result of receiving a message from the Server. For example the Server may send a remote configuration to the Agent. Once the Agent processes such a request the Agent's status changes (e.g. the effective configuration of the Agent changes). Such status change should result in the Client sending a status report to the Server.
So, essentially in such cases the sequence of messages may look like this:
Agent Client Server
│ │ ServerToAgent │
┌───────┤◄──────────────────────────────────────┤
│ │ │
▼ │ │
┌────────┐ │ │
│Process │ │ │
│Received│ │ │
│Data │ │ │
└───┬────┘ │ │
│ │ │
│Status │ │
│Changed│ AgentToServer │ ┌─────────┐
└──────►├──────────────────────────────────────►├──►│ │
│ │ │ Process │
│ ServerToAgent │ │ Status │
│◄──────────────────────────────────────┤◄──┤ │
│ │ └─────────┘
When the Client receives a ServerToAgent message the Client MUST NOT send a status report unless processing of the message received from the Server resulted in actual change of the Agent status (e.g. the configuration of the Agent has changed). The sequence diagram in this case look like this:
Agent Client Server
│ │ ServerToAgent │
┌──────┤◄──────────────────────────────────────┤
│ │ │
▼ │ │
┌────────┐ │ │
│Process │ │ │
│Received│ │ │
│Data │ │ │
└───┬────┘ │ │
│ │ │
▼ │ │
No Status │ │
Changes │ │
│ │
│ │
Important: if the Client does not follow these rules the operation may result in an infinite loop of messages sent back and forth between the Client and the Server.
The Client notifies the Server about Agent's status by sending AgentToServer messages. The status for example includes the Agent description, its effective configuration, the status of the remote configuration it received from the Server and the status of the packages. The Server tracks the status of the Agent using the data specified in the sub-messages referenced from AgentToServer message.
The Client MAY compress the AgentToServer message by omitting the sub-messages that have not changed since that particular data was reported last time. The following sub-messages can be subject to such compression: AgentDescription, AgentHealth, EffectiveConfig, RemoteConfigStatus and PackageStatuses.
The compression is done by omitting the sub-message in the AgentToServer message. If any of the fields in the sub-message has changed then the compression cannot be used for that particular sub-message and the sub-message with all its relevant fields MUST be present.
If all AgentToServer messages are reliably delivered to the Server and the Server correctly processes them then such compression is safe and the Server should always have the correct latest status of the Agent.
However, it is possible that the Client and Server lose the synchronization and the Client believes the Server has the latest data while in reality the Server doesn't. This is possible for example if the Server is restarted while the Client keeps running and sends AgentToServer messages, which the Server does not receive because it is temporarily down.
In order to detect this situation and recover from it, the AgentToServer message contains a sequence_num field. The field is an integer number that is incremented every time the Client has a new AgentToServer message to send.
When the Server receives an AgentToServer message sequence_num field value that is not exactly by one greater than the previously received sequence_num value then the Server knows it does not have full status of the AgentToServer message.
When this situation is encountered, to recover the lost status the Server MUST request
the Agent to report the omitted data. To make this request the Server MUST send
a ServerToAgent message to the Agent and set the ReportFullState bit in
the flags field of ServerToAgent message.
The AgentDescription message has the following structure:
message AgentDescription {
repeated KeyValue identifying_attributes = 1;
repeated KeyValue non_identifying_attributes = 2;
}Attributes that identify the Agent.
Keys/values are according to OpenTelemetry semantic conventions, see: https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/resource/semantic_conventions
For standalone running Agents (such as OpenTelemetry Collector) the following attributes SHOULD be specified:
- service.name should be set to the same value that the Agent uses in its own telemetry.
- service.namespace if it is used in the environment where the Agent runs.
- service.version should be set to version number of the Agent build.
- service.instance.id should be set. It may be be set equal to the Agent's instance uid (equal to ServerToAgent.instance_uid field) or any other value that uniquely identifies the Agent in combination with other attributes.
- any other attributes that are necessary for uniquely identifying the Agent's own telemetry.
The Agent SHOULD also include these attributes in the Resource of its own telemetry. The combination of identifying attributes SHOULD be sufficient to uniquely identify the Agent's own telemetry in the destination system to which the Agent sends its own telemetry.
Attributes that do not necessarily identify the Agent but help describe where it runs.
The following attributes SHOULD be included:
- os.type, os.version - to describe where the Agent runs.
- host.* to describe the host the Agent runs on.
- cloud.* to describe the cloud where the host is located.
- any other relevant Resource attributes that describe this Agent and the environment it runs in.
- any user-defined attributes that the end user would like to associate with this Agent.
The AgentHealth message has the following structure:
message AgentHealth {
bool healthy = 1;
fixed64 start_time_unix_nano = 2;
string last_error = 3;
}Set to true if the Agent is up and healthy.
Timestamp since the Agent is up, i.e. when the agent was started. Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January 1970. If the agent is not running MUST be set to 0.
Human-readable error message if the Agent is in erroneous state. SHOULD be set when healthy==false.
The EffectiveConfig message has the following structure:
message EffectiveConfig {
AgentConfigMap config_map = 1;
}The effective config of the Agent.
See AgentConfigMap message definition in the Configuration section.
The RemoteConfigStatus message has the following structure:
message RemoteConfigStatus {
bytes last_remote_config_hash = 1;
enum Status {
// The value of status field is not set.
UNSET = 0;
// Remote config was successfully applied by the Agent.
APPLIED = 1;
// Agent is currently applying the remote config that it received earlier.
APPLYING = 2;
// Agent tried to apply the config received earlier, but it failed.
// See error_message for more details.
FAILED = 3;
}
Status status = 2;
string error_message = 3;
}The hash of the remote config that was last received by this Agent in the AgentRemoteConfig.config_hash field. The Server SHOULD compare this hash with the config hash it has for the Agent and if the hashes are different the Server MUST include the remote_config field in the response in the ServerToAgent message.
The status of the Agent's attempt to apply a previously received remote configuration.
Optional error message if status==FAILED.
The PackageStatuses message describes the status of all packages that the Agent has or was offered. The message has the following structure:
message PackageStatuses {
map<string, PackageStatus> packages = 1;
bytes server_provided_all_packages_hash = 2;
string error_message = 3;
}A map of PackageStatus messages, where the keys are package names. The key MUST match the name field of PackageStatus message.
The aggregate hash of all packages that this Agent previously received from the Server via PackagesAvailable message.
The Server SHOULD compare this hash to the aggregate hash of all packages that it has for this Agent and if the hashes are different the Server SHOULD send an PackagesAvailable message to the Agent.
This field is set if the Agent encountered an error when processing the PackagesAvailable message and that error is not related to any particular single package.
The field must be unset is there were no processing errors.
The PackageStatus has the following structure:
message PackageStatus {
string name = 1;
string agent_has_version = 2;
bytes agent_has_hash = 3;
string server_offered_version = 4;
bytes server_offered_hash = 5;
enum Status {
INSTALLED = 0;
INSTALLING = 1;
INSTALL_FAILED = 2;
}
Status status = 6;
string error_message = 7;
}Package name. MUST be always set and MUST match the key in the packages field of PackageStatuses message.
The version of the package that the Agent has.
MUST be set if the Agent has this package.
MUST be empty if the Agent does not have this package. This may be the case for example if the package was offered by the Server but failed to install and the Agent did not have this package previously.
The hash of the package that the Agent has.
MUST be set if the Agent has this package.
MUST be empty if the Agent does not have this package. This may be the case for example if the package was offered by the Server but failed to install and the Agent did not have this package previously.
The version of the package that the Server offered to the Agent.
MUST be set if the installation of the package is initiated by an earlier offer from the Server to install this package.
MUST be empty if the Agent has this package but it was installed locally and was not offered by the Server.
Note that it is possible for both agent_has_version and server_offered_version fields to be set and to have different values. This is for example possible if the Agent already has a version of the package successfully installed, the Server offers a different version, but the Agent fails to install that version.
The hash of the package that the Server offered to the Agent.
MUST be set if the installation of the package is initiated by an earlier offer from the Server to install this package.
MUST be empty if the Agent has this package but it was installed locally and was not offered by the Server.
Note that it is possible for both agent_has_hash and server_offered_hash fields to be set and to have different values. This is for example possible if the Agent already has a version of the package successfully installed, the Server offers a different version, but the Agent fails to install that version.
The status of this package. The possible values are:
INSTALLED: Package is successfully installed by the Agent. The error_message field MUST NOT be set.
INSTALLING: Agent is currently downloading and installing the package. server_offered_hash field MUST be set to indicate the version that the Agent is installing. The error_message field MUST NOT be set.
INSTALL_FAILED: Agent tried to install the package but installation failed. server_offered_hash field MUST be set to indicate the version that the Agent tried to install. The error_message may also contain more details about the failure.
An error message if the status is erroneous.
OpAMP includes features that allow the Server to manage Agent's connection settings for all of the destinations that the Agent connects to, as well as the OpAMP Client's connection settings.
The following diagram shows a typical Agent that is managed by OpAMP Servers, sends its own telemetry to an OTLP backend and also connects to other destinations to perform its work:
┌────────────┬────────┐ ┌─────────┐
│ │ OpAMP │ OpAMP │ OpAMP │
│ │ ├──────────►│ │
│ │ Client │ │ Server │
│ └────────┤ └─────────┘
│ │
│ ┌────────┤ ┌─────────┐
│ │OTLP │ OTLP/HTTP │OTLP │
│ Agent │ ├──────────►│Telemetry│
│ │Client │ │Backend │
│ └────────┤ └─────────┘
│ │
│ ┌────────┤
│ │Other ├──────────► Other
│ │ ├──────────►
│ │Clients ├──────────► Destinations
└────────────┴────────┘
When connecting to the OpAMP Server and to other destinations it is typically expected that Agents (or OpAMP Clients connecting on Agent's behalf) will use some sort of header-based authorization mechanism (e.g. an "Authorization" HTTP header or an access token in a custom header) and optionally also client-side certificates for TLS connections (also known as mutual TLS).
OpAMP protocol allows the Server to offer settings for each of these connections and for the Agent to accept or reject such offers. This mechanism can be used to direct the Agent to a specific destination, as well as for access token and TLS certificate registration, revocation and rotation as needed.
The Server can offer connection settings for the following 3 classes of destinations:
- The OpAMP Server itself. This is typically used to manage credentials such as the TLS certificate or the request headers that are used for authorization. The Server MAY also offer a different destination endpoint to direct the OpAMP Client to connect to a different OpAMP Server.
- The destinations for the Agent to send its own telemetry: metrics, traces and logs using OTLP/HTTP protocol.
- A set of additional "other" connection settings, with a string name associated with each. How the Agent type uses these is Agent-specific. Typically the name represents the name of the destination to connect to (as it is known to the Agent). For example OpenTelemetry Collector can use the named connection settings for its exporters, one named connection setting per correspondingly named exporter.
The Server may make an offer for the particular connection class only if the corresponding capability to use the connection is reported by the Agent via AgentToServer.capabilities field:
- If ReportsOwnTraces capability bit is set the Server may offer connection settings for traces using own_traces field.
- If ReportsOwnMetrics capability bit is set the Server may offer connection settings for metrics using own_metrics field.
- If ReportsOwnLogs capability bit is set the Server may offer connection settings for logs using own_logs field.
- If AcceptsOpAMPConnectionSettings capability bit is set the Server may offer connection settings for OpAMP connection using opamp field.
- If AcceptsOtherConnectionSettings capability bit is set the Server may offer connection settings for other destinations using other_connections field.
Depending on which connection settings are offered the sequence of operations is slightly different. The handling of connection settings for own telemetry is described in Own Telemetry Reporting. The handling of connection settings for "other" destinations is described in Connection Settings for "Other" Destinations. The handling of OpAMP connection settings is described below.
Here is how the OpAMP connection settings change happens:
Client Server
│ │ Initiate
│ Connect │ Settings
├──────────────────────────────────────►│ Change
│ ... │ │
│ │◄───────┘
│ │ ┌───────────┐
│ ├─────────►│ │
│ │ Generate │Credentials│
┌───────────┐ │ServerToAgent{ConnectionSettingsOffers}│ and Save │ Store │
│ │◄───────┤◄──────────────────────────────────────┤◄─────────┤ │
│Credentials│ Save │ │ └───────────┘
│ Store │ │ Disconnect │
│ ├───────►├──────────────────────────────────────►│
└───────────┘ │ │
│ Connect, New settings │ ┌───────────┐
├──────────────────────────────────────►├─────────►│ │
│ │ Delete │Credentials│
┌───────────┐ │ Connection established │ old │ Store │
│ │◄───────┤◄─────────────────────────────────────►│◄─────────┤ │
│Credentials│Delete │ │ └───────────┘
│ Store │old │ │
│ ├───────►│ │
└───────────┘ │ │
- Server generates new connection settings and saves it in Server's credentials store, associating the new settings with the Agent instance UID.
- Server sends the ServerToAgent message that includes ConnectionSettingsOffers message. The opamp field contains the new OpAMPConnectionSettings offered.
- Client receives the settings offer and saves the updated connection settings in the local store, marking it as "candidate" (if Client crashes it will retry "candidate" validation steps 5-9).
- Client disconnects from the Server.
- Client connects to the Server, using the new settings.
- Connection is successfully established, any TLS verifications required are passed and the Server indicates a successful authorization.
- Server deletes the old connection settings for this Agent (using Agent instance UID) from its credentials store.
- Client deletes the old settings from its credentials store and marks the new connection settings as "valid".
- If step 6 fails the Client deletes the new settings and reverts to the old settings and reconnects.
Note: Clients which are unable to persist new connection settings and have access only to ephemeral storage SHOULD reject certificate offers otherwise they risk losing access after restarting and losing the offered certificate.
OpAMP Clients that want to use TLS with a client certificate but do not initially have a certificate can use the Trust On First Use (TOFU) flow. The sequence is the following:
- Client connects to the Server using regular TLS (validating Server's identity) but without a client certificate. Client sends the Agent's Status Report so that it can be identified.
- The Server accepts the connection and status and awaits for an approval to generate a client certificate for the OpAMP Client.
- Server either waits for a manual approval by a human or automatically approves all TOFU requests if the Server is configured to do so (can be a Server-side option).
- Once approved the flow is essentially identical to OpAMP Connection Setting Offer Flow steps, except that there is no old client certificate to delete.
TOFU flow allows to bootstrap a secure environment without the need to perform Client-side installation of certificates.
Exact same TOFU approach can be also used for OpAMP Clients that don't have the necessary authorization headers to access the Server. The Server can detect such access and upon approval send the authorization headers to the Client.
In some use cases it may be desirable to equip newly installed Agents with an initial connection settings that are good for the first use, but generate a new set of connection credentials after the first connection is established.
This can be achieved very similarly to how the TOFU flow works. The only difference is that the first connection will be properly authenticated, but the Server will immediately generate and offer new connection settings to the Agent. The Client will then persist the setting and will use them for all subsequent operations.
This allows deploying a large number of Agents using one pre-defined set of connection credentials (authorization headers, certificates, etc), but immediately after successful connection each Agent will acquire their own unique connection credentials. This way individual Agent's credentials may be revoked without disrupting the access to all other Agents.
Since the Server knows what access headers and a client certificate the Client uses, the Server can revoke access to individual Agents by marking the corresponding connection settings as "revoked" and disconnecting the Client. Subsequent connections using the revoked credentials can be rejected by the Server essentially prohibiting the Client to access the Server.
Since the Server has control over the connection settings of all 3 destination types of the Agent (because it can offer the connection settings) this revocation may be performed for either of the 3 types of the destinations, provided that the Server previously offered and the Agent accepted the particular type of destination.
For own telemetry and "other" destinations the Server MUST also communicate the revocation fact to the corresponding destinations so that they can begin rejecting access to connections that use the revoked credentials.
Client certificates that the Server generates may be self-signed, signed by a private Certificate Authority or signed by a public Certificate Authority. The Server is responsible for generating client certificates such that they are trusted by the destination the certificate is intended for. This requires that either the destinations remember and trust the individual self-signed client certificate's public key directly or they trust the Certificate Authority that is used for signing the client certificate so that the trust chain can be verified.
How exactly the client certificates are generated is outside the scope of the OpAMP specification.
TBD
ConnectionSettingsOffers message describes connection settings for the Agent to use:
message ConnectionSettingsOffers {
bytes hash = 1;
OpAMPConnectionSettings opamp = 2;
TelemetryConnectionSettings own_metrics = 3;
TelemetryConnectionSettings own_traces = 4;
TelemetryConnectionSettings own_logs = 5;
map<string,OtherConnectionSettings> other_connections = 6;
}Hash of all settings, including settings that may be omitted from this message because they are unchanged.
Settings to connect to the OpAMP Server. If this field is not set then the Client should assume that the settings are unchanged and should continue using existing settings. The Client MUST verify the offered connection settings by actually connecting before accepting the setting to ensure it does not lose access to the OpAMP Server due to invalid settings.
Settings to connect to an OTLP metrics backend to send Agent's own metrics to. If this field is not set then the Agent should assume that the settings are unchanged.
Settings to connect to an OTLP metrics backend to send Agent's own traces to. If this field is not set then the Agent should assume that the settings are unchanged.
Settings to connect to an OTLP metrics backend to send Agent's own logs to. If this field is not set then the Agent should assume that the settings are unchanged.
Another set of connection settings, with a string name associated with each. How the Agent uses these is Agent-specific. Typically the name represents the name of the destination to connect to (as it is known to the Agent). If this field is not set then the Agent should assume that the other_connections settings are unchanged.
The OpAMPConnectionSettings message is a collection of fields which comprise an offer from the Server to the OpAMP Client to use the specified settings for OpAMP connection.
message OpAMPConnectionSettings {
string destination_endpoint = 1;
Headers headers = 2;
TLSCertificate certificate = 3;
}OpAMP Server URL This MUST be a WebSocket or HTTP URL and MUST be non-empty, for example: "wss://example.com:4318/v1/opamp"
Optional headers to use when connecting. Typically used to set access tokens or other authorization headers. For HTTP-based protocols the Client should set these in the request headers. For example: key="Authorization", Value="Basic YWxhZGRpbjpvcGVuc2VzYW1l".
The Client should use the offered certificate to connect to the destination from now on. If the Client is able to validate and connect using the offered certificate the Client SHOULD forget any previous client certificates for this connection. This field is optional: if omitted the client SHOULD NOT use a client-side certificate. This field can be used to perform a client certificate revocation/rotation.
The TelemetryConnectionSettings message is a collection of fields which comprise an offer from the Server to the Agent to use the specified settings for a network connection to report own telemetry.
message TelemetryConnectionSettings {
string destination_endpoint = 1;
Headers headers = 2;
TLSCertificate certificate = 3;
}The value MUST be a full URL an OTLP/HTTP/Protobuf receiver with path. Schema SHOULD begin with "https://", for example "https://example.com:4318/v1/metrics" The Agent MAY refuse to send the telemetry if the URL begins with "http://".
Optional headers to use when connecting. Typically used to set access tokens or other authorization headers. For HTTP-based protocols the Agent should set these in the request headers. For example: key="Authorization", Value="Basic YWxhZGRpbjpvcGVuc2VzYW1l".
The Agent should use the offered certificate to connect to the destination from now on. If the Agent is able to validate and connect using the offered certificate the Agent SHOULD forget any previous client certificates for this connection. This field is optional: if omitted the client SHOULD NOT use a client-side certificate. This field can be used to perform a client certificate revocation/rotation.
The OtherConnectionSettings message is a collection of fields which comprise an offer from the Server to the Agent to use the specified settings for a network connection. It is not required that all fields in this message are specified. The Server may specify only some of the fields, in which case it means that the Server offers the Agent to change only those fields, while keeping the rest of the fields unchanged.
For example the Server may send a ConnectionSettings message with only the certificate field set, while all other fields are unset. This means that the Server wants the Agent to use a new certificate and continue sending to the destination it is currently sending using the current header and other settings.
For fields which reference other messages the field is considered unset when the reference is unset.
For primitive field (string) we rely on the "flags" to describe that the field is not set (this is done to overcome the limitation of old protoc compilers don't generate methods that allow to check for the presence of the field.
message OtherConnectionSettings {
string destination_endpoint = 1;
Headers headers = 2;
TLSCertificate certificate = 3;
map<string, string> other_settings = 4;
}A URL, host:port or some other destination specifier.
Optional headers to use when connecting. Typically used to set access tokens or other authorization headers. For HTTP-based protocols the Agent should set these in the request headers. For example: key="Authorization", Value="Basic YWxhZGRpbjpvcGVuc2VzYW1l".
The Agent should use the offered certificate to connect to the destination from now on. If the Agent is able to validate and connect using the offered certificate the Agent SHOULD forget any previous client certificates for this connection. This field is optional: if omitted the client SHOULD NOT use a client-side certificate. This field can be used to perform a client certificate revocation/rotation.
Other connection settings. These are Agent-specific and are up to the Agent interpret.
message Headers {
repeated Header headers = 1;
}
message Header {
string key = 1;
string value = 2;
}
The message carries a TLS certificate that can be used as a client-side certificate.
The (public_key,private_key) certificate pair should be issued and signed by a Certificate Authority that the destination Server recognizes.
Alternatively the certificate may be self-signed, assuming the Server can verify the certificate. In this case the ca_public_key field can be omitted.
message TLSCertificate {
bytes public_key = 1;
bytes private_key = 2;
bytes ca_public_key = 3;
}PEM-encoded public key of the certificate. Required.
PEM-encoded private key of the certificate. Required.
PEM-encoded public key of the CA that signed this certificate. Optional, MUST be specified if the certificate is CA-signed. Can be stored by intermediary TLS-terminating proxies in order to verify the connecting client's certificate in the future.
Own Telemetry Reporting is an optional capability of OpAMP protocol. The Server can offer to the Agent a destination to which the Agent can send its own telemetry (metrics, traces or logs). If the Agent is capable of producing telemetry and wishes to do so then it should sends its telemetry to the offered destination using OTLP/HTTP protocol:
┌────────────┬────────┐ ┌─────────┐
│ │ OpAMP │ OpAMP │ OpAMP │
│ │ ├──────────►│ │
│ │ Client │ │ Server │
│ └────────┤ └─────────┘
│ Agent │
│ ┌────────┤ ┌─────────┐
│ │OTLP │ OTLP/HTTP │ OTLP │
│ │ ├──────────►│ Metric │
│ │Exporter│ │ Backend │
└────────────┴────────┘ └─────────┘
The Server makes the offer by sending a ServerToAgent message with a populated connection_settings field that contains one or more of the own_metrics, own_traces, own_logs fields set. Each of these fields describes a destination, which can receive telemetry using OTLP protocol.
The Server SHOULD populate the connection_settings field when it sends the first ServerToAgent message to the particular Agent (normally in response to the first status report from the Client), unless there is no OTLP backend that can be used. The Server SHOULD also populate the field on subsequent ServerToAgent if the destination has changed. If the destination is unchanged the connection_settings field SHOULD NOT be set. When the Agent receives a ServerToAgent with an unset connection_settings field the Agent SHOULD continue sending its telemetry to the previously offered destination.
The Agent SHOULD periodically report its metrics to the destination offered in the own_metrics field. The recommended reporting interval is 10 seconds. Here is the diagram that shows the operation sequence:
Agent Client Server
Metric
│ │ │ Backend
│ │ServerToAgent{ConnectionSettingsOffer}│
┌─────────│◄─────────────────────────────────────┤ │
│ │ │
▼ │
┌────────┐ │
│Collect │ OTLP Metrics │ ──┐
│Own ├───────────────────────────────────────────────►│ │
│Metrics │ │ │
└────────┘ ... . │ Repeats
│ │
┌────────┐ │ │ Periodically
│Collect │ OTLP Metrics │ │
│Own ├───────────────────────────────────────────────►│ │
│Metrics │ │ ──┘
└────────┘ │
The Agent SHOULD report metrics of the Agent process (or processes) and any custom metrics that describe the Agent state. Reported process metrics MUST follow the OpenTelemetry conventions for processes.
Similarly, the Agent SHOULD report its traces to the destination offered in the own_traces field and logs to the destination offered in the own_logs field.
All attributes specified in the identifying_attributes field in AgentDescription message SHOULD be also specified in the Resource of the reported OTLP telemetry.
Attributes specified in the non_identifying_attributes field in AgentDescription message may be also specified in the Resource of the reported OTLP telemetry, in which case they SHOULD have exactly the same values.
Agent configuration is an optional capability of OpAMP protocol. Remote configuration capability can be disabled if necessary (for example when using existing configuration capabilities of an orchestration system such as Kubernetes).
The Server can offer a Remote Configuration to the Agent by setting the remote_config field in the ServerToAgent message. Since the ServerToAgent message is normally sent by the Server in response to a status report the Server has the Agent's description and may tailor the configuration it offers to the specific Agent if necessary.
The OpAMP Client MUST set the AcceptsRemoteConfig bit of AgentToServer.capabilities if the Agent is capable of accepting remote configuration. If the bit is not set the Server MUST not offer a remote configuration to the Agent.
The Agent's actual configuration that it uses for running may be different from the Remote Configuration that is offered by the Server. This actual configuration is called the Effective Configuration of the Agent. The Effective Configuration is typically formed by the Agent after merging the Remote Configuration with other inputs available to the Agent, e.g. a locally available configuration.
Once the Effective Configuration is formed the Agent uses it for its operation and the Client will also report the Effective Configuration to the OpAMP Server via the effective_config field of status report. The Server typically allows the end user to see the effective configuration alongside other data reported in the status reported by the Client.
The Client MUST set the ReportsEffectiveConfig bit of AgentToServer.capabilities if the Agent is capable of reporting effective configuration. If the bit is not set the Server should not expect the AgentToServer.effective_config field to be set.
Here is the typical configuration sequence diagram:
Agent Client Server
│ │ AgentToServer{} │ ┌─────────┐
│ ├──────────────────────────────────►├──►│ Process │
│ │ │ │ Status │
Local │ Remote │ │ │ and │
Config │ Config │ ServerToAgent{AgentRemoteConfig} │ │ Fetch │
│ │ ┌────────┤◄──────────────────────────────────┤◄──┤ Config │
▼ │ ▼ │ │ └─────────┘
┌─────────┐ │ │
│ Config │ │ │
│ Merger │ │ │
└─────┬───┘ │ │
│ │ │
│Effective │ │
│Config │ AgentToServer{} │
└──────────►├──────────────────────────────────►│
│ │
│ │
EffectiveConfig and RemoteConfigStatus fields are included in the AgentToServer message if the fields have changed.
Note: the Client SHOULD NOT send AgentToServer message if the Effective Configuration or other fields that are reported via AgentToServer message are unchanged. If the Client does not follow this rule the operation may result in an infinite loop of messages sent back and forth between the Client and the Server.
The Server may also initiate sending of a remote configuration on its own, without waiting for a status report from the Client. This can be used to re-configure an Agent that is connected but which has nothing new to report. The sequence diagram in this case looks like this:
Agent Client Server
│ │ │
│ │ │
│ │ │ ┌────────┐
Local │ Remote │ │ │Initiate│
Config │ Config │ ServerToAgent{AgentRemoteConfig} │ │and │
│ │ ┌────────┤◄──────────────────────────────────┤◄──┤Send │
▼ │ ▼ │ │ │Config │
┌─────────┐ │ │ └────────┘
│ Config │ │ │
│ Merger │ │ │
└────┬────┘ │ │
│ │ │
│Effective │ │
│Config │ AgentToServer{} │
└──────────►├──────────────────────────────────►│
│ │
│ │
The Agent may ignore the Remote Configuration offer if it does not want its configuration to be remotely controlled by the Server.
The configuration of the Agent is a collection of named configuration files (this applies both to the Remote Configuration and to the Effective Configuration).
The file names MUST be unique within the collection. It is possible that the Remote and Local Configuration MAY contain a file with the same name but with a different content. How these files are merged to form an Effective Configuration is Agent type-specific and is not part of the OpAMP protocol.
If there is only one configuration file in the collection then the file name MAY be empty.
The collection of configuration files is represented using a AgentConfigMap message:
message AgentConfigMap {
map<string, AgentConfigFile> config_map = 1;
}The config_map field of the AgentConfigSet message is a map of configuration files, where keys are file names.
For Agents that use a single config file the config_map field SHOULD contain a single entry and the key MAY be an empty string.
The AgentConfigFile message represents one configuration file and has the following structure:
message AgentConfigFile {
bytes body = 1;
string content_type = 2;
}The body field contains the raw bytes of the configuration file. The content, format and encoding of the raw bytes is Agent type-specific and is outside the concerns of OpAMP protocol.
content_type is an optional field. It is a MIME Content-Type that describes what's contained in the body field, for example "text/yaml". The content_type reported in the Effective Configuration in the Agent's status report may be used for example by the Server to visualize the reported configuration nicely in a UI.
Remote Configuration is a potentially dangerous functionality that may be exploited by malicious actors. For example if the Agent is capable of collecting local files and sending over the network then a compromised OpAMP Server may offer a malicious remote configuration to the Agent and compel the Agent to collect a sensitive local file and send to a specific network destination.
See Security section for general recommendations and recommendations specifically for remote reconfiguration capabilities.
The message has the following structure:
message AgentRemoteConfig {
AgentConfigMap config = 1;
}Each Agent is composed of one or more packages. A package has a name and content stored in a file. The content of the file, functionality provided by the packages, how they are stored and used by the Agent side is Agent type-specific and is outside the concerns of the OpAMP protocol.
There are two types of packages: top-level and sub-packages.
There is normally only one top-level package, which implements the primary functionality of the Agent. When there is only one top-level package it may have an empty name.
Sub-packages are also known as addons or plugins. The sub-packages can be installed at the Agent for added functionality (hence the name addons).
The Agent may have one or more packages installed. Each package has a name. The Agent cannot have more than one package of the particular name installed.
Different package may have files with the same name, file names are not globally unique, they are only unique within the scope of a particular package.
Package may be provided and installed locally (e.g. by a local user). The packages may be also offered to the Agent remotely by the Server, in which case the Agent may download and install the packages.
To offer packages to the Agent the Server sets the packages_available field in the ServerToAgent message that is sent either in response to a status report form the Agent or by Server's initiative if the Server wants to push packages to the Agent.
The PackagesAvailable message describes the packages that are available on the Server for this Agent. For each package the message describes the file that has the package's content and provides the URL from which the file can be downloaded using an HTTP GET request. The URLs point to package files on a Download Server (which may be on the same host as the OpAMP Server or a different host).
The protocol supports only a single downloadable file per package. If the Agent's packages conceptually are composed of multiple files then the Agent and Server can agree to store the files in any file format that allows storing multiple files in a single file, e.g. a zip or tar file. After downloading the single package file the Agent may extract the files contained in it. How exactly this is done is Agent specific and is beyond the scope of the protocol.
The Server is allowed to make a package offer only if the OpAMP Client indicated that the Agent can accept packages via AcceptsPackages bit of AgentToServer.capabilities field.
After receiving the PackagesAvailable message the Agent SHOULD follow this download procedure:
Compare the aggregate hash of all packages it has with the aggregate hash offered by the Server in the all_packages_hash field.
If the aggregate hash is the same then consider the download procedure done, since it means all packages on the Agent are the same as offered by the Server. Otherwise, go to Step 2.
For each package offered by the Server the Agent SHOULD check if it should download the particular package:
- If the Agent does not have a package with the specified name then it SHOULD download the package. See Step 3 on how to download each package file.
- If the Agent has the package the Agent SHOULD compare the hash of the package that the Agent has with the hash of the package offered by the Server in the hash field in the PackageAvailable message. If the hashes are the same then the package is the same and processing for this package is done, proceed to the next package. If the hashes are different then check the package file as described in Step 3.
Finally, if the Agent has any packages that are not offered by the Server the packages SHOULD be deleted by the Agent.
For the file of the package offered by the Server the Agent SHOULD check if it should download the file:
- If the Agent does not have a file with the specified name then it SHOULD download the file.
- If the Agent has the file then the Agent SHOULD compare the hash of the file it has locally with the hash field in the DownloadableFile message. If hashes are the same the processing of this file is done. Otherwise, the offered file is different and the file SHOULD be downloaded from the location specified in the download_url field of the DownloadableFile message. The Agent SHOULD use an HTTP GET message to download the file.
The procedure outlined above allows the Agent to efficiently download only new or changed packages and only download new or changed files.
After downloading the packages the Agent can perform any additional processing that is Agent type-specific (e.g. "install" or "activate" the packages in any way that is specific to the Agent).
During the downloading and installation process the Agent MAY periodically report the status of the process. To do this the OpAMP Client SHOULD send an AgentToServer message and set the package_statuses field accordingly.
Once the downloading and installation of all packages is done (succeeded or failed) the Client SHOULD report the status of all packages to the Server.
Here is the typical sequence diagram for the package downloading and status reporting process:
Download Agent/Client OpAMP
Server Server
│ │ │
│ │ ServerToAgent{PackagesAvailable}│
│ │◄─────────────────────────────────┤
│ HTTP GET │ │
│◄────────────────┤ │
│ Download file #1│ │
├────────────────►│ │
│ │ AgentToServer{PackageStatuses} │
│ ├─────────────────────────────────►│
│ HTTP GET │ │
│◄────────────────┤ │
│ Download file #2│ │
├────────────────►│ │
│ │ AgentToServer{PackageStatuses} │
│ ├─────────────────────────────────►│
│ │ │
. . .
│ HTTP GET │ │
│◄────────────────┤ │
│ Download file #N│ │
├────────────────►│ │
│ │ AgentToServer{PackageStatuses} │
│ ├─────────────────────────────────►│
│ │ │
The Client MUST always include all packages the Agent has or is processing (downloading or installing) in PackageStatuses message.
Note that the Client MAY also report the status of packages the Agent has installed locally, not only the packages it was offered and downloaded from the Server.
The Agent and the Server use hashes to identify content of files and packages such that the Agent can decide what files and packages need to be downloaded.
The calculation of the hashes is performed by the Server. The Server MUST choose a strong hash calculation method with minimal collision probability (and it may seed random values into calculation to guarantee hash uniqueness if such guarantees are needed by the implementation).
The hashes are opaque to the Agent, the Agent never calculates hashes, it only stores and compares them.
There are 3 levels of hashes:
The hash of the packages file content. This is stored in the content_hash field in the DownloadableFile message. This value SHOULD be used by the Agent to determine if the particular file it has is different on the Server and needs to be re-downloaded.
The package hash that identifies the entire package (package name and file content). This hash is stored in the hash field in the PackageAvailable message.
This value SHOULD be used by the Agent to determine if the particular package it has is different on the Server and needs to be re-downloaded.
The all packages hash is the aggregate hash of all packages for the particular Agent. The hash is calculated as an aggregate of all package names and content. This hash is stored in the all_packages_hash field in the PackagesAvailable message.
This value SHOULD be used by the Agent to determine if any of the packages it has are different from the ones available on the Server and need to be re-downloaded.
Note that the aggregate hash does not include the packages that are available on the Agent locally and were not downloaded from the download Server.
Downloading packages remotely is a potentially dangerous functionality that may be exploited by malicious actors. If packages contain executable code then a compromised OpAMP Server may offer a malicious package to the Agent and compel the Agent to execute arbitrary code.
See Security section for general recommendations and recommendations specifically for code signing capabilities.
The message has the following structure:
message PackagesAvailable {
map<string, PackageAvailable> packages = 1;
bytes all_packages_hash = 2;
}
A map of packages. Keys are package names.
Aggregate hash of all remotely installed packages.
The Client SHOULD include this value in subsequent PackageStatuses messages. This in turn allows the Server to identify that a different set of packages is available for the Agent and specify the available packages in the next ServerToAgent message.
This field MUST be always set if the Server supports sending packages to the Agents and if the Agent indicated it is capable of accepting packages.
This message is an offer from the Server to the Agent to install a new package or initiate an upgrade or downgrade of a package that the Agent already has. The message has the following structure:
message PackageAvailable {
PackageType type = 1;
string version = 2;
DownloadableFile file = 3;
bytes hash = 4;
}The type of the package, either an addon or a top-level package.
enum PackageType {
TopLevelPackage = 0;
AddonPackage = 1;
}The package version that is available on the Server side. The Agent may for example use this information to avoid downloading a package that was previously already downloaded and failed to install.
The downloadable file of the package.
The hash of the package. SHOULD be calculated based on all other fields of the PackageAvailable message and content of the file of the package. The hash is used by the Agent to determine if the package it has is different from the package the Server is offering.
The message has the following structure:
message DownloadableFile {
string download_url = 1;
bytes content_hash = 2;
bytes signature = 3;
}The URL from which the file can be downloaded using HTTP GET request. The Server at the specified URL SHOULD support range requests to allow for resuming downloads.
The SHA256 hash of the file content. Can be used by the Agent to verify that the file was downloaded correctly.
Optional signature of the file content. Can be used by the Agent to verify the authenticity of the downloaded file, for example can be the detached GPG signature. The exact signing and verification method is Agent specific. See Code Signing for recommendations.
The Client connects to the Server by establishing an HTTP(S) connection.
If WebSocket transport is used then the connection is upgraded to WebSocket as defined by WebSocket standard.
After the connection is established the Client MUST send the first status report and expect a response to it.
If the Client is unable to establish a connection to the Server it SHOULD retry connection attempts and use exponential backoff strategy with jitter to avoid overwhelming the Server.
When retrying connection attempts the Client SHOULD honour any throttling responses it receives from the Server.
To close a connection the Client MUST first send an AgentToServer message with agent_disconnect field set. The Client MUST then send a WebSocket Close control frame and follow the procedure defined by WebSocket standard.
To close a connection the Server MUST then send a WebSocket Close control frame and follow the procedure defined by WebSocket standard.
The Client is considered logically disconnected as soon as the OpAMP HTTP response is completed. It is not necessary for the Client to send AgentToServer message with agent_disconnect field set since it is always implied anyway that the Client connection is gone after the HTTP response is completed.
HTTP keep-alive may be used by the Client and the Server but it has no effect on the logical operation of the OpAMP protocol.
The Server may use its own business logic to decide what it considers an active Agent (e.g. an Client that continuously polls) vs an inactive Agent (e.g. a Client that has not made an HTTP request for a specific period of time). This business logic is outside the scope of OpAMP specification.
If an established WebSocket connection is broken (disconnected) unexpectedly the Client SHOULD immediately try to re-connect. If the re-connection fails the Client SHOULD continue connection attempts with backoff as described in Establishing Connection.
Each Client instance SHOULD connect no more than once to the Server. If the Client needs to re-connect to the Server the Client MUST ensure that it sends an AgentDisconnect message first, then closes the existing connection and only then attempts to connect again.
The Server MAY disconnect or deny serving requests if it detects that the same Client instance has more than one simultaneous connection or if multiple Agent instances are using the same instance_uid.
The Server SHOULD detect duplicate instance_uids (which may happen for example when Agents are using bad UID generators or due to cloning of the VMs where the Agent runs). When a duplicate instance_uid is detected, Server SHOULD generate a new instance_uid, and send it as new_instance_uid value of AgentIdentification.
The Client and the Server MAY use authentication methods supported by HTTP, such as Basic authentication or Bearer authentication. The authentication happens when the HTTP connection is established before it is upgraded to a WebSocket connection.
The Server MUST respond with 401 Unauthorized if the Client authentication fails.
If the Server receives a malformed AgentToServer message the Server SHOULD respond with a ServerToAgent message with error_response set accordingly. The type field MUST be set to BAD_REQUEST and error_message SHOULD be a human readable description of the problem with the AgentToServer message.
The Client SHOULD NOT retry sending an AgentToServer message to which it received a BAD_REQUEST response.
The Client MAY retry sending AgentToServer message if:
- AgentToServer message that requires a response was sent, however no response was received within a reasonable time (the timeout MAY be configurable).
- AgentToServer message that requires a response was sent, however the connection was lost before the response was received.
- After receiving an UNAVAILABLE response from the Server as described in the Throttling section.
For messages that require a response if the Server receives the same message more than once the Server MUST respond to each message, not just the first message, even if the Server detects the duplicates and processes the message once.
Note that the Client is not required to keep a growing queue of messages that it wants to send to the Server if the connection is unavailable. The Client typically only needs to keep one up-to-date message of each kind that it wants to send to the Server and send it as soon as the connection is available.
For example, the Client should keep track of the Agent's status and compose a AgentToServer message that is ready to be sent at the first opportunity. If the Client is unable to send the AgentToServer message (for example if the connection is not yet available) the Client does not need to create a new AgentToServer every time the Agent's status changes and keep all these AgentToServer messages in a queue ready to be sent. The Client simply needs to keep one up-to-date AgentToServer message and send it at the first opportunity. This of course requires the AgentToServer message to contain all changes since it was last reported and to correctly reflect the current (last) state of the Agent.
Similarly, all other Agent reporting capabilities, such as Addon Status Reporting or Agent Package Installation Status Reporting require the Client to only keep one up-to-date status message and send it at the earliest opportunity.
The exact same logic is true in the opposite direction: the Server normally only needs to keep one up-to-date message of a particular kind that it wants to deliver to the Agent and send it as soon as the connection to the Client is available.
When the Server is overloaded and is unstable to process the AgentToServer message it SHOULD respond with an ServerToAgent message, where error_response is filled with type field set to UNAVAILABLE. The Client SHOULD disconnect, wait, then reconnect again and resume its operation. The retry_info field may be optionally set with retry_after_nanoseconds field specifying how long the Client SHOULD wait before reconnecting:
message RetryInfo {
uint64 retry_after_nanoseconds = 1;
}If retry_info is not set then the Client SHOULD implement an exponential backoff strategy to gradually increase the interval between retries.
In the case when plain HTTP transport is used as well as when WebSocket is used and the Server is overloaded and is unable to upgrade the HTTP connection to WebSocket the Server MAY return HTTP 503 Service Unavailable or HTTP 429 Too Many Requests response and MAY optionally set Retry-After header to indicate when SHOULD the Client attempt to reconnect. The Client SHOULD honour the corresponding requirements of HTTP specification.
The minimum recommended retry interval is 30 seconds.
Remote configuration, downloadable packages are a significant security risk. By sending a malicious Server-side configuration or a malicious package the Server may compel the Agent to perform undesirable work. This section defines recommendations that reduce the security risks for the Agent.
Guidelines in this section are optional for implementation, but are highly recommended for sensitive applications.
We recommend that the Agent employs the zero-trust security model and does not automatically trust the remote configuration or other offers it receives from the Server. The data received from the Server SHOULD be verified and sanitized by the Agent in order to limit and prevent the damage that may be caused by malicious actors. We recommend the following:
- The Agent SHOULD run at the minimum possible privilege to prevent itself from accessing sensitive files or perform high privilege operations. The Agent SHOULD NOT run as root user, otherwise a compromised Agent may result in total control of the machine by malicious actors.
- If the Agent is capable of collecting local data it SHOULD limit the collection to a specific set of directories. This limitation SHOULD be locally specified and SHOULD NOT be overridable via remote configuration. If this rule is not followed the remote configuration functionality may be exploited to access sensitive information on the Agent's machine.
- If the Agent is capable of executing external code located on the machine where it runs and this functionality can be specified in the Agent's configuration then the Agent SHOULD limit such functionality only to specific scripts located in a limited set of directories. This limitation SHOULD be locally specified and SHOULD NOT be overridable via remote configuration. If this rule is not followed the remote configuration functionality may be exploited to perform arbitrary code execution on the Agent's machine.
The Agent is recommended to restrict what it may be compelled to do via remote configuration.
Particularly, if it is possible via a configuration to ask the Agent to collect data from the machine it runs on (as it is often the case for telemetry collecting Agents) then we recommend to have Agent-side restrictions as to what directories or files the Agent is allowed to collect. Upon receiving a remote config the Agent must validate the configuration against the list of restrictions and refuse to apply the configuration either fully or partially if it violates the restrictions or sanitize the configuration such that it does not collect data from prohibited directories or files.
Similarly, if the configuration provides means to order the Agent to execute processes or scripts on the machine it runs on we recommend to have Agent-side restrictions as to what executable files from what directories the Agent is allowed to run.
It is recommended that the restrictions are specified in the form of "allow list" instead of the "deny list". The restrictions may be hard-coded or may be end-user definable in a local config file. It should not be possible to override these restrictions by sending a remote config from the Server to the Agent.
It is recommended that remote configuration capabilities are not enabled in the Agent by default. The capabilities should be opt-in by the user.
Any executable code that is part of a package should be signed to prevent a compromised Server from delivering malicious code to the Agent. We recommend the following:
- Any downloadable executable code (e.g. executable packages) need to be code-signed. The actual code-signing and verification mechanism is Agent specific and is outside the concerns of the OpAMP specification.
- The Agent SHOULD verify executable code in downloaded files to ensure the code signature is valid.
- The downloadable code can be signed with the signature included in the file content or have a detached signature recorded in the DownloadableFile message's signature field. Detached signatures may be used for example with GPG signing.
- If Certificate Authority is used for code signing it is recommended that the Certificate Authority and its private key is not co-located with the OpAMP Server, so that a compromised Server cannot sign malicious code.
- The Agent SHOULD run any downloaded executable code (the packages and or any code that it runs as external processes) at the minimum possible privilege to prevent the code from accessing sensitive files or perform high privilege operations. The Agent SHOULD NOT run downloaded code as root user.
OpAMP defines a number of capabilities for the Agent and the Server. Most of these capabilities are optional. The Agent or the Server should be prepared that the peer does not support a particular capability.
Both the Agent and the Server indicate the capabilities that they support during the initial message exchange. The Client sets the capabilities bit-field in the AgentToServer message, the Server sets the capabilities bit-field in the ServerToAgent message.
Each set bit in the capabilities field indicates that the particular capability is supported. The list of Agent capabilities is here. The list of Server capabilities is here.
After the Server learns about the capabilities of the particular Agent the Server MUST stop using the capabilities that the Agent does not support.
Similarly, after the Agent learns about the capabilities of the Server the Agent MUST stop using the capabilities that the Server does not support.
The specifics of what in the behavior of the Agent and the Server needs to change when they detect that the peer does not support a particular capability are described in this document where relevant.
There are 2 ways OpAMP enables interoperability between an implementation of the current version of this specification and an implementation of a future, extended version of OpAMP that adds more capabilities that are not described in this specification.
For the new capabilities that extend the functionality in such a manner that they can be silently ignored by the peer a new field may be added to any Protobuf message. The sender that implements this new capability will set the new field. A recipient that implements an older version of the specification that is unaware of the new capability will simply ignore the new field. The Protobuf encoding ensures that the rest of the fields are still successfully deserialized by the recipient.
For the new capabilities that extend the functionality in such a manner that they cannot be silently ignored by the peer a different approach is used.
The capabilities fields in AgentToServer and ServerToAgent messages contains a number of reserved bits. These bits SHOULD be used for indicating support of new capabilities that will be added to OpAMP in the future.
The Client and the Server MUST set these reserved bits to 0 when sending the message. This allows the recipient, which implements a newer version of OpAMP to learn that the sender does not support the new capability and adjust its behavior correspondingly.
The AgentToServer and ServerToAgent messages are the first messages exchanged by the Client and Server which allows them to learn about the capabilities of the peer and adjust their behavior appropriately. How exactly the behavior is adjusted for future capabilities MUST be defined in the future specification of the new capabilities.
TBD
WebSocket is a bidirectional, message-oriented protocol that uses plain HTTP for establishing the connection and then uses the HTTP's existing TCP connection to deliver messages. It has been an RFC standard for a decade now. It is widely supported by browsers, servers, proxies and load balancers, has libraries in virtually all popular programming languages, is supported by network inspection and debugging tools, is secure and efficient and provides the exact message-oriented semantics that we need for OpAMP.
We could roll out our own message-oriented implementation over TCP but there are no benefits over WebSocket which is an existing widely supported standard. A custom TCP-based solution would be more work to design, more work to implement and more work to troubleshoot since existing network tools would not recognize it.
Regular HTTP is a half-duplex protocol, which makes delivery of messages from the Server to the client tied to the request time of the client. This means that if the Server needs to send a message to the client the client either needs to periodically poll the Server to give the Server an opportunity to send a message or we should use something like long polling.
Periodic polling is expensive. OpAMP protocol is largely idle after the initial connection since there is typically no data to deliver for hours or days. To have a reasonable delivery latency the client would need to poll every few seconds and that would significantly increase the costs on the Server side (we aim to support many millions simultaneous of Agents, which would mean servicing millions of polling requests per second).
Long polling is more complicated to use than WebSocket since it only provides one-way communication, from the Server to the client and necessitates the second connection for client-to-Server delivery direction. The dual connection needed for a long polling approach would make the protocol more complicated to design and implement without much gains compared to WebSocket approach.
gRPC is a big dependency that some implementations are reluctant to take. gRPC requires HTTP/2 support from all intermediaries and is not supported in some load balancers. As opposed to that, WebSocket is usually a small library in most language implementations (or is even built into runtime, like it is in browsers) and is more widely supported by load balancers since it is based on HTTP/1.1 transport.
Feature-wise gRPC streaming would provide essentially the same functionality as WebSocket messages, but it is a more complicated dependency that has extra requirements with no additional benefits for our use case (benefits of gRPC like ability to multiplex multiple streams over one connection are of no use to OpAMP).
Define specification for Concentrating Proxy that can serve as intermediary to reduce the number of connections to the Server when a very large number (millions and more) of Agents are managed.
- Splunk Deployment Server.
- Centralized Configuration of vRealize Log Insight Agents.
- Google Cloud Guest Agent uses HTTP long polling.
- Uber Flipr.
- Facebook's Holistic Configuration Management (push).
- mTLS in Go: https://kofo.dev/how-to-mtls-in-golang
- e2e audit https://pwn.recipes/posts/roll-your-own-e2ee-protocol/
- ACME certificate management protocol https://datatracker.ietf.org/doc/html/rfc8555
- ACME for client certificates http://www.watersprings.org/pub/id/draft-moriarty-acme-client-01.html
- AWS: https://aws.amazon.com/elasticloadbalancing/features/
- GCP: https://cloud.google.com/appengine/docs/flexible/go/using-websockets-and-session-affinity
- Azure: https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-websocket
--
Copyright The OpenTelemetry Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.