Enable AMQP 1.0 clients to manage topologies

[ansd]

What?

• Allow AMQP 1.0 clients to dynamically create and delete RabbitMQ
  topologies (exchanges, queues, bindings).
• Provide an Erlang AMQP 1.0 client that manages topologies.

Why?

Today, RabbitMQ topologies can be created via:

• Management HTTP API (including Management UI and messaging-topology-operator)
• Definition Import
• AMQP 0.9.1 clients

Up to RabbitMQ 3.13 the RabbitMQ AMQP 1.0 plugin auto creates queues
and bindings depending on the terminus address format.

Such implicit creation of topologies is limiting and obscure.
For some address formats, queues will be created, but not deleted.

Some of RabbitMQ's success is due to its flexible routing topologies
that AMQP 0.9.1 clients can create and delete dynamically.

This commit allows dynamic management of topologies for AMQP 1.0 clients.
This commit builds on top of Native AMQP 1.0 (PR #9022) and will be
available in RabbitMQ 4.0.

How?

This commits adds the following management operations for AMQP 1.0 clients:

• declare queue
• delete queue
• purge queue
• bind queue to exchange
• unbind queue from exchange
• declare exchange
• delete exchange
• bind exchange to exchange
• unbind exchange from exchange

Hence, at least the AMQP 0.9.1 management operations are supported for
AMQP 1.0 clients.

In addition the operation

• get queue

is provided which - similar to declare queue - returns queue
information including the current leader and replicas.
This allows clients to publish or consume locally on the node that hosts
the queue.

Compared to AMQP 0.9.1 whose commands and command fields are fixed, the
new AMQP Management API is extensible: New operations and new fields can
easily be added in the future.

There are different design options how management operations could be
supported for AMQP 1.0 clients:

1. Use a special exchange type as done in https://github.com/rabbitmq/rabbitmq-management-exchange
   This has the advantage that any protocol client (e.g. also STOMP clients) could
   dynamically manage topologies. However, a special exchange type is the wrong abstraction.
2. Clients could send "special" messages with special headers that the broker interprets.

This commit decided for a variation of the 2nd option using a more
standardized way by re-using a subest of the following latest AMQP 1.0 extension
specifications:

• AMQP Request-Response Messaging with Link Pairing Version 1.0 - Committee Specification 01 (February 2021)
• HTTP Semantics and Content over AMQP Version 1.0 - Working Draft 06 (July 2019)
• AMQP Management Version 1.0 - Working Draft 16 (July 2019)

An important goal is to keep the interaction between AMQP 1.0 client and RabbitMQ
simple to increase usage, development and adoptability of future RabbitMQ AMQP 1.0
client library wrappers.

The AMQP 1.0 client has to create a link pair to the special /management node.
This allows the client to send and receive from the management node.
Similar to AMQP 0.9.1, there is no need for a reply queue since the reply
will be sent directly to the client.

Requests and responses are modelled via HTTP, but sent via AMQP using
the HTTP Semantics and Content over AMQP extension (henceforth HTTP over AMQP extension).

This commit tries to follow the HTTP over AMQP extension as much as
possible but deviates where this draft spec doesn't make sense.

The projected mode §4.1 is used as opposed to tunneled mode §4.2.
A named relay /management is used (§6.3) where the message field to is the URL.

Deviations are

• §3.1 mandates that URIs are not encoded in an AMQP message.
  However, we percent encode URIs in the AMQP message. Otherwise there
  is for example no way to distinguish a / in a queue name from the
  URI path separator /.
• §4.1.4 mandates a data section. This commit uses an amqp-value section
  as it's a better fit given that the content is AMQP encoded data.

Using an HTTP API allows for a common well understood interface and future extensibility.
Instead of re-using the current RabbitMQ HTTP API, this commit uses a
new HTTP API (let's call it v2) which could be used as a future API for
plain HTTP clients.

HTTP API v1

The current HTTP API (let's call it v1) is not used since v1
comes with a couple of weaknesses:

1. Deep level of nesting becomes confusing and difficult to manage.

Examples of deep nesting in v1:

/api/bindings/vhost/e/source/e/destination/props
/api/bindings/vhost/e/exchange/q/queue/props

2. Redundant endpoints returning the same resources

v1 has 9 endpoints to list binding(s):

/api/exchanges/vhost/name/bindings/source
/api/exchanges/vhost/name/bindings/destination
/api/queues/vhost/name/bindings
/api/bindings
/api/bindings/vhost
/api/bindings/vhost/e/exchange/q/queue
/api/bindings/vhost/e/exchange/q/queue/props
/api/bindings/vhost/e/source/e/destination
/api/bindings/vhost/e/source/e/destination/props

3. Verbs in path names
   Path names should be nouns instead.
   v1 contains verbs:

/api/queues/vhost/name/get
/api/exchanges/vhost/name/publish

AMQP Management extension

Only few aspects of the AMQP Management extension are used.

The central idea of the AMQP management spec is dynamic discovery such that broker independent AMQP 1.0
clients can discover objects, types, operations, and HTTP endpoints of specific brokers.
In fact, clients are only conformant if:

All request addresses are dynamically discovered starting from the discovery document.
A requesting container MUST NOT use fixed assumptions about the addressing structure of the management API.

While this is a nice and powerful idea, no AMQP 1.0 client and no AMQP 1.0 server implement the
latest AMQP 1.0 management spec from 2019, partly presumably due to its complexity.
Therefore, the idea of such dynamic discovery has failed to be implemented in practice.

The AMQP management spec mandates that the management endpoint returns a discovery document containing
broker specific collections, types, configuration, and operations including their endpoints.

The API endpoints of the AMQP management spec are therefore all designed around dynamic discovery.

For example, to create either a queue or an exchange, the client has to

POST /$management/entities

which shows that the entities collection acts as a generic factory, see section 2.2.
The server will then create the resource and reply with a location header containing a URI pointing to the resource.
For RabbitMQ, we don’t need such a generic factory to create queues or exchanges.

To list bindings for a queue Q1, the spec suggests

GET /$management/Queues/Q1/$management/entities

which again shows the generic entities endpoint as well as a $management endpoint under Q1 to
allow a queue to return a discovery document.
For RabbitMQ, we don’t need such generic endpoints and discovery documents.

Given we aim for our own thin RabbitMQ AMQP 1.0 client wrapper libraries which expose
the RabbitMQ model to the developer, we can directly use fixed HTTP endpoint assumptions
in our RabbitMQ specific libraries.

This is by far simpler than using the dynamic endpoints of the management spec.
Simplicity leads to higher adoption and enables more developers to write RabbitMQ AMQP 1.0 client
library wrappers.

The AMQP Management extension also suffers from deep level of nesting in paths
Examples:

/$management/Queues/Q1/$management/entities
/$management/Queues/Q1/Bindings/Binding1

as well as verbs in path names: Section 7.1.4 suggests using verbs in path names,
for example “purge”, due to the dynamic operations discovery document.

HTTP API v2

This commit introduces a new HTTP API v2 following best practices.
It could serve as a future API for plain HTTP clients.

This commit and RabbitMQ 4.0 will only implement a minimal set of
HTTP API v2 endpoints and only for HTTP over AMQP.
In other words, the existing HTTP API v1 Cowboy handlers will continue to be
used for all plain HTTP requests in RabbitMQ 4.0 and will remain untouched for RabbitMQ 4.0.
Over time, after 4.0 shipped, we could ship a pure HTTP API implementation for HTTP API v2.
Hence, the new HTTP API v2 endpoints for HTTP over AMQP should be designed such that they
can be re-used in the future for a pure HTTP implementation.

The minimal set of endpoints for RabbitMQ 4.0 are:

GET / PUT / DELETE
/vhosts/:vhost/queues/:queue

read, create, delete a queue

DELETE
/vhosts/:vhost/queues/:queue/messages

purges a queue

GET / DELETE
/vhosts/:vhost/bindings/:binding

read, delete bindings
where :binding is a binding ID of the following path segment:

src=e1;dstq=q2;key=my-key;args=

Binding arguments args has an empty value by default, i.e. there are no binding arguments.
If the binding includes binding arguments, args will be an Erlang portable term hash
provided by the server similar to what’s provided in HTTP API v1 today.
Alternatively, we could use an arguments scheme of:

args=k1,utf8,v1&k2,uint,3

However, such a scheme leads to long URIs when there are many binding arguments.
Note that it’s perfectly fine for URI producing applications to include URI
reserved characters = / ; / , / $ in a path segment.

To create a binding, the client therefore needs to POST to a bindings factory URI:

POST
/vhosts/:vhost/bindings

To list all bindings between a source exchange e1 and destination exchange e2 with binding key k1:

GET
/vhosts/:vhost/bindings?src=e1&dste=e2&key=k1

This endpoint will be called by the RabbitMQ AMQP 1.0 client library to unbind a
binding with non-empty binding arguments to get the binding ID before invoking a

DELETE
/vhosts/:vhost/bindings/:binding

In future, after RabbitMQ 4.0 shipped, new API endpoints could be added.
The following is up for discussion and is only meant to show the clean and simple design of HTTP API v2.

Bindings endpoint can be queried as follows:

to list all bindings for a given source exchange e1:

GET
/vhosts/:vhost/bindings?src=e1

to list all bindings for a given destination queue q1:

GET
/vhosts/:vhost/bindings?dstq=q1

to list all bindings between a source exchange e1 and destination queue q1:

GET
/vhosts/:vhost/bindings?src=e1&dstq=q1

multiple bindings between source exchange e1 and destination queue q1 could be deleted at once as follows:

DELETE /vhosts/:vhost/bindings?src=e1&dstq=q1

GET could be supported globally across all vhosts:

/exchanges
/queues
/bindings

Publish a message:

POST
/vhosts/:vhost/queues/:queue/messages

Consume or peek a message (depending on query parameters):

GET
/vhosts/:vhost/queues/:queue/messages

Note that the AMQP 1.0 client omits the /vhost/:vhost path prefix.
Since an AMQP connection belongs to a single vhost, there is no need to
additionally include the vhost in every HTTP request.

Pros of HTTP API v2:

1. Low level of nesting

Queues, exchanges, bindings are top level entities directly under vhosts.
Although the HTTP API doesn’t have to reflect how resources are stored in the database,
v2 does nicely reflect the Khepri tree structure.

2. Nouns instead of verbs
   HTTP API v2 is very simple to read and understand as shown by

POST    /vhosts/:vhost/queues/:queue/messages	to post messages, i.e. publish to a queue.
GET     /vhosts/:vhost/queues/:queue/messages	to get messages, i.e. consume or peek from a queue.
DELETE  /vhosts/:vhost/queues/:queue/messages	to delete messages, i.e. purge a queue.

A separate new HTTP API v2 allows us to ship only handlers for HTTP over AMQP for RabbitMQ 4.0
and therefore move faster while still keeping the option on the table to re-use the new v2 API
for pure HTTP in the future.
In contrast, re-using the HTTP API v1 for HTTP over AMQP is possible, but dirty because separate handlers
(HTTP over AMQP and pure HTTP) replying differently will be needed for the same v1 endpoints.

[lhoguin]

I am only reviewing with regard to what this implies for a future HTTP API v2 (best practices, etc.).

The URI space and methods are correct.

The data returned is strongly validated; there can't be an extra unknown field. I would suggest ignoring field names that are not known as this allows for better extensibility. On the other hand, there should be a check that all required fields are found. In the future schemas could be used for validation; this is not necessary at this time. So:

• Ignore unknown field names
• Check that all required fields are found

All points below do not require action but should be taken into consideration for future improvements if we want this API and the hypothetical HTTP API v2 to be common, either from a code PoV or a spec PoV.

There is currently no content-type for the data returned; this is mandated by the spec. For HTTP there will need to be a content type for each type of data returned. (No action necessary for this PR.)

Ideally we shouldn't have to list bindings to get a URI with a hash to perform the DELETE operation. The hash algorithm should be a generally available algorithm, not an Erlang-specific one. To use a different hash algorithm in the future we will need to use a different query argument name and phase out the current argument. The whole search_binding_uri code will become unnecessary then. (No action necessary for this PR.)

There is currently no relationship returned when a URI is provided, which currently only happens when listing bindings to be deleted. Link relations will be important for building smarter clients. This can be added at a later time. (No action necessary for this PR.)

Building URIs manually is not recommended. In the future URI templates (Cowlib implementation) should be used. URI templates allow for smarter clients as the server can provide the client with the template. (No action necessary for this PR.)

All responses with bodies should include links (URIs + relations + types) of parents, siblings (if applicable), children (if applicable)... to allow quick and easy navigation. For example a queue details would have a link to the .../messages URI for the queue's messages. In addition it should also include a canonical link of itself. (No action necessary for this PR.)

The client currently doesn't follow redirects (201, 3xx); this will become necessary should we want to move URIs around, or not provide information of created objects systematically (for example creating a queue with an empty 201 response, the client could optionally do a request to get the details). This will become important later on for increased extensibility. (No action necessary for this PR.)

[lhoguin]

The actionable items I had have been addressed.

Again this review is only an approval of the API design. People more familiar with AMQP 1.0 should review these parts. Thank you.