Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Enable AMQP 1.0 clients to manage topologies
## What? * Allow AMQP 1.0 clients to dynamically create and delete RabbitMQ topologies (exchanges, queues, bindings). * Provide an Erlang AMQP 1.0 client that manages topologies. ## Why? Today, RabbitMQ topologies can be created via: * [Management HTTP API](https://www.rabbitmq.com/docs/management#http-api) (including Management UI and [messaging-topology-operator](https://github.com/rabbitmq/messaging-topology-operator)) * [Definition Import](https://www.rabbitmq.com/docs/definitions#import) * AMQP 0.9.1 clients Up to RabbitMQ 3.13 the RabbitMQ AMQP 1.0 plugin auto creates queues and bindings depending on the terminus [address format](https://github.com/rabbitmq/rabbitmq-server/tree/v3.13.x/deps/rabbitmq_amqp1_0#routing-and-addressing). Such implicit creation of topologies is limiting and obscure. For some address formats, queues will be created, but not deleted. Some of RabbitMQ's success is due to its flexible routing topologies that AMQP 0.9.1 clients can create and delete dynamically. This commit allows dynamic management of topologies for AMQP 1.0 clients. This commit builds on top of Native AMQP 1.0 (PR #9022) and will be available in RabbitMQ 4.0. ## How? This commits adds the following management operations for AMQP 1.0 clients: * declare queue * delete queue * purge queue * bind queue to exchange * unbind queue from exchange * declare exchange * delete exchange * bind exchange to exchange * unbind exchange from exchange Hence, at least the AMQP 0.9.1 management operations are supported for AMQP 1.0 clients. In addition the operation * get queue is provided which - similar to `declare queue` - returns queue information including the current leader and replicas. This allows clients to publish or consume locally on the node that hosts the queue. Compared to AMQP 0.9.1 whose commands and command fields are fixed, the new AMQP Management API is extensible: New operations and new fields can easily be added in the future. There are different design options how management operations could be supported for AMQP 1.0 clients: 1. Use a special exchange type as done in https://github.com/rabbitmq/rabbitmq-management-exchange This has the advantage that any protocol client (e.g. also STOMP clients) could dynamically manage topologies. However, a special exchange type is the wrong abstraction. 2. Clients could send "special" messages with special headers that the broker interprets. This commit decided for a variation of the 2nd option using a more standardized way by re-using a subest of the following latest AMQP 1.0 extension specifications: * [AMQP Request-Response Messaging with Link Pairing Version 1.0 - Committee Specification 01](https://docs.oasis-open.org/amqp/linkpair/v1.0/cs01/linkpair-v1.0-cs01.html) (February 2021) * [HTTP Semantics and Content over AMQP Version 1.0 - Working Draft 06](https://groups.oasis-open.org/higherlogic/ws/public/document?document_id=65571) (July 2019) * [AMQP Management Version 1.0 - Working Draft 16](https://groups.oasis-open.org/higherlogic/ws/public/document?document_id=65575) (July 2019) An important goal is to keep the interaction between AMQP 1.0 client and RabbitMQ simple to increase usage, development and adoptability of future RabbitMQ AMQP 1.0 client library wrappers. The AMQP 1.0 client has to create a link pair to the special `/management` node. This allows the client to send and receive from the management node. Similar to AMQP 0.9.1, there is no need for a reply queue since the reply will be sent directly to the client. Requests and responses are modelled via HTTP, but sent via AMQP using the `HTTP Semantics and Content over AMQP` extension (henceforth `HTTP over AMQP` extension). This commit tries to follow the `HTTP over AMQP` extension as much as possible but deviates where this draft spec doesn't make sense. The projected mode §4.1 is used as opposed to tunneled mode §4.2. A named relay `/management` is used (§6.3) where the message field `to` is the URL. Deviations are * §3.1 mandates that URIs are not encoded in an AMQP message. However, we percent encode URIs in the AMQP message. Otherwise there is for example no way to distinguish a `/` in a queue name from the URI path separator `/`. * §4.1.4 mandates a data section. This commit uses an amqp-value section as it's a better fit given that the content is AMQP encoded data. Using an HTTP API allows for a common well understood interface and future extensibility. Instead of re-using the current RabbitMQ HTTP API, this commit uses a new HTTP API (let's call it v2) which could be used as a future API for plain HTTP clients. ### HTTP API v1 The current HTTP API (let's call it v1) is **not** used since v1 comes with a couple of weaknesses: 1. Deep level of nesting becomes confusing and difficult to manage. Examples of deep nesting in v1: ``` /api/bindings/vhost/e/source/e/destination/props /api/bindings/vhost/e/exchange/q/queue/props ``` 2. Redundant endpoints returning the same resources v1 has 9 endpoints to list binding(s): ``` /api/exchanges/vhost/name/bindings/source /api/exchanges/vhost/name/bindings/destination /api/queues/vhost/name/bindings /api/bindings /api/bindings/vhost /api/bindings/vhost/e/exchange/q/queue /api/bindings/vhost/e/exchange/q/queue/props /api/bindings/vhost/e/source/e/destination /api/bindings/vhost/e/source/e/destination/props ``` 3. Verbs in path names Path names should be nouns instead. v1 contains verbs: ``` /api/queues/vhost/name/get /api/exchanges/vhost/name/publish ``` ### AMQP Management extension Only few aspects of the AMQP Management extension are used. The central idea of the AMQP management spec is **dynamic discovery** such that broker independent AMQP 1.0 clients can discover objects, types, operations, and HTTP endpoints of specific brokers. In fact, clients are only conformant if: > All request addresses are dynamically discovered starting from the discovery document. > A requesting container MUST NOT use fixed assumptions about the addressing structure of the management API. While this is a nice and powerful idea, no AMQP 1.0 client and no AMQP 1.0 server implement the latest AMQP 1.0 management spec from 2019, partly presumably due to its complexity. Therefore, the idea of such dynamic discovery has failed to be implemented in practice. The AMQP management spec mandates that the management endpoint returns a discovery document containing broker specific collections, types, configuration, and operations including their endpoints. The API endpoints of the AMQP management spec are therefore all designed around dynamic discovery. For example, to create either a queue or an exchange, the client has to ``` POST /$management/entities ``` which shows that the entities collection acts as a generic factory, see section 2.2. The server will then create the resource and reply with a location header containing a URI pointing to the resource. For RabbitMQ, we don’t need such a generic factory to create queues or exchanges. To list bindings for a queue Q1, the spec suggests ``` GET /$management/Queues/Q1/$management/entities ``` which again shows the generic entities endpoint as well as a `$management` endpoint under Q1 to allow a queue to return a discovery document. For RabbitMQ, we don’t need such generic endpoints and discovery documents. Given we aim for our own thin RabbitMQ AMQP 1.0 client wrapper libraries which expose the RabbitMQ model to the developer, we can directly use fixed HTTP endpoint assumptions in our RabbitMQ specific libraries. This is by far simpler than using the dynamic endpoints of the management spec. Simplicity leads to higher adoption and enables more developers to write RabbitMQ AMQP 1.0 client library wrappers. The AMQP Management extension also suffers from deep level of nesting in paths Examples: ``` /$management/Queues/Q1/$management/entities /$management/Queues/Q1/Bindings/Binding1 ``` as well as verbs in path names: Section 7.1.4 suggests using verbs in path names, for example “purge”, due to the dynamic operations discovery document. ### HTTP API v2 This commit introduces a new HTTP API v2 following best practices. It could serve as a future API for plain HTTP clients. This commit and RabbitMQ 4.0 will only implement a minimal set of HTTP API v2 endpoints and only for HTTP over AMQP. In other words, the existing HTTP API v1 Cowboy handlers will continue to be used for all plain HTTP requests in RabbitMQ 4.0 and will remain untouched for RabbitMQ 4.0. Over time, after 4.0 shipped, we could ship a pure HTTP API implementation for HTTP API v2. Hence, the new HTTP API v2 endpoints for HTTP over AMQP should be designed such that they can be re-used in the future for a pure HTTP implementation. The minimal set of endpoints for RabbitMQ 4.0 are: `` GET / PUT / DELETE /vhosts/:vhost/queues/:queue ``` read, create, delete a queue ``` DELETE /vhosts/:vhost/queues/:queue/messages ``` purges a queue ``` GET / DELETE /vhosts/:vhost/bindings/:binding ``` read, delete bindings where `:binding` is a binding ID of the following path segment: ``` src=e1;dstq=q2;key=my-key;args= ``` Binding arguments `args` has an empty value by default, i.e. there are no binding arguments. If the binding includes binding arguments, `args` will be an Erlang portable term hash provided by the server similar to what’s provided in HTTP API v1 today. Alternatively, we could use an arguments scheme of: ``` args=k1,utf8,v1&k2,uint,3 ``` However, such a scheme leads to long URIs when there are many binding arguments. Note that it’s perfectly fine for URI producing applications to include URI reserved characters `=` / `;` / `,` / `$` in a path segment. To create a binding, the client therefore needs to POST to a bindings factory URI: ``` POST /vhosts/:vhost/bindings ``` To list all bindings between a source exchange e1 and destination exchange e2 with binding key k1: ``` GET /vhosts/:vhost/bindings?src=e1&dste=e2&key=k1 ``` This endpoint will be called by the RabbitMQ AMQP 1.0 client library to unbind a binding with non-empty binding arguments to get the binding ID before invoking a ``` DELETE /vhosts/:vhost/bindings/:binding ``` In future, after RabbitMQ 4.0 shipped, new API endpoints could be added. The following is up for discussion and is only meant to show the clean and simple design of HTTP API v2. Bindings endpoint can be queried as follows: to list all bindings for a given source exchange e1: ``` GET /vhosts/:vhost/bindings?src=e1 ``` to list all bindings for a given destination queue q1: ``` GET /vhosts/:vhost/bindings?dstq=q1 ``` to list all bindings between a source exchange e1 and destination queue q1: ``` GET /vhosts/:vhost/bindings?src=e1&dstq=q1 ``` multiple bindings between source exchange e1 and destination queue q1 could be deleted at once as follows: ``` DELETE /vhosts/:vhost/bindings?src=e1&dstq=q1 ``` GET could be supported globally across all vhosts: ``` /exchanges /queues /bindings ``` Publish a message: ``` POST /vhosts/:vhost/queues/:queue/messages ``` Consume or peek a message (depending on query parameters): ``` GET /vhosts/:vhost/queues/:queue/messages ``` Note that the AMQP 1.0 client omits the `/vhost/:vhost` path prefix. Since an AMQP connection belongs to a single vhost, there is no need to additionally include the vhost in every HTTP request. Pros of HTTP API v2: 1. Low level of nesting Queues, exchanges, bindings are top level entities directly under vhosts. Although the HTTP API doesn’t have to reflect how resources are stored in the database, v2 does nicely reflect the Khepri tree structure. 2. Nouns instead of verbs HTTP API v2 is very simple to read and understand as shown by ``` POST /vhosts/:vhost/queues/:queue/messages to post messages, i.e. publish to a queue. GET /vhosts/:vhost/queues/:queue/messages to get messages, i.e. consume or peek from a queue. DELETE /vhosts/:vhost/queues/:queue/messages to delete messages, i.e. purge a queue. ``` A separate new HTTP API v2 allows us to ship only handlers for HTTP over AMQP for RabbitMQ 4.0 and therefore move faster while still keeping the option on the table to re-use the new v2 API for pure HTTP in the future. In contrast, re-using the HTTP API v1 for HTTP over AMQP is possible, but dirty because separate handlers (HTTP over AMQP and pure HTTP) replying differently will be needed for the same v1 endpoints.
- Loading branch information
