From a132130c0b03c90c270d9d7642fe2c1a0a9caab0 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 14 Jul 2023 14:46:17 -0700 Subject: [PATCH 1/9] Define a simple request/response abstraction --- simple-request-response/README.md | 75 +++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 simple-request-response/README.md diff --git a/simple-request-response/README.md b/simple-request-response/README.md new file mode 100644 index 000000000..614e172b6 --- /dev/null +++ b/simple-request-response/README.md @@ -0,0 +1,75 @@ +# A simple request/response abstraction + +| Lifecycle Stage | Maturity | Status | Latest Revision | +| --------------- | ------------- | ------ | --------------- | +| 1A | Working Draft | Active | r0, 2023-07-14 | + +Authors: [@MarcoPolo] + +todo: others welcome to join +Interest Group: [@mxinden @marten-seemann] + +[@marten-seemann]: https://github.com/marten-seemann +[@MarcoPolo]: https://github.com/MarcoPolo +[@mxinden]: https://github.com/mxinden + +## Introduction + +Many application protocols map well to request/response semantics where a peer +makes a request to another peer, and receives a response back. For example, +requesting a file from a peer and getting the file back. In contrast to +request/response, stream semantics allow two peers to continously send data +back and forth with no specific ordering. Both styles have their uses. This +document focuses on _one_ way of implementing request/response semantics and +how to map it to stream based transports and an HTTP transport. + +## Goals of this document + +A goal of this document is to define _a_ simple way to write request/response +style application protocols that can make use of all libp2p transports +seamlessly (all stream transports and the HTTP transport). This is only one way +of writing request/response style protocols, and users may choose to not use it. + +Another goal is to be backwards compatible with existing application protocols +that follow these semantics to make upgrading them easier so that they may take +advantage of the new HTTP transport without sacrificing backwards compatibility. + +The final goal is to define something simple such that it's easy to follow, +implement, and use. + +## The abstraction + +This abstraction takes in a blob from the requesting peer (aka client) that +represents the request and delivers the blob to the responding peer (aka +server). The server responds with a blob that respresents the response. This +abstraction is agnostic to what the blobs are or how they are encoded. That's up +to the application protocol to decide. + +## How to run on top of a libp2p stream + +Each request and response should happen in a single stream. There MUST NOT be +pipelining. After sending a request, the client SHOULD close its write side +(signalling EOF to the peer). After handling the response, the client SHOULD +close the stream. After sending the response the server SHOULD close the stream. + +## How to run on top of an HTTP transport + +The client's request is placed in the body of an HTTP POST request. The server +places it's response in the body of the HTTP response. Headers are unused by the +application protocol (but may be used by the libp2p implementation to provide +authentication). The HTTP path used is defined by the server's +`.well-known/libp2p` HTTP resource (see the [HTTP](../http/README.md) spec for +more details). + + +## Prior Art + +This spec is inspired by existing work to use request response protocols on top +of libp2p streams including, but not limited to: +- go-libp2p-kad-dht +- Identify Push +- AutoNATV2 + +This is also inspired by rust-libp2p's [Request/Response +crate](https://docs.rs/libp2p-request-response/0.25.0/libp2p_request_response/). + From 37670f45a565e74e6617161ab99eeef51727c504 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Mon, 17 Jul 2023 13:30:21 -0700 Subject: [PATCH 2/9] typo --- simple-request-response/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/simple-request-response/README.md b/simple-request-response/README.md index 614e172b6..07c738048 100644 --- a/simple-request-response/README.md +++ b/simple-request-response/README.md @@ -55,7 +55,7 @@ close the stream. After sending the response the server SHOULD close the stream. ## How to run on top of an HTTP transport The client's request is placed in the body of an HTTP POST request. The server -places it's response in the body of the HTTP response. Headers are unused by the +places its response in the body of the HTTP response. Headers are unused by the application protocol (but may be used by the libp2p implementation to provide authentication). The HTTP path used is defined by the server's `.well-known/libp2p` HTTP resource (see the [HTTP](../http/README.md) spec for From bc696cb16fc573f8de5fd69c90ab6eda671ea3a0 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Mon, 17 Jul 2023 13:30:59 -0700 Subject: [PATCH 3/9] Update simple-request-response/README.md Co-authored-by: Max Inden --- simple-request-response/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/simple-request-response/README.md b/simple-request-response/README.md index 07c738048..094d9dda1 100644 --- a/simple-request-response/README.md +++ b/simple-request-response/README.md @@ -50,7 +50,7 @@ to the application protocol to decide. Each request and response should happen in a single stream. There MUST NOT be pipelining. After sending a request, the client SHOULD close its write side (signalling EOF to the peer). After handling the response, the client SHOULD -close the stream. After sending the response the server SHOULD close the stream. +close the stream. After sending the response the server SHOULD close its write side (signalling EOF to the peer). ## How to run on top of an HTTP transport From 1e1c96dfcb6215798d54f40e82438f62af926ac4 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Mon, 17 Jul 2023 13:32:14 -0700 Subject: [PATCH 4/9] Update simple-request-response/README.md Co-authored-by: Max Inden --- simple-request-response/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/simple-request-response/README.md b/simple-request-response/README.md index 094d9dda1..79c27316c 100644 --- a/simple-request-response/README.md +++ b/simple-request-response/README.md @@ -7,7 +7,7 @@ Authors: [@MarcoPolo] todo: others welcome to join -Interest Group: [@mxinden @marten-seemann] +Interest Group: [@mxinden] [@marten-seemann] [@marten-seemann]: https://github.com/marten-seemann [@MarcoPolo]: https://github.com/MarcoPolo From bb709b6549860db71d4f4947b1ed2afcd99df0af Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Tue, 18 Jul 2023 09:46:55 -0700 Subject: [PATCH 5/9] Server closes stream too --- simple-request-response/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/simple-request-response/README.md b/simple-request-response/README.md index 79c27316c..b227a5d11 100644 --- a/simple-request-response/README.md +++ b/simple-request-response/README.md @@ -50,7 +50,8 @@ to the application protocol to decide. Each request and response should happen in a single stream. There MUST NOT be pipelining. After sending a request, the client SHOULD close its write side (signalling EOF to the peer). After handling the response, the client SHOULD -close the stream. After sending the response the server SHOULD close its write side (signalling EOF to the peer). +close the stream. After sending the response the server SHOULD close the stream +side (signalling EOF to the peer as well as signalling no future writes will be accepted.). ## How to run on top of an HTTP transport From 0cc0cb86fa23ccc43d04e859cd563f5dbee9a095 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Tue, 18 Jul 2023 09:57:15 -0700 Subject: [PATCH 6/9] Add considerations for applications --- simple-request-response/README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/simple-request-response/README.md b/simple-request-response/README.md index b227a5d11..31205e449 100644 --- a/simple-request-response/README.md +++ b/simple-request-response/README.md @@ -62,6 +62,14 @@ authentication). The HTTP path used is defined by the server's `.well-known/libp2p` HTTP resource (see the [HTTP](../http/README.md) spec for more details). +## Considerations for applications + +- Applications should define a reasonable maximum amount of expected data, and + limit the amount of data they receive at any time. For example, Kademlia may + limit the maximum size of a request to + [16KiB](https://github.com/libp2p/rust-libp2p/blob/master/protocols/kad/src/protocol.rs#L48) + or + [4MiB](https://github.com/libp2p/go-libp2p/blob/master/core/network/network.go#L23). ## Prior Art From 82ac1c67094273a5a5de9e977465bcffc103b19a Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Wed, 19 Jul 2023 11:58:55 -0700 Subject: [PATCH 7/9] Edits after conversation with Thomas --- simple-request-response/README.md | 63 +++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 20 deletions(-) diff --git a/simple-request-response/README.md b/simple-request-response/README.md index 31205e449..3ecc705fd 100644 --- a/simple-request-response/README.md +++ b/simple-request-response/README.md @@ -1,4 +1,4 @@ -# A simple request/response abstraction +# A request/response interface for application protocols on any transport | Lifecycle Stage | Maturity | Status | Latest Revision | | --------------- | ------------- | ------ | --------------- | @@ -6,12 +6,12 @@ Authors: [@MarcoPolo] -todo: others welcome to join -Interest Group: [@mxinden] [@marten-seemann] +Interest Group: [@mxinden] [@marten-seemann] [@thomaseizinger] [@marten-seemann]: https://github.com/marten-seemann [@MarcoPolo]: https://github.com/MarcoPolo [@mxinden]: https://github.com/mxinden +[@thomaseizinger]: https://github.com/thomaseizinger ## Introduction @@ -23,12 +23,43 @@ back and forth with no specific ordering. Both styles have their uses. This document focuses on _one_ way of implementing request/response semantics and how to map it to stream based transports and an HTTP transport. +## The interface + +At its core, request-response means: +- taking _one_ blob from the requesting peer (aka the client) +- delivering this blob to the responding peer (aka the server) +- taking _one_ blob from the responding peer +- delivering this blob to the requesting peer + +The defining characteristics are: + +- Each party can only send one message +- The interaction is linear in time (the response can only be initiated after the request has been received) + +This is in contrast to for example streaming protocols where the responding peer may send any number of messages or event/notification-based protocols where a message may be received without a prior request. + +What the blobs are and how they are encoded is subject to the application +protocol and not defined in this interface. + +A rough suggestion for implementors to follow is provide something like: +``` +async fn handleRequest(request) -> response +``` + +where the request/response may be read incrementally and asynchronously and the +request/response may also be sent incrementally and asynchronously. This is +similar to how many implementations do HTTP request/response: + +- JS [Fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) api returns a [`Response`](https://fetch.spec.whatwg.org/#response) object as soon as the server returns headers. The + body of both requests and responses are `ReadableStreams`. +- Go's [http.Request](https://pkg.go.dev/net/http@go1.20.5#Request) and [http.Response](https://pkg.go.dev/net/http@go1.20.5#Response) bodies are both [`io.ReadCloser`](https://pkg.go.dev/io@go1.20.5#ReadCloser) types. +- The Rust [hyper](https://docs.rs/hyper) crate has a similar [`HTTPBody`](https://docs.rs/hyper/latest/hyper/body/trait.HttpBody.html) that reads request/response data + incrementally and asynchronously. + ## Goals of this document -A goal of this document is to define _a_ simple way to write request/response -style application protocols that can make use of all libp2p transports -seamlessly (all stream transports and the HTTP transport). This is only one way -of writing request/response style protocols, and users may choose to not use it. +The primary goal of this document is to define an interface that application +protocols can build on and transports can fulfill. Another goal is to be backwards compatible with existing application protocols that follow these semantics to make upgrading them easier so that they may take @@ -37,15 +68,7 @@ advantage of the new HTTP transport without sacrificing backwards compatibility. The final goal is to define something simple such that it's easy to follow, implement, and use. -## The abstraction - -This abstraction takes in a blob from the requesting peer (aka client) that -represents the request and delivers the blob to the responding peer (aka -server). The server responds with a blob that respresents the response. This -abstraction is agnostic to what the blobs are or how they are encoded. That's up -to the application protocol to decide. - -## How to run on top of a libp2p stream +## How to map to a libp2p stream Each request and response should happen in a single stream. There MUST NOT be pipelining. After sending a request, the client SHOULD close its write side @@ -53,14 +76,14 @@ pipelining. After sending a request, the client SHOULD close its write side close the stream. After sending the response the server SHOULD close the stream side (signalling EOF to the peer as well as signalling no future writes will be accepted.). -## How to run on top of an HTTP transport +## How to map to an HTTP transport The client's request is placed in the body of an HTTP POST request. The server places its response in the body of the HTTP response. Headers are unused by the application protocol (but may be used by the libp2p implementation to provide -authentication). The HTTP path used is defined by the server's -`.well-known/libp2p` HTTP resource (see the [HTTP](../http/README.md) spec for -more details). +authentication). The HTTP path used for the application protocol is defined by +the server's `.well-known/libp2p` HTTP resource (see the +[HTTP](../http/README.md) spec for more details). ## Considerations for applications From 9db523ba4437da3a917383badf6dcfafab259a9b Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Wed, 19 Jul 2023 11:59:40 -0700 Subject: [PATCH 8/9] Change to SHOULD NOT pipeline to keep backwards compat with things like go-kad-dht --- simple-request-response/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/simple-request-response/README.md b/simple-request-response/README.md index 3ecc705fd..c0409d7a7 100644 --- a/simple-request-response/README.md +++ b/simple-request-response/README.md @@ -70,7 +70,7 @@ implement, and use. ## How to map to a libp2p stream -Each request and response should happen in a single stream. There MUST NOT be +Each request and response should happen in a single stream. There SHOULD NOT be pipelining. After sending a request, the client SHOULD close its write side (signalling EOF to the peer). After handling the response, the client SHOULD close the stream. After sending the response the server SHOULD close the stream From 932622d7cd20466fd5d4bd3db7743427ad3f6abe Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Wed, 19 Jul 2023 13:57:22 -0700 Subject: [PATCH 9/9] Rename folder --- {simple-request-response => request-response-interface}/README.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {simple-request-response => request-response-interface}/README.md (100%) diff --git a/simple-request-response/README.md b/request-response-interface/README.md similarity index 100% rename from simple-request-response/README.md rename to request-response-interface/README.md