ILP Over HTTP

[emschwartz]

This is a write-up of the protocol used in the ILPv3 experiment.

Thanks to @sappenin for the prodding to write it up as a spec.

Note this protocol would be compatible with #347.

@justmoon brought up a good point that by replacing the ILP packet encoding with purely semantically defined fields, we could be giving up on an important type of extensibility. Based on an idea from @sappenin, this document specifies that all headers that start with ILP- MUST be forwarded. This could be a way of achieving the kind of extensibility that @justmoon was concerned about, which would effectively say that the "transfer" object can be extended with arbitrary key-value pairs (as UTF-8 strings). What do you think?

[emschwartz]

Note that to support both a response body and the additional headers, we would probably want to add a data property to the LPIv2 error, in addition to the additionalInfo.

[adrianhopebailie]

@justmoon brought up a good point that by replacing the ILP packet encoding with purely semantically defined fields, we could be giving up on an important type of extensibility

Not sure I agree with this. If there is data that needs to be forwarded blindly by the connector then it should be in the body.

[adrianhopebailie]

Don't think this is necessary. anything that goes in the headers is ledger layer data and by implication the server should understand it.

[emschwartz]

Regarding @adrianhopebailie's comment:

If there is data that needs to be forwarded blindly by the connector then it should be in the body.

Right now the body is intended for communication between the sender and receiver. What's missing from that is any kind of information the sender may want to communicate to the connectors or that connectors may want to attach to the payment. If transport layer protocols don't have a way to tell the receiver to ignore some portion of the data, attaching additional information would cause the payment to fail if the receiver HMACs the data to generate the fulfillment.

I think the questions we need to answer about the type of extensibility we want are:

1. Who is going to set the additional fields? Will they be supported and set by Transport Layer protocols, by the sender's ILP implementation, or by connectors?
2. Are we looking for the type of extensibility where ILP's IETF would define new fields on the Interledger layer, or something more open like HTTP headers where anyone can define new ones?
3. How do we ensure that the new features are forwarded even by connectors that don't understand them (which would allow for rolling upgrades, rather than requiring everyone to upgrade to use the new feature at once)?

[adrianhopebailie]

In my opinion we are falling into the trap of mixing protocol layers again.

Right now the body is intended for communication between the sender and receiver.

That's not strictly speaking true. The body is intended for use by any protocol in the stack above ILP.

If a transport layer protocol defines data that is only for the senders and receivers but also data that must be visible to the intermediaries then it should define an envelop that is carried in the body of the ledger layer message and distinguishes between these.

PSK does this already as one example.

I'd consider any transport layer that doesn't provide a way to pass both public and protected (encrypted by sender/receiver) data as a non-starter anyway.

Consider a transport layer protocol that needs to provide compliance data to each intermediary. So each connector attaches a whole lot of data to the transfer on each hop that describes who they are and a signature etc.

There is no way you want that passed in the headers of the ledger layer protocol.

To answer your questions:

1. Protocols don't set fields. They define which fields can be set by who. A good transport layer protocol will define both fields that can be set by intermediaries and by the sender and receiver but these will all be in the body of the ledger layer message.

2. Yes. The ILP layer is very rigid and is unlikely to change again. It carries a local transfer amount, an address, a condition, a timeout and the transport layer data. There is nothing left to remove and very unlikely to be anything more to add that shouldn't be in the transport layer.

3. Put them in the body of the transfer message.

[emschwartz]

I agree that transport protocols that want to expose data for intermediaries to see can do so.

I think this is what @justmoon is worried about:

The ILP layer is very rigid...and very unlikely to be anything more to add that shouldn't be in the transport layer.

It comes down to whether the Interledger layer itself should have extensibility. We can add extensibility at the ledger and transport layers easily, but if there's a field we realized we forgot or desperately need at the Interledger layer it would be a huge pain or near impossibility to add later.

[adrianhopebailie]

It comes down to whether the Interledger layer itself should have extensibility

I agree and I think the answer is an emphatic no. The inter-networking layer needs to be simple and never change.

[sappenin]

A good transport layer protocol will define both fields that can be set by intermediaries and by the sender and receiver but these will all be in the body of the ledger layer message.

I don't think this spec should be prescriptive about what goes into the body of the HTTP request. For example, imagine some sender/receiver that don't want to stuff all of their payment data into this layer at all, but instead simply want to hash it, encrypt it, and then propagate a link to that data?

This protocol should be able to allow a sender and receiver to do that without involving the <body> at all. As an example, to satisfy the above, one could define a URL-link to shared data that might be too large to send across every hop, resulting in an HTTP request that looks like this:

{
POST / HTTP/1.1
ILP-Destination: g.crypto.bitcoin.1XPTgDRhN8RFnzniWCddobD9iKZatrvH4.~asdf1234
ILP-Condition: x73kz0AGyqYqhw/c5LqMhSgpcOLF3rBS8GdR52hLpB8=
ILP-Expiry: 2017-12-07T18:47:59.015Z
ILP-Amount: 1000
ILP-SenderReceiver-Encrypted-Data: https://s3.exmple.com?id=32e77ef1961f1c9c5135bff6e939b7ef3b4e8c7defa61d50dca797c86f97cc91
}

[sappenin]

It comes down to whether the Interledger layer itself should have extensibility. We can add extensibility at the ledger and transport layers easily, but if there's a field we realized we forgot or desperately need at the Interledger layer it would be a huge pain or near impossibility to add later.

It seems like all we need is an extensible way to allow intermediate hops (per this spec) to be able to identify how to handle headers according to the following:

1. Which header's data do I need to adjust, and then forward? (e.g., Amount, expiresAt)
2. Which header's data MUST I forward unchanged? (e.g., destination, executionCondition)
3. Which header's data SHOULD I forward unchanged?
4. Which header's data is meant only for me?

Is seems like we could use a combination of header-prefixes and/or this spec to cover all of the above cases, and allow for extensibility.

Maybe a simple rule would be: Read the spec, do what it says, and then: for any other field you encounter that starts with ILP-..., you MUST forward to any next-hop, unchanged.

[adrianhopebailie]

Don't think this is necessary. Transport layers that want to expose the destination amount should do it in the body. No transport layer data should be carried in the headers.

[sappenin]

Seems like if this spec has a well-defined way to handle reserved-prefix ILP-... headers, it shouldn't matter if somebody uses headers or the body, no?

[adrianhopebailie]

As above, don't think this is necessary

[adrianhopebailie]

Don't think this is necessary, as above.

[michielbdejong]

I would go further and say 'this is what Interledger looks like'.

[michielbdejong]

base64url encode the condition as x73kz0AGyqYqhw_c5LqMhSgpcOLF3rBS8GdR52hLpB8?

[adrianhopebailie]

What is the motivation for using base64url (slightly obscure) vs base64?

Most standards I have come across use base64 unless they need to put the encoded data in a URL.

Also, as @michielbdejong points out, we should be consistent on padding

[sappenin]

Do we ever anticipate putting a condition into a URL? Given that this is specifying an HTTP binding for ILP-core, it seems like the answer would be, "probably."

If that answer is accurate, then we probably want to be consistent and allow for that (adding conditions to URLs) to happen easily without transformation (?).

For example, might some client (like an admin system or something) want to be able to query an ILP server for transfers, possibly identifying them by condition, like:

GET: /transfers?condition=x73kz0AGyqYqhw_c5LqMhSgpcOLF3rBS8GdR52hLpB8

While we're at it, is there an argument to use HEX encoding here? That way nobody will be confused about whether we used URL encoding or not? (There was some confusion about this in the crypto-conditions spec and @justmoon had a good rationale for when to use which encoding, but can't remember the details -- maybe it was just HEX for the DER?)

[emschwartz]

My vote would either be for normal base64 or hex.

Using a module that supports base64url or rewriting the function that converts it all over the place (see the JS code for examples) is annoying enough that I'd prefer to avoid it. It's just a little bit less common and we're not actively using the condition in any URLs at the moment. A different protocol that has support for features like the one you describe could choose an entirely different encoding format or strategy, so I don't think this protocol needs to be designed with that in mind.

[michielbdejong]

An advantage of hex is that it's more human-readable when comparing raw buffers in debug output:

> Buffer.from('0a101033', 'hex')
<Buffer 0a 10 10 33>
> Buffer.from('0a101033', 'base64')
<Buffer d1 ad 74 d7 4d f7>

Another advantage of hex would be that it would be less confusing to change to hex (obviously different from base64url) than to change from base64url to base64 (more subtle change, easier to confuse).

[sappenin]

+1 to all comments above from @emschwartz, @michielbdejong, and @adrianhopebailie .

I think we have consensus that we should not use Base64Url, and from my perspective I lean slightly towards HEX, but would be fine supporting Base64.

With that in mind, I propose that we use HEX (and if there's even one dissenter saying that we should use Base64, then let's just use that).

Any dissenters?

[emschwartz]

Actually I would lean towards normal base64. Since this is specifically for encoding a hash or random 32-byte value, you don't need to do byte comparisons. I would agree with @michielbdejong's point if we were talking about a data field, but the hashes will always be completely different. Base64 saves 20 characters, and while this isn't really optimizing for bytes on the wire, I think we might as well save those characters because we're communicating the same information anyway.

[sappenin]

No objections from me.

[sharafian]

one thing we should specify for base64 is whether padding is optional or not. Many implementations of base64 (not node.js's version, but the openssl base64 command, for instance) won't accept it if it doesn't include the =s for padding.

[michielbdejong]

Same, cz_9RGv1PVjhKIOoyPvWkAs8KrBpIJh8UrYsQ8j34CQ?

[sappenin]

Do we ever anticipate putting a condition into a URL? Given that this is specifying an HTTP binding for ILP-core, it seems like the answer would be, "probably."

If that answer is accurate, then we probably want to be consistent and allow for that (adding conditions to URLs) to happen easily without transformation (?).

For example, might some client (like an admin system or something) want to be able to query an ILP server for transfers, possibly identifying them by condition, like:

GET: /transfers?condition=x73kz0AGyqYqhw_c5LqMhSgpcOLF3rBS8GdR52hLpB8

While we're at it, is there an argument to use HEX encoding here? That way nobody will be confused about whether we used URL encoding or not? (There was some confusion about this in the crypto-conditions spec and @justmoon had a good rationale for when to use which encoding, but can't remember the details -- maybe it was just HEX for the DER?)

[sappenin]

A good transport layer protocol will define both fields that can be set by intermediaries and by the sender and receiver but these will all be in the body of the ledger layer message.

I don't think this spec should be prescriptive about what goes into the body of the HTTP request. For example, imagine some sender/receiver that don't want to stuff all of their payment data into this layer at all, but instead simply want to hash it, encrypt it, and then propagate a link to that data?

This protocol should be able to allow a sender and receiver to do that without involving the <body> at all. As an example, to satisfy the above, one could define a URL-link to shared data that might be too large to send across every hop, resulting in an HTTP request that looks like this:

{
POST / HTTP/1.1
ILP-Destination: g.crypto.bitcoin.1XPTgDRhN8RFnzniWCddobD9iKZatrvH4.~asdf1234
ILP-Condition: x73kz0AGyqYqhw/c5LqMhSgpcOLF3rBS8GdR52hLpB8=
ILP-Expiry: 2017-12-07T18:47:59.015Z
ILP-Amount: 1000
ILP-SenderReceiver-Encrypted-Data: https://s3.exmple.com?id=32e77ef1961f1c9c5135bff6e939b7ef3b4e8c7defa61d50dca797c86f97cc91
}

[sappenin]

It comes down to whether the Interledger layer itself should have extensibility. We can add extensibility at the ledger and transport layers easily, but if there's a field we realized we forgot or desperately need at the Interledger layer it would be a huge pain or near impossibility to add later.

It seems like all we need is an extensible way to allow intermediate hops (per this spec) to be able to identify how to handle headers according to the following:

1. Which header's data do I need to adjust, and then forward? (e.g., Amount, expiresAt)
2. Which header's data MUST I forward unchanged? (e.g., destination, executionCondition)
3. Which header's data SHOULD I forward unchanged?
4. Which header's data is meant only for me?

Is seems like we could use a combination of header-prefixes and/or this spec to cover all of the above cases, and allow for extensibility.

Maybe a simple rule would be: Read the spec, do what it says, and then: for any other field you encounter that starts with ILP-..., you MUST forward to any next-hop, unchanged.

[sappenin]

Seems like if this spec has a well-defined way to handle reserved-prefix ILP-... headers, it shouldn't matter if somebody uses headers or the body, no?

[emschwartz]

Removed the extra ILP- header thing. @adrianhopebailie @sappenin @justmoon, thoughts?

[sappenin]

Removed the extra ILP- header thing. @adrianhopebailie @sappenin @justmoon, thoughts?

I like the simplicity, but what's the story on extensibility? We just need a new version then, and hope the "network" upgrades?

[emschwartz]

If we take out that type of extensibility we would be saying the Interledger layer itself is not extensible. This is the extensibility story we'd end up with:

Supported

• Transport layer protocols can have their own extensibility mechanisms to communicate more details between the sender and receiver
• Transport layer protocols designed to communicate some information to intermediary connectors (they would define a format for the fields, which would go in the data / body)
• Connectors forwarding other fields alongside the core ILP details

Not Supported

• Connectors forwarding additional fields that they do not understand (meaning that any additional fields that some connectors forward would be dropped by the first party that does not understand them)
• Connectors adding to or modifying the data / body
• The sender's ILP module adding fields to the data without the Transport Protocol supporting that

[justmoon]

I would make the following changes:

• Remove the following headers:
  • ILP-Destination
  • ILP-Destination-Amount
  • ILP-Error-Code
  • ILP-Error-Name
  • ILP-Error-Triggered-By
  • ILP-Error-Triggered-At
  • ILP-Error-Forwarded-By
• Define the body to be an ILP packet instead of transport layer data

This is because I think we should maintain the separation of the ledger and Interledger layers, see this comment.

[sharafian]

What's the encoding of the ILP packet in the body? Octet stream or would we put it in base64?

[emschwartz]

What's the encoding of the ILP packet in the body? Octet stream or would we put it in base64?

Definitely keep it binary. Since HTTP supports binary data there's no reason to base64 it.

[adrianhopebailie]

Propose we define a new content type for this that makes it explicit that the body is an ILP packet.

[sharafian]

Wouldn't that confuse a lot of HTTP clients?

[adrianhopebailie]

Wouldn't that confuse a lot of HTTP clients?

It shouldn't if we follow the standard.

I'd recommend something like application/interledger;encoding=oer

Could be shortened to application/interledger where the default encoding is understood to be OER.

Any client that sees a type of application should treat the HTTP body as an opaque blob. If it recognizes the MIME type then it knows what to do with it.

[michielbdejong]

I realize that there are many different ways of doing this, and my personal preference, for symmetry with our new way of using BTP Message/Response/Error, would be to follow @justmoon's proposal of making the HTTP body an ILP packet.

But for now, @earizon and I are using the current version as the API between Quilt and Amundsen. On the Amundsen side, I'll be using this super-simple ilp-plugin-http, combined with the newly refactored ilp-connector, to quickly code this up.

[emschwartz]

Closing this now because we decided to go with a specific packet encoding. We can open another PR to create a version of an HTTP-based ledger protocol that sends the ILP packets in the HTTP body.