Multi-hop packet routing in IBC

[cwgoes]

Followup from #126.

Think about multi-hop case, prepare IBC protocol for eventual usage as such.

• Construct data structures accordingly(e.g. channels are end-to-end)
• Write a trivial routing module which can handle loopback, and has the right interface for multi-hop
• Currently fail on attempts to open multi-hop channels

[cwgoes]

Types of multi-hop routing:

1. "Application-layer multihop"
   A talks to H, H talks to B
   No routing at IBC protocol layer, just two connections / channels.
   We can do this already.
2. "Validity proxy multihop"
   A talks to B using H's validity predicate for B
A talks to H • B , B talks to H • A
A verifies the state of B using the root + block height validated by H.
   Connection and channel from A to B using proxy validity predicates.
   This is relatively easy, just a different type of client.
3. "Routing layer multihop"
   A has a connection to H, H has a connection to B.
   A has a channel to B through H.
   A verifies the state of H, H verifies the state & block height of B, vice versa.
   This will require a routing module in the IBC core protocol.

Hops could be called "links".

[cwgoes]

What we need to think about immediately is just what data structure alterations will be required for (3).

[cwgoes]

• Channel opening handshake will specify a list of connection hops instead of a single connection
• Packets will specify a list of connection hops instead of single origination & destination connections
• An "IBC routing" module will differentiate packets intended for this chain and packets intended for forwarding, and in the latter case pop the current hop off the hop list and forward the packet to the next hop
• This routing module will also handle a special sentinel loopback connection from a chain to itself
• Channel semantics are end-to-end, so timeouts will need to be conveyed through intermediary hops (this may require extra "packet types")

[zmanian]

Some things confuse me about the multihop use case.

If the system is end to end, do I show validaty proofs in consensus for all the intermediary hops to the end hops or just the validity proofs?

Can we think of this as wrapping an IBC packet inside another IBC packet?

I can sort of imagine in two ways.

In one way, effectively Alice and Charlie have a channel but no connection. The connection is provided by Bob'ss connection to Alice and Bob's connection to Charlie.

Must packets in that channel be source routed so that they say they are only valid if they have originated over the specified connections?

[cwgoes]

If the system is end to end, do I show validaty proofs in consensus for all the intermediary hops to the end hops or just the validity proofs?

In model (3), the verification steps for a packet from A to B through H would be as follows:

• A commits outgoing packet to H with routing information indicating a next-hop from H to B
• H receives incoming packet, checks that A committed outgoing correctly
• H parses packet routing metadata (but not the packet data itself), identifies next-hop to B
• H strips the first hop from the routing metadata and commits the packet outgoing to B
• B verifies that H committed the outgoing packet (which is tagged as being originally-from A)
• B processes accordingly (e.g. mints token vouchers)

I believe this is the first way you are describing (though I am not quite sure which was the second way), and once the handshake takes place (the handshake would involve A, B, and H for such a channel, and create a routing entry in a routing table on H) that validity checking will take place.

[ancazamfir]

• Channel opening handshake will specify a list of connection hops instead of a single connection

How does A know the connections/ hops to reach B?

• Packets will specify a list of connection hops instead of single origination & destination connections

Is this needed if the channel is "pinned" to a sequence of connections?

• An "IBC routing" module will differentiate packets intended for this chain and packets intended for forwarding, and in the latter case pop the current hop off the hop list and forward the packet to the next hop

When a channel is setup by A, the "IBC routing" module on H determines the next hop/ connection towards the destination B. At this point, if we follow a typical routing implementation, a "forwarding" entry is set so a simple lookup for future packets destined to B will be performed. In the presence of pre-established channels I don't understand why the packets need to be source routed.
I know this is not planned any time soon but I believe it would be useful to think over a more complex topology of blockchains/ hubs.

[cwgoes]

How does A know the connections/ hops to reach B?

A doesn't know anything in particular (although it may have existing connections). The user or module creating the channel must know the connections which will be used.

Is this needed if the channel is "pinned" to a sequence of connections?

I think so - just for lookup purposes, not semantically. Alternatively we could decide that channel names are unique on a single chain and avoid this.

When a channel is setup by A, the "IBC routing" module on H determines the next hop/ connection towards the destination B. At this point, if we follow a typical routing implementation, a "forwarding" entry is set so a simple lookup for future packets destined to B will be performed. In the presence of pre-established channels I don't understand why the packets need to be source routed.
I know this is not planned any time soon but I believe it would be useful to think over a more complex topology of blockchains/ hubs.

I am not quite sure I follow. Are you suggesting packet-switched routing? I don't think that makes sense for IBC because the security assumptions change depending on the route - source routing enables the sender to choose all the hops and thus all the security assumptions.

[ancazamfir]

I am not quite sure I follow. Are you suggesting packet-switched routing? I don't think that makes sense for IBC because the security assumptions change depending on the route - source routing enables the sender to choose all the hops and thus all the security assumptions.

Aren't packets over a channel following the same route as the channel itself? why do they need again the source route?

[cwgoes]

Aren't packets over a channel following the same route as the channel itself? why do they need again the source route?

They need the connection identifiers to distinguish between channels which might have the same name but use different connections.

[ancazamfir]

They need the connection identifiers to distinguish between channels which might have the same name but use different connections.

I understand this but I have a feeling the intermediate connection info is redundant and only source and destination connections are required; since the source route has been previously passed in during channel handshake the next hop/ connection should be known by intermediate nodes when they look up the destination channel ID+connection.

[cwgoes]

I understand this but I have a feeling the intermediate connection info is redundant and only source and destination connections are required; since the source route has been previously passed in during channel handshake the next hop/ connection should be known by intermediate nodes when they look up the destination channel ID+connection.

Agreed, that would probably work, the relevant routes could be stored in a routing table and should be unique by channel and source connection.

[ancazamfir]

Some thoughts/ questions on the channel handshake:
The Try (A->B), Ack (A<-B) and Confirm(A->B) messages must also include the list of connections, correct? And since they should show in reverse order in the Ack, maybe there shouldn't be any popping done when Try is sent over the sequence of connections (so B can construct the reverse path).
Or if there is, a reverse path should be constructed (push the popped connection to some reverse route metadata) and sent as part of the handshake message as it is forwarded by each hop.

[cwgoes]

Data structure changes for future compatibility:

• Channels are created with list of connection "hops" instead of source / dest connection identifiers.
• Channel opening handshake datagrams altered to contain these hops.
• Currently, trying to open a channel with more than one hop fails.

Is anything else required? Packet metadata can change later without an issue.

[ancazamfir]

Loopback will be handled when multi-hop is supported?

[cwgoes]

Loopback will be handled when multi-hop is supported?

Ah, let's do that now (no reason not to), excellent point.

[cwgoes]

What needs to be in the store key? @mossid asked.

@cwgoes - I think we can keep it as is, since handshake will set up hops.

[cwgoes]

Alternatively, channel identifiers could be unique per-port - that seems to make more sense...

Then the channel key = port id || channel id

[cwgoes]

Note also points raised in #65 (comment) (closed in favor of this issue).

[hxrts]

Multi-hop transaction pricing also seems like huge UX barrier if you're traversing multiple chains and may stall before completion.

edit: just noticed this issue was raised in comment 65 above

[soring323]

is there any progress on this problem?

[crodriguezvega]

is there any progress on this problem?

The spec changes required to add support for multi hop channels have been merged. There is an implementation here and we plan to upstream it to ibc-go in the first half of the year.

[crodriguezvega]

@AdityaSripal Do you think the ICS 33 PR covers the requirements described here? If yes, can we then close the issue?