Skip to content

Latest commit

 

History

History
847 lines (758 loc) · 62.2 KB

v0.18.md

File metadata and controls

847 lines (758 loc) · 62.2 KB
Raw

Kubo changelog v0.18

v0.18.1

This release includes improvements around Pubsub message deduplication, libp2p resource management, and more.

🔦 Highlights

New default Pubsub.SeenMessagesStrategy

A new optional Pubsub.SeenMessagesStrategy configuration option has been added.

This option allows you to choose between two different strategies for deduplicating messages: first-seen and last-seen.

When unset, the default strategy is last-seen, which calculates the time-to-live (TTL) countdown based on the last time a message is seen. This means that if a message is received and then seen again within the specified TTL window based on the last time it was seen, it won't be emitted.

If you prefer the old behavior, which calculates the TTL countdown based on the first time a message is seen, you can set Pubsub.SeenMessagesStrategy to first-seen.

Improving libp2p resource management integration

TL;DR: limit autoscaling improved, most users should start with default settings. If you have old configuration, switch to implicit defaults:

ipfs config --json -- Swarm.ResourceMgr '{}'
ipfs config --json -- Swarm.ConnMgr '{}'

IF you run a server and want to utilize more than half of memory and file descriptors to p2p work, adjust Swarm.ResourceMgr.MaxMemory and Swarm.ResourceMgr.MaxFileDescriptors.

The 0.18.1 builds on the default protection nodes get against DoS (resource exhaustion) and eclipse attacks with the go-libp2p Network Resource Manager/Accountant that was fine-tuned in Kubo 0.18.

Adding default hard-limits from the Resource Manager/Accountant after the fact is tricky, and some additional improvements have been made to improve the computed defaults.

As much as possible, the aim is for a user to only think about how much memory they want to bound libp2p to, and not need to think about translating that to hard numbers for connections, streams, etc. More updates are likely in future Kubo releases, but with this release:

  1. System.StreamsInbound is no longer bounded directly
  2. System.ConnsInbound, Transient.Memory, Transiet.ConnsInbound have higher default computed values.

📝 Changelog

Full Changelog
  • github.com/ipfs/kubo:
    • Add overview section
    • Adjust inbound connection limits depending on memory.
    • feat: Pubsub.SeenMessagesStrategy (#9543) (ipfs/kubo#9543)
    • chore: update version
  • github.com/libp2p/go-libp2p-pubsub (v0.8.2 -> v0.8.3):

👨‍👩‍👧‍👦 Contributors

Contributor Commits Lines ± Files Changed
Mohsin Zaidi 2 +511/-55 12
Antonio Navarro Perez 2 +57/-57 5
galargh 1 +1/-1 1

v0.18.0

Overview

Below is an outline of all that is in this release, so you get a sense of all that's included.

🔦 Highlights

Content routing

Default InterPlanetary Network Indexer

Content routing is the process of discovering which peers provide a piece of content. Kubo has traditionally only supported libp2p's implementation of Kademlia DHT for content routing.

Kubo can now bridge networks by including support for the delegated routing HTTP API. Users can compose content routers using the Routing.Routers config to pick content routers with different tradeoffs than a Kademlia DHT (e.g., high-performance and high-capacity centralized endpoints, dedicated Kademlia DHT nodes, routers with unique provider records, privacy-focused content routers).

One example is InterPlanetary Network Indexers, which are HTTP endpoints that cache records from both the IPFS network and other sources such as web3.storage and Filecoin. This improves not only content availability by enabling Kubo to transparently fetch content directly from Filecoin storage providers, but also improves IPFS content routing latency by an order of magnitude and decreases resource consumption.

Note: it's possible to retrieve content stored by Filecoin Storage Providers (SPs) from Kubo if the SPs service Bitswap requests. As of this release, some SPs are advertising Bitswap. You can follow the roadmap progress for IPNIs and Bitswap in SPs here.

In this release, the default content router is changed from dht to auto. The auto router includes the IPFS DHT in addition to the cid.contact IPNI instance. In future releases, we plan to expand the functionality of auto to encompass automatic discovery of content routers, which will improve performance and content availability (for example, see IPIP-342).

Previous behavior can be restored by setting Routing.Type to dht.

Alternative routing rules, including alternative IPNI endpoints, can be configured in Routing.Routers after setting Routing.Type to custom.

Learn more in the Routing docs.

Increase provider record republish interval and expiration

Default Reprovider.Interval changed from 12h to 22h to match new defaults for the Provider Record Expiration (48h) in go-libp2p-kad-dht v0.20.0.

The rationale for increasing this can be found in RFM 17: Provider Record Liveness Report, kubo#9326, and the upstream DHT specifications at libp2p/specs#451.

Learn more in the Reprovider config.

Gateways

(DAG-)JSON and (DAG-)CBOR response formats

The IPFS project has reserved the corresponding media types at IANA:

This release implements them as part of IPIP-328 and adds Gateway support for CIDs with json (0x0200), cbor (0x51), dag-json (0x0129) and dag-cbor (0x71) codecs.

To specify the response Content-Type explicitly, the HTTP client can override the codec present in the CID by using the format parameter or setting the Accept HTTP header:

  • Plain JSON: ?format=json or Accept: application/json
  • Plain CBOR: ?format=cbor or Accept: application/cbor
  • DAG-JSON: ?format=dag-json or Accept: application/vnd.ipld.dag-json
  • DAG-CBOR: ?format=dag-cbor or Accept: application/vnd.ipld.dag-cbor

In addition, when DAG-JSON or DAG-CBOR is requested with the Accept header set to text/html, the Gateway will return a basic HTML page with download options, improving the user experience in web browsers.

Example 1: DAG-CBOR and DAG-JSON Conversion on Gateway

The Gateway supports conversion between DAG-CBOR and DAG-JSON for efficient end-to-end data structure management: author in CBOR or JSON, store as binary CBOR and retrieve as JSON via HTTP:

$ echo '{"test": "json"}' | ipfs dag put # implicit --input-codec dag-json --store-codec dag-cbor
bafyreico7mjtqtqhvawro3yud5uqn6sc33nzqb7b5j2d7pdmzer5nab4t4

$ ipfs block get bafyreico7mjtqtqhvawro3yud5uqn6sc33nzqb7b5j2d7pdmzer5nab4t4 | xxd
00000000: a164 7465 7374 646a 736f 6e              .dtestdjson

$ ipfs dag get bafyreico7mjtqtqhvawro3yud5uqn6sc33nzqb7b5j2d7pdmzer5nab4t4 # implicit --output-codec dag-json
{"test":"json"}

$ curl "http://127.0.0.1:8080/ipfs/bafyreico7mjtqtqhvawro3yud5uqn6sc33nzqb7b5j2d7pdmzer5nab4t4?format=dag-json"
{"test":"json"}
Example 2: Traversing CBOR DAGs

Placing a CID in CBOR Tag 42 enables the creation of arbitrary DAGs. The equivalent DAG-JSON notation for linking to different blocks is represented by { "/": "cid" }.

The Gateway supports traversing these links, enabling access to data referenced by structures other than regular UnixFS directories:

$ echo '{"test.jpg": {"/": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"}}' | ipfs dag put
bafyreihspwy3zlkzgphmec5d3xb5g5njrqwotd46lyubnelbzktnmsxkq4 # dag-cbor document linking to unixfs file

$ ipfs resolve /ipfs/bafyreihspwy3zlkzgphmec5d3xb5g5njrqwotd46lyubnelbzktnmsxkq4/test.jpg
/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

$ ipfs dag stat bafyreihspwy3zlkzgphmec5d3xb5g5njrqwotd46lyubnelbzktnmsxkq4
Size: 119827, NumBlocks: 2

$ curl "http://127.0.0.1:8080/ipfs/bafyreihspwy3zlkzgphmec5d3xb5g5njrqwotd46lyubnelbzktnmsxkq4/test.jpg" > test.jpg
Example 3: UnixFS directory listing as JSON

Finally, Gateway now supports the same logical format projection from DAG-PB to DAG-JSON as the ipfs dag get command, enabling the retrieval of directory listings as JSON instead of HTML:

$ export DIR_CID=bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq
$ curl -H "Accept: application/vnd.ipld.dag-json" "http://127.0.0.1:8080/ipfs/$DIR_CID" | jq
$ curl "http://127.0.0.1:8080/ipfs/$DIR_CID?format=dag-json" | jq
{
  "Data": {
    "/": {
      "bytes": "CAE"
    }
  },
  "Links": [
    {
      "Hash": {
        "/": "Qmc3zqKcwzbbvw3MQm3hXdg8BQoFjGdZiGdAfXAyAGGdLi"
      },
      "Name": "1 - Barrel - Part 1 - alt.txt",
      "Tsize": 21
    },
    {
      "Hash": {
        "/": "QmdMxMx29KVYhHnaCc1icWYxQqXwUNCae6t1wS2NqruiHd"
      },
      "Name": "1 - Barrel - Part 1 - transcript.txt",
      "Tsize": 195
    },
    {
      "Hash": {
        "/": "QmawceGscqN4o8Y8Fv26UUmB454kn2bnkXV5tEQYc4jBd6"
      },
      "Name": "1 - Barrel - Part 1.png",
      "Tsize": 24862
    }
  ]
}
$ ipfs dag get $DIR_CID
{"Data":{"/":{"bytes":"CAE"}},"Links":[{"Hash":{"/":"Qmc3zqKcwzbbvw3MQm3hXdg8BQoFjGdZiGdAfXAyAGGdLi"},"Name":"1 - Barrel - Part 1 - alt.txt","Tsize":21},{"Hash":{"/":"QmdMxMx29KVYhHnaCc1icWYxQqXwUNCae6t1wS2NqruiHd"},"Name":"1 - Barrel - Part 1 - transcript.txt","Tsize":195},{"Hash":{"/":"QmawceGscqN4o8Y8Fv26UUmB454kn2bnkXV5tEQYc4jBd6"},"Name":"1 - Barrel - Part 1.png","Tsize":24862}]}
🐎 Fast directory listings with DAG sizes

Fast listings are now enabled for all UnixFS directories: big and small. There is no linear slowdown caused by reading size metadata from child nodes, and the size of DAG representing child items is always present.

As an example, the CID bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm represents a UnixFS directory with over 10k files. Listing big directories was fast since Kubo 0.13, but in this release it will also include the size column.

QUIC and WebTransport

WebTransport enabled by default

WebTransport is a new libp2p transport that was introduced in v0.16 that is based on top of QUIC and HTTP3.

This allows browser-based nodes to contact Kubo nodes, so now instead of just serving requests for other system-level application nodes, you can also serve requests directly to a node running inside a browser page.

For the full story see connectivity.libp2p.io.

QUIC and WebTransport share a single port

WebTransport is enabled by default in part because go-libp2p now supports running WebTransport and QUIC transports on the same QUIC listener. No additional port needs to be opened.

To use this feature, register two listen addresses on the same /ipX/.../udp/XXX prefix.

Differentiating QUIC versions

go-libp2p now differentiates the first version of QUIC that was originally implemented, Draft-29, from the ratified protocol in RFC9000, QUICv1. This was done for performance (time to first byte) reasons as outlined here.

This manifests as two different multiaddr components /quic (old Draft-29) and /quic-v1. go-libp2p do supports listening with both QUIC versions on one single listener. WebTransport has only supported QUICv1. /webtransport now needs to be prefixed by a /quic-v1 component instead of a /quic component.

Support for QUIC Draft-29 will be removed at some point in 2023 (tracking issue). As a result, new deployments should use /quic-v1 instead of /quic.

QUICv1 and WebTransport config migration

To support QUICv1 and WebTransport by default a new config migration (v13) is run which automatically adds entries in addresses-related fields:

  • Replace all /quic/webtransport to /quic-v1/webtransport.
  • For all /quic listeners, keep the Draft-29 listener, and on the same ip and port, add /quic-v1 and /quic-v1/webtransport listeners.

Improving libp2p resource management integration

To help protect nodes from DoS (resource exhaustion) and eclipse attacks, Kubo enabled the go-libp2p Network Resource Manager by default in Kubo 0.17.

Introducing limits like this by default after the fact is tricky, and various improvements have been made to improve the UX including:

  1. Dedicated docs concerning the resource manager integration. This is a great place to go to learn more or get your FAQs answered.
  2. Increasing the default limits for the resource manager.
  3. Enabling the Swarm.ConnMgr by default and reducing it thresholds so it can intelligently prune connections in many cases before the indiscriminate resource manager kicks in.
  4. Adjusted log messages and levels to make clear that the resource manager is likely doing your node a favor by bounding resources.
  5. Other miscellaneous config and command bugs reported by users.

📝 Changelog

Full Changelog

👨‍👩‍👧‍👦 Contributors

Contributor Commits Lines ± Files Changed
Marten Seemann 154 +8826/-6369 911
Gus Eggert 7 +2792/-1444 40
Marco Munizaga 26 +2324/-752 101
hannahhoward 7 +695/-1587 50
Rod Vagg 30 +1508/-668 106
Henrique Dias 13 +1321/-431 85
Yahya Hassanzadeh 4 +984/-158 9
galargh 17 +519/-520 20
Steve Loeppky 11 +612/-418 25
Antonio Navarro Perez 30 +742/-88 47
Marcin Rataj 19 +377/-407 52
Ian Davis 2 +419/-307 7
whyrusleeping 5 +670/-28 17
Piotr Galar 8 +211/-417 25
web3-bot 28 +282/-264 75
Will Scott 10 +428/-103 19
julian88110 2 +367/-55 27
Will 5 +282/-131 65
Jorropo 25 +263/-94 38
Wondertan 10 +203/-87 24
Mohsin Zaidi 1 +269/-0 4
Dennis Trautwein 3 +230/-21 7
Prithvi Shahi 1 +116/-77 1
Masih H. Derkani 5 +130/-37 11
Iulian Pascalau 1 +151/-16 2
Scott Martin 1 +166/-0 3
Daniel Vernall 1 +92/-45 2
Steven Allen 7 +114/-15 11
Hlib Kanunnikov 4 +100/-28 6
Peter Rabbitson 4 +59/-65 5
Lucas Molas 1 +60/-57 7
nisdas 3 +107/-6 5
why 2 +80/-20 5
ShengTao 2 +46/-45 16
nisainan 2 +40/-50 12
Mikel Cortes 3 +44/-36 10
Chinmay Kousik 1 +64/-14 6
ZenGround0 2 +62/-15 6
Antonio Navarro 3 +58/-3 8
Michael Muré 2 +49/-2 2
Dirk McCormick 1 +3/-42 1
kixelated 1 +20/-20 4
Russell Dempsey 1 +19/-17 3
Karthik Nallabolu 1 +17/-17 1
protolambda 1 +26/-4 4
cliffc-spirent 1 +25/-5 2
Raúl Kripalani 1 +29/-0 1
Håvard Anda Estensen 1 +9/-19 6
vyzo 1 +11/-12 1
anorth 1 +15/-8 3
shade34321 1 +21/-1 2
Toby 2 +9/-13 6
Nishant Das 1 +9/-9 5
Jeromy Johnson 1 +17/-0 3
Oleg 1 +14/-1 1
Hector Sanjuan 6 +4/-11 6
Łukasz Magiera 2 +10/-4 2
Aayush Rajasekaran 1 +7/-7 1
Adin Schmahmann 1 +4/-3 1
Stojan Dimitrovski 1 +6/-0 1
Mathew Jacob 1 +3/-3 1
Vladimir Ivanov 1 +2/-2 1
Nex Zhu 1 +4/-0 2
Michele Mastrogiovanni 1 +2/-2 1
Louis Thibault 1 +4/-0 1
Eric Myhre 1 +3/-1 1
Kubo Mage 2 +2/-1 2
tabcat 1 +1/-1 1
Viacheslav 1 +1/-1 1
Max Inden 1 +1/-1 1
Manic Security 1 +1/-1 1
Jc0803kevin 1 +1/-1 1
David Brouwer 1 +2/-0 2
Rong Zhou 1 +1/-0 1
Neel Virdy 1 +1/-0 1