Skip to content

Release 0.4.18
Compare
Choose a tag to compare
@Stebalien Stebalien released this 02 Nov 03:41
· 5728 commits to master since this release
v0.4.18
This tag was signed with the committer’s verified signature.
Stebalien Steven Allen
GPG key ID: C125ECFD1713943C
Learn about vigilant mode.
aefc746

This is probably one of the largest go-ipfs releases in recent history, 3 months
in the making.

Features

The headline features this release are experimental QUIC support, the gossipsub
pubsub routing algorithm, pubsub message signing, and a refactored ipfs p2p
command. However, that's just scratching the surface.

QUIC

First up, on the networking front, this release has also introduced experimental
support for the QUIC protocol. QUIC is a new UDP-based network transport that
solves many of the long standing issues with TCP.

For us, this means (eventually):

  • Fewer local resources. TCP requires a file-descriptor per connection while
    QUIC (and most UDP based transports) can share a single file descriptor
    between all connections. This should allow us to dial faster and keep more
    connections open.
  • Faster connection establishment. When client authentication is included,
    QUIC has a three-way handshake like TCP. However, unlike TCP, this handshake
    brings us from all the way from 0 to a fully encrypted, authenticated, and
    multiplexed connection. In theory (not yet in practice), this should
    significantly reduce the latency of DHT queries.
  • Behaves better on lossy networks. When multiplexing multiple requests over
    a single TCP connection, a single dropped packet will bring the entire
    connection to a halt while the packet is re-transmitted. However, because QUIC
    handles multiplexing internally, dropping a single packets affects only the
    related stream.
  • Better NAT traversal. TL;DR: NAT hole-punching is significantly easier
    and, in many cases, more reliable with UDP than with TCP.

However, we still have a long way to go. While we encourage users to test this,
the IETF QUIC protocol is still being actively developed and will change. You
can find instructions for enabling it
here.

Pubsub

In terms of pubsub, go-ipfs now supports the gossipsub routing algorithm and
message signing.

The gossipsub routing algorithm is significantly more efficient than the
current floodsub routing algorithm. Even better, it's fully backwards compatible
so you can enable it and still talk to nodes using the floodsub algorithm. You
can find instructions to enable gossipsub in go-ipfs
here.

Messages are now signed by their authors. While signing has now been enabled by
default, strict signature verification has not been and will not be for at least
one release (probably multiple) to avoid breaking existing applications. You can
read about how to configure this feature
here.

Commands

In terms of new toys, this release introduces a new ipfs cid subcommand for
working with CIDs, a completely refactored ipfs p2p command, streaming name
resolution, and complete inline block support.

The new ipfs cid command allows users to both inspect CIDs and convert them
between various formats and versions. For example:

# Print out the CID metadata (prefix)
> ipfs cid format -f %P QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
cidv0-protobuf-sha2-256-32

# Get the hex sha256 hash from the CID.
> ipfs cid format -b base16 -f '0x%D' QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
0x46d44814b9c5af141c3aaab7c05dc5e844ead5f91f12858b021eba45768b4c0e

# Convert a base58 v0 CID to a base32 v1 CID.
> ipfs cid base32 QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby

The refactored ipfs p2p command allows forwarding TCP streams through two IPFS
nodes from one host to another. It's ssh -L but for IPFS. You can find
documentation
here.
It's still experimental but we don't expect too many breaking changes at this
point (it will very likely be stabilized in the next release). Quick summary of
breaking changes:

  • We don't stop listening for local (forwarded) connections after accepting a
    single connection.
  • ipfs p2p stream ls output now returns more useful output, first address is
    always the initiator address.
  • ipfs p2p listener ls is renamed to ipfs p2p ls
  • ipfs p2p listener close is renamed to ipfs p2p close
  • Protocol names have to be prefixed with /x/ and are now just passed to
    libp2p as handler name. Previous version did this 'under the hood' and with
    /p2p/ prefix. There is a --allow-custom-protocol flag which allows you
    to use any libp2p handler name.
  • ipfs p2p listener open and ipfs p2p stream dial got renamed:
    • ipfs p2p listener open p2p-test /ip4/127.0.0.1/tcp/10101
      new becomes ipfs p2p listen /x/p2p-test /ip4/127.0.0.1/tcp/10101
    • ipfs p2p stream dial $NODE_A_PEERID p2p-test /ip4/127.0.0.1/tcp/10102
      is now ipfs p2p forward /x/p2p-test /ip4/127.0.0.1/tcp/10102 /ipfs/$NODE_A_PEERID

There is now a new flag for ipfs name resolve - --stream. When the command
is invoked with the flag set, it will start returning results as soon as they
are discovered in the DHT and other routing mechanisms. This enables certain
applications to start prefetching/displaying data while the discovery is still
running. Note that this command will likely return many outdated records
before it finding and returning the latest. However, it will always return
valid records (even if a bit stale).

Finally, in the previous release, we added support for extracting blocks inlined
into CIDs. In this release, we've added support for creating these CIDs. You can
now run ipfs add with the --inline flag to inline blocks less than or equal
to 32 bytes in length into a CID, instead of writing an actual block. This
should significantly reduce the size of filesystem trees with many empty
directories and tiny files.

IPNS

You can now publish and resolve paths with namespaces other than /ipns and
/ipfs through IPNS. Critically, IPNS can now be used with IPLD paths (paths
starting with /ipld).

WebUI

Finally, this release includes the shiny updated
webui. You can view it by
installing go-ipfs and visiting http://localhost:5001/webui.

Performance

This release includes some significant performance improvements, both in terms
of resource utilization and speed. This section will go into some technical
details so feel free to skip it if you're just looking for shiny new features.

Resource Utilization

In this release, we've (a) fixed a slow memory leak in libp2p and (b)
significantly reduced the allocation load. Together, these should improve both
memory and CPU usage.

Datastructures

We've changed two of our most frequently used datastructures, CIDs and
Multiaddrs, to reduce allocation load.

First, we now store CIDs encode as strings, instead of decoded in structs
(behind pointers). In addition to being more compact, our Cid type is now a
valid map key so we no longer have to encode CIDs every time we want to use
them in a map/set. Allocations when inserting CIDs into maps/sets was showing up
as a significant source of allocations under heavy load so this change should
improve memory usage.

Second, we've changed many of our multiaddr parsing/processing/formatting
functions to allocate less. Much of our DHT related-work includes processing
multiaddrs so this should reduce CPU utilization when heavily using the DHT.

Streams and Yamux

Streams have always plagued us in terms of memory utilization. This was
partially solved by introducing the connection manager, keeping our maximum
connection count to a reasonable number but they're still a major memory sink.

This release sees two improvements on this front:

  1. A memory leak in identify
    has been fixed. This was slowly causing us to leak connections (locking up
    the memory used by the connections' streams).
  2. Yamux streams now use a buffer-pool backed, auto shrinking read buffer.
    Before, this read buffer would grow to its maximum size (a few megabytes) and
    never shrink but these buffers now shrink as they're emptied.

Bitswap Performance

Bitswap will now pack multiple small blocks into a single message thanks
ipfs/go-bitswap#5. While this won't
help when transferring large files (with large blocks), this should help when
transferring many tiny files.

Refactors and Endeavors

This release saw yet another commands-library refactor, work towards the
CoreAPI, and the first step towards reliable base32 CID support.

Commands Lib

We've completely refactored our commands library (again). While it still needs
quite a bit of work, it now requires significantly less boilerplate and should
be significantly more robust. The refactor immediately found two broken tests
and probably fixed quite a few bugs around properly returning and handling
errors.

CoreAPI

CoreAPI is a new way to interact with IPFS from Go. While it's still not
final, most things you can do via the CLI or HTTP interfaces, can now be done
through the new API.

Currently there is only one implementation, backed by go-ipfs node, and there are
plans to start http-api backed one soon. We are also looking into creating RPC
interface using this API, which could help performance in some use cases.

You can track progress in #4498

IPLD paths

We introduced new path type which introduces distinction between IPLD and
IPFS (unixfs) paths. From now on paths prefixed with /ipld/ will always
use IPLD link traversal and /ipfs/ will use unixfs path resolver, which
takes things like shardnig into account.

Note that this is only initial support and there likely are some bugs in
how the paths are handled internally, so consider this feature
experimental for now.

CIDv1/Base32 Migration

Currently, IPFS is usually used in browsers by browsing to
https://SOME_GATEWAY/ipfs/CID/.... There are two significant drawbacks to this
approach:

  1. From a browser security standpoint, all IPFS "sites" will live under the same
    origin (SOME_GATEWAY).
  2. From a UX standpoint, this doesn't feel very "native" (even if the gateway is
    a local IPFS node).

To fix the security issue, we intend to switch IPFS gateway links
https://ipfs.io/ipfs/CID to to https://CID.ipfs.dweb.link. This way, the CID
will be a part of the
"origin" so
each IPFS website will get a separate security origin.

To fix the UX issue, we've been working on adding support for ipfs://CID/...
to web browsers through our
ipfs-companion add-on and some new,
experimental extension APIs from Mozilla. This has the same effect of putting
the CID in the URL origin but has the added benefit of looking "native".

Unfortunately, origins must be case insensitive. Currently, most CIDs users
see are CIDv0 CIDs (those starting with Qm) which are always base58
encoded and are therefore case-sensitive.

Fortunately, CIDv1 (the latest CID format) supports arbitrary bases using the
multibase standard. Unfortunately,
IPFS has always treated equivalent CIDv0 and CIDv1 CIDs as distinct. This means
that files added with CIDv0 CIDs (the default) can't be looked up using the
equivalent CIDv1.

This release makes some significant progress towards solving this issue by
introducing two features:

(1) The previous mentioned ipfs cid base32 command for converting CID to a
case intensive encoding required by domain names. This command converts a CID to
version 1 and encodes it using base32.

(2) A hack to allow locally looking up blocks associated with a CIDv0 CID using
the equivalent CIDv1 CID (or the reverse). This hack will eventually
be replaced with a multihash indexed blockstore, which is agnostic to both the
CID version and multicodec content type.

go-ipfs changelog

Features (i.e., users take heed):

Fixes (i.e., users take note):

Regressions Fixes (fixes for bugs introduced since the last release):

Extractions:

Documentation:

Testing:

Internal:

Cleanup:

related changelogs

Changes to sub-packages go-ipfs depends on. This does not include libp2p or multiformats.

github.com/ipfs/go-log

github.com/ipfs/go-datastore

github.com/ipfs/go-cid

github.com/ipfs/go-ds-flatfs

github.com/ipfs/go-ds-measure

github.com/ipfs/go-ds-leveldb

github.com/ipfs/go-metrics-prometheus

github.com/ipfs/go-metrics-interface

github.com/ipfs/go-ipld-format

github.com/ipfs/go-ipld-cbor

github.com/ipfs/go-ipfs-cmds

github.com/ipfs/go-ipfs-cmdkit

github.com/ipfs/go-ds-badger

github.com/ipfs/go-ipfs-addr

github.com/ipfs/go-ipfs-chunker

github.com/ipfs/go-ipfs-routing

github.com/ipfs/go-ipfs-blockstore

github.com/ipfs/go-ipns

github.com/ipfs/go-bitswap

github.com/ipfs/go-cidutil

github.com/ipfs/go-ipfs-config

github.com/ipfs/go-merkledag

github.com/ipfs/go-mfs

github.com/ipfs/go-path

  • fix: don't dag.Get in ResolveToLastNode when not needed (ipfs/go-path#1)

github.com/ipfs/go-unixfs

Assets 38
Loading