go-libp2p v0.18.0

[marten-seemann]

🗺 What's left for release

Resource Manager

• Network Resource Manager interface go-libp2p-core#229
• use the ResourceManager go-tcp-transport#110
• use the Resource Manager go-libp2p-transport-upgrader#99
• use the Resource Manager go-libp2p-quic-transport#249
• add a MemoryManager go-yamux#69
• pass the PeerScope to yamux (satifiying its MemoryManger interface) go-libp2p-yamux#46
• implement the new network.MuxedConn interface go-libp2p-mplex#29
• Network Resource Manager Implementation go-libp2p-resource-manager#1
• implement the new Multiplexer.NewConn interface go-stream-muxer-multistream#22
• add support for the resource manager go-libp2p-swarm#308
• use the resource manager #1275
• Basic resource manager integration tests #1296
• Salvage mplex in the age of resource management go-mplex#99
• Additional go-mplex fixes
• Tests passing for feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) ipfs/kubo#8680

Housekeeping

transport.Upgrader interface

• introduce a transport.Upgrader interface go-libp2p-core#232
• use the transport.Upgrader interface go-tcp-transport#111
• use the transport.Upgrader interface go-ws-transport#110
• use the new transport.Upgrader interface go-libp2p-transport-upgrader#100
• use the transport.Upgrader interface go-libp2p-swarm#309
• use the transport.Upgrader interface go-libp2p-circuit#149
• use the transport.Upgrader interface #1277

netutil deprecation

• deprecate deprecate this repo go-libp2p-netutil#60
• use the netutil package that was moved to go-libp2p-testing #1263

go-addr-util deprecation

• remove dependency on go-addr-util go-libp2p-swarm#300
• move ResolveUnspecifiedAddress(es) and FilterAddrs here from libp2p/go-addr-util multiformats/go-multiaddr#168
• remove direct dependency on libp2p/go-addr-util #1279
• deprecate this repo go-addr-util#44

autonat consolidation

• move go-libp2p-autonat to p2p/host/autonat #1273
• deprecate this repo go-libp2p-autonat#124

go-sockaddr deprecation

• move some functions from go-sockaddr here, remove go-sockaddr dependency go-netroute#22
• deprecate this repo go-sockaddr#25

discovery consolidation

• move go-libp2p-discovery here #1291

connmgr consolidation

• move the go-libp2p-connmgr here #1297

update all repos to go-log v2

• chore: update go-log to v2 go-libp2p-quic-transport#242
• chore: update go-log to v2 go-libp2p-discovery#76

🛠 BREAKING CHANGES

<any breaking changes with this release, including changing of defaults >

🔦 Highlights

< top highlights for this release notes >

Changelog

< changelog generated by scripts/mkreleaselog >

✅ Release Checklist

• Stage 0 - Finishing Touches
  • Go through relevant libp2p repos looking for unreleased changes that should make it into the release. If you find any, cut releases.
  • Run go get -u ./... to see if there are any out-of-date deps that look important. If there are, bubble them. Try to avoid directly updating indirect deps in go-libp2p's go.mod when possible.
  • Make sure local tests are passing.
• Stage 1 - Upstream Testing
  • Create testing branches in lotus & go-ipfs with the new go-libp2p release and run CI/tests. Many upstream projects are tested in CI, but lotus & go-ipfs are not.
    • filecoin-project/lotus
    • ipfs/go-ipfs
  • (someday) Run upstream testground tests. Unfortunately, this is too time consuming at the moment.
    • (someday) Run bitswap testground tests.
    • (someday) Run DHT testground tests.
• Stage 2 - Infrastructure Testing
  • How: Using the testing branches created above, work with the infrastructure team to deploy the new libp2p versions to our infrastructure.
  • Where:
    • A go-ipfs gateway.
      • Deploy
      • Analyze
        • Look at pprof profile dumps, especially CPU profiles and heap allocation profiles, noting any significant changes.
    • A go-ipfs bootstrapper.
      • Deploy
      • Analyze
        • Look at pprof profile dumps, especially CPU profiles and heap allocation profiles, noting any significant changes.
        • Check peers (e.g., ipfs swarm peers) to make sure we're connecting to peers on all transports.
        • Check advertised addresses and protocols (e.g., ipfs id) to make sure they're sane.
• Stage 3 - Release
  • Tag the release on master.
  • Publish the release through the GitHub UI, adding the release notes. Some users rely on this to receive notifications of new releases.
  • Announce the release on the discuss.libp2p.io.
• Stage 4 - Update Upstream
  • Update the upstream testing branches to the final release and create PRs.
    • filecoin-project/lotus
    • ipfs/go-bitswap
    • ipfs/go-ipfs
    • libp2p/go-libp2p-kad-dht
    • libp2p/go-libp2p-pubsub
    • libp2p/go-libp2p-daemon
  • deprecate go-libp2p-discovery
  • deprecate go-libp2p-connmgr
• Make required changes to the release process.

[BigLep]

2022-03-01 update on where the release is at:

1. Additional fixes/improvements for mplex need to be made for it to work better with resource management: Mplex salvage operations, part II go-mplex#102

• ECD 2022-03-02

2. (Stage 1 above) go-libp2p v0.18 integration into go-ipfs needs to have the tests pass. This is happening in feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) ipfs/kubo#8680

• ECD 2022-03-02

3. (Stage 2 above) A go-ipfs build with go-lip2p v0.18 integration needs to sit on the ipfs.io gateway and bake for 1 business day to shake out any issues with the ResourceManager in production.

• Deployment 2022-03-03
• Finish baking 2022-03-04

4. Fix any issues that result from baking in the Gateway.

• ECD depends on if/what issues are found

5. (Stage 3 and 4 above) Do the release

• Earliest ECD is Friday, 2022-03-11 because of maintainer travel.

Notes:

1. ECD = expected completion date
2. All of these items above are in the issue description checklist.

@marten-seemann : please correct if this is wrong/off at all.

[arajasek]

@BigLep Thanks a lot for the detailed timeline! Lotus is gonna drop the update from its February release (1.15.0), but we're keen to get it into our March release -- code freeze scheduled for March 8, but we can wait until the week after to gobble up the release if it's out by then :)

[BigLep]

@marten-seemann : totally fine if new things have been learned and timeline needs to adjust, but I assume we aren't deploying to the ipfs.io gateway given no new pushes to ipfs/kubo#8680.

[marten-seemann]

(Stage 1 above) go-libp2p v0.18 integration into go-ipfs needs to have the tests pass. This is happening in ipfs/kubo#8680
ECD 2022-03-02

The tests are passing, but we're still not able to set limits, see ipfs/kubo#8680 (comment) and ipfs/kubo#8680 (comment). The deployment on the gateways is blocked until this is possible.

[BigLep]

Thanks @marten-seemann . It looks like we're getting traction on those go-ipfs issues.

While it doesn't block the go-libp2p release, for my own bookkeeping, I'm writing down that we'll need to confirm we're providing the proper information so Lotus can debug issues that are manifesting because of new go-libp2p release with resource management (e.g., filecoin-project/lotus#8129). Obviously issues here don't mean it's the resource manager, but could be an issue in Lotus code or that a higher limit should be set. (I don't have the context yet - am waiting to get Slack access. Again, this is just so I have all threads related to libp2p resource management in one place.)

[jennijuju]

@marten-seemann @BigLep Hi team - may i get an updated ETA on the final release, please? The lotus march release is scheduled to be code freeze tmr and im trying to figure out if we need to revert the v1.18.0-rcX from the master to avoid it blocks the code freeze.

[BigLep]

@jennijuju : I'll work on updating the status now, but the release isn't going to be complete by EOD today (2022-03-14). As a result, I think Lotus needs to back out go-libp2p-rcX again. We'll let Lotus decide what that means for them. More info to come shortly.

[vyzo]

or it can wait a day or two, this wont be the first time an rc has a delay.

[BigLep]

2022-03-14 update on where the release is at:

1. (Stage 1 above) go-libp2p v0.18 integration into go-ipfs needs to have the tests pass.

• DONE - @BigLep believes we effectively have this in feat: export rcmgr metrics to prometheus ipfs/kubo#8785

2. (Stage 2 above) A go-ipfs build with go-lip2p v0.18 integration needs to sit on the ipfs.io gateway and bake for 1 business day to shake out any issues with the ResourceManager in production.

• Deployment 2022-03-03 2022-03-14: https://github.com/protocol/bifrost-infra/issues/1722
• Finish baking 2022-03-04 2022-03-15

3. Review results from Gateway baking

• This may involve needing to create some new Prometheus dashboards
• Dashboard link will be posted here
• Chat/discussion on this will happen here (in IPFS Discord #libp2p-dev): https://discord.com/channels/806902334369824788/952970972103843891

5. Fix any issues that result from baking in the Gateway.

• ECD depends on if/what issues are found

6. (Stage 3 and 4 above) Do the release

• Earliest ECD is Wednesday, 2022-03-11 2022-03-17

I misstepped to suggest what Lotus should do here. Above is the timeline, and I'll let Lotus decide accordingly. libp2p will stay focused on creating a vetted release and communicating the status so consumers can plan accordingly.

[jennijuju]

@BigLep - thank you for informing us!
Jfyi on lotus side:
@vyzo kindly offered to work on a pr to lotus that will check the connmgr limit when initializing the rcmgr, and if they are high, auto adjust the rcmgr limits. We are comfortable saying with PR, we are fine with go ahead and feature code freeze the v1.15.1-rc1 as scheduled tmr, then update to the final v1.18.0 release as it’s published later this week in a later rc.

thank you @vyzo for your timely support!

Cc @arajasek

[arajasek]

@jennijuju Thanks for staying on top of this! The plan sounds good to me, I only have one open question: Are we comfortable shipping with the libp2p RC if it came to it, or would we drop it out of the last RC if needed?

I think it's fine that the Lotus 1.15.1-rc will be using a libp2p RC, but am unsure about whether we want to ship the final release with an RC dependency. How large is the changeset for this release (per https://github.com/libp2p/go-libp2p/releases/tag/v0.18.0-rc1 sounds like it's mostly the resource manager)?

Of course the hope (and expectation) is that the final libp2p release will be ready by then, but I'm wondering what the contingency plan is.

[marten-seemann]

v0.18.0 just shipped!