Release 0.21

[BigLep]

Meta

• Release owner: @hacdias
• Release engineer: @hacdias
• Release reviewer: @Jorropo
• Expected RC date: 2023-06-01 2023-06-08 2023-06-12
• 🚢 Expected final release date: 2023-06-08 2023-06-15 2023-07-04
• Changelog: https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.21.md
• Accompanying PR for improving the release process: (example: docs: release process updates from v0.17.0 #9391)
• Accompanying Issue for updating the infra: (example: https://github.com/protocol/bifrost-infra/issues/2221)

Items in scope

Required for RC1

• Accelerated DHT improvements: Move AcceleratedDHTClient from being experimental #9703 - This is an operational improvement item that has been open for multiple months and needs to be closed out for operational hygeine.
• Routing table resilience items: fix: don't add unresponsive DHT servers to the Routing Table libp2p/go-libp2p-kad-dht#820 - This is an operational improvement item that has been open for multiple months and needs to be closed out for operational hygeine.
• Async pinner changes: pin: follow async pinner changes #9859 - Community contribution that has been open for a month plus that publicly said would include.
• go-libp2p update to 0.28 if it's ready (see comment), and if not use 0.27.5
• Boxo release with latest changes Release 0.9.0 boxo#325 - Much of this Kubo release depends on improvements there. In addition, we need to address v0.20.0-rc2: cannot build plugins #9839 since that slipped the last release.
• If secp256k1 key support is missing pem-pkcs8-cleartext import/export #9912 isn't fixed by the time of final release, then revert 67e1a17 and cc original author.
• fix: correct list of addresses for delegated routing #9920 (fix for HTTP PUT provided by IPNI team)
• dag import error handling fixes fix(cmd): useful errors in dag import #9945 moved to 0.22
• dag import not pinning by default + regression tests feat!: dag import - don't pin roots by default #9926 moved to 0.22

Required for RC2

• feat!: dag import - don't pin roots by default #9926
• fix(cmd): useful errors in dag import #9945
• fix: fix abba bug in UsefullNewPeer libp2p/go-libp2p-kbucket#122
• fix: issues discovered in kubo v0.21.0-rc1 libp2p/go-libp2p-kad-dht#851
• feat: webui@4.0.1 #9940
• go-libp2p update to 0.27.6

Required for RC3

• fix(gateway): allow CAR trustless requests with path boxo#358
• go-libp2p update to 0.27.7 chore: bump go-libp2p-kad-dht for deadlock fix #9980

Others

These are items that we had stated at the beginning of the release that we said we wanted to include. We aren't blocking the relse on these. There are also items in 0.21 iteration.

• HTTP Routing v1 streaming support: routing/http: feat: add streaming support boxo#18
• Prepare to deprecate pubsub APIs: Deprecate then Remove /api/v0/pubsub/* RPC API and ipfs pubsub Commands #9717
• Polishing process items of Boxo
• Landing changes for caching proxy for HTTP Routing v1

✅ Release Checklist

Labels

If an item should be executed for a specific release type, it should be labeled with one of the following labels:

• execute ONLY when releasing a Release Candidate
• execute ONLY when releasing a Final Release

Otherwise, it means it should be executed for ALL release types.

Patch releases should follow the same process as .0 releases. If some item should NOT be executed for a Patch Release, it should be labeled with:

• do NOT execute when releasing a Patch Release

Before the release

This section covers tasks to be done ahead of the release.

• Verify you have access to all the services and tools required for the release
  • GPG signature configured in local git and in GitHub
  • admin access to IPFS Discourse
    • ask the previous release owner (or @2color) for an invite
  • access to #shared-pl-marketing-requests channel in FIL Slack
    • ask the previous release owner for an invite
  • access to IPFS network metrics dashboards in Grafana
    • open an access request in the pldw
    • example
  • kuboreleaser checked out on your system (only if you're using kuboreleaser)
  • Thunderdome checked out on your system and configured (see the Thunderdome release docs for setup)
  • docker installed on your system (only if you're using kuboreleaser)
  • npm installed on your system (only if you're NOT using kuboreleaser)
  • zsh installed on your system
  • kubo checked out under $(go env GOPATH)/src/github.com/ipfs/kubo
    • you can also symlink your clone to the expected location by running mkdir -p $(go env GOPATH)/src/github.com/ipfs && ln -s $(pwd) $(go env GOPATH)/src/github.com/ipfs/kubo
  • Reddit account
• Upgrade Go used in CI to the latest patch release available in CircleCI in:
  • ipfs/distributions
    • example
  • ipfs/ipfs-docs
    • example
• Verify there is nothing left for release
• Create a release process improvement PR
  • update the release issue template as you go
  • link it in the Meta section

The release

This section covers tasks to be done during each release (current: 0.21.0)

• Prepare the release branch and update version numbers accordingly
  using kuboreleaser release --version vX.Y.Z(-rcN) prepare-branch or ...

  • create a new branch release-vX.Y.Z
    • use master as base if Z == 0
    • use release as base if Z > 0
  • update the CurrentVersionNumber in version.go in the master branch to vX.Y+1.0-dev
    • example
  • update the CurrentVersionNumber in version.go in the release-vX.Y branch to vX.Y.Z(-RCN)
    • example
  • create a draft PR from release-vX.Y to release
    • example
  • Cherry-pick commits from master to the release-vX.Y.Z using git cherry-pick -x <commit>
  • Add full changelog and contributors to the changelog
    • Replace the Changelog and Contributors sections of the changelog with the stdout of ./bin/mkreleaselog
      • do NOT copy the stderr
  • verify all CI checks on the PR from release-vX.Y to release are passing
  • Merge the PR from release-vX.Y to release using the Create a merge commit
    • do NOT use Squash and merge nor Rebase and merge because we need to be able to sign the merge commit
    • do NOT delete the release-vX.Y branch
• @Jorropo Run Thunderdome testing, see the Thunderdome release docs for details
  • create a PR and merge the experiment config into Thunderdome
• Create the release tag
  using kuboreleaser release --version vX.Y.Z(-rcN) tag or ...

  • This is a dangerous operation! Go and Docker publishing are difficult to reverse! Have the release reviewer verify all the commands marked with ⚠️!
  • ⚠️ tag the HEAD commit using git tag -s vX.Y.Z(-RCN) -m 'Prerelease X.Y.Z(-RCN)'
  • ⚠️ tag the HEAD commit of the release branch using git tag -s vX.Y.Z(-RCN) -m 'Release X.Y.Z(-RCN)'
  • ⚠️ verify the tag is signed and tied to the correct commit using git show vX.Y.Z(-RCN)
  • ⚠️ push the tag to GitHub using git push origin vX.Y.Z(-RCN)
    • do NOT use git push --tags because it pushes all your local tags
• Publish the release to DockerHub
  using kuboreleaser --skip-check-before --skip-run release --version vX.Y.Z(-rcN) publish-to-dockerhub or ...

  • Wait for Publish docker image workflow run initiated by the tag push to finish
  • verify the image is available on Docker Hub
• Publish the release to ipfs.tech
  using kuboreleaser release --version vX.Y.Z(-rcN) publish-to-distributions or ...

  • check out ipfs/distributions
  • run ./dist.sh add-version kubo vX.Y.Z(-RCN) to add the new version to the versions file
    • usage
  • create and merge the PR which updates dists/kubo/versions and dists/go-ipfs/versions ( and dists/kubo/current_version and dists/go-ipfs/current_version)
    • example
  • wait for the CI workflow run initiated by the merge to master to finish
  • verify the release is available on dist.ipfs.io
• Publish the release to NPM
  using kuboreleaser release --version vX.Y.Z(-rcN) publish-to-npm or ...

  • run the Release to npm workflow
  • check Release to npm workflow run logs to verify it discovered the new release
  • verify the release is available on NPM
• Publish the release to GitHub
  using kuboreleaser release --version vX.Y.Z(-rcN) publish-to-github or ...

  • create a new release on GitHub
    • RC example
    • FINAL example
    • use the vX.Y.Z(-RCN) tag
    • link to the release issue
    • link to the changelog in the description
    • check the This is a pre-release checkbox
    • copy the changelog (without the header) in the description
    • do NOT check the This is a pre-release checkbox
  • run the sync-release-assets workflow
  • wait for the sync-release-assets workflow run to finish
  • verify the release assets are present in the GitHub release
• Promote the release
  using kuboreleaser release --version vX.Y.Z(-rcN) promote or ...

  • create an IPFS Discourse topic
    • prerelease example
    • release example
    • use Kubo vX.Y.Z(-RCN) is out! as the title
    • use kubo and go-ipfs as topics
    • repeat the title as a heading (##) in the description
    • link to the GitHub Release, binaries on IPNS, docker pull command and release notes in the description
  • pin the IPFS Discourse topic globally
    • you can make the topic a banner if there is no banner already
  • verify the IPFS Discourse topic was copied to:
    • #ipfs-chatter in IPFS Discord
    • #ipfs-chatter in FIL Slack
    • #ipfs-chatter:ipfs.io in Matrix
  • Add the link to the IPFS Discourse topic to the GitHub Release description
    • example
  • create an issue comment mentioning early testers on the release issue
    • example
  • create an issue comment linking to the release on the release issue
    • example
  • ask the marketing team to tweet about the release in #shared-pl-marketing-requests in FIL Slack
    • example
  • post the link to the GitHub Release to Reddit
    • example
• Test the new version with ipfs-companion
  using kuboreleaser release --version vX.Y.Z(-rcN) test-ipfs-companion or ...

  • run the e2e
    • use vX.Y.Z(-RCN) as the Kubo image version
  • wait for the e2e workflow run to finish
• Update Kubo in interop
  using kuboreleaser release --version vX.Y.Z(-rcN) update-interop or ...

  • check out ipfs/interop
  • run npm install
  • create a PR which updates package.json and package-lock.json
  • merge the PR
• Update Kubo in ipfs-desktop
  using kuboreleaser release --version vX.Y.Z(-rcN) update-ipfs-desktop or ...

  • check out ipfs/ipfs-desktop
  • run npm install
  • create a PR which updates package.json and package-lock.json
  • add @SgtPooki and @whizzzkid as reviewers
• Update Kubo docs
  using kuboreleaser release --version vX.Y.Z(-rcN) update-ipfs-docs or ...

  • run the update-on-new-ipfs-tag.yml workflow
  • merge the PR created by the update-on-new-ipfs-tag.yml workflow run
• Create a blog entry on ipfs.tech
  using kuboreleaser release --version vX.Y.Z(-rcN) update-ipfs-blog --date YYYY-MM-DD or ...

  • create a PR which adds a release note for the new Kubo version
    • example
  • merge the PR
  • verify the blog entry was published
• Merge the release branch back into master, ignoring the changes to version.go (keep the -dev) version,
  using kuboreleaser release --version vX.Y.Z(-rcN) merge-branch or ...

  • create a new branch merge-release-vX.Y.Z from release
  • create and merge a PR from merge-release-vX.Y.Z to master
• Prepare for the next release
  using kuboreleaser release --version vX.Y.Z(-rcN) prepare-next or ...

  • Create the next changelog
  • Link to the new changelog in the CHANGELOG.md file
  • Create the next release issue
• Create a dependency update PR
  • check out ipfs/kubo
  • run go get -u in root directory
  • run go mod tidy in root directory
  • run go mod tidy in docs/examples/kubo-as-a-library directory
  • create a PR which updates go.mod and go.sum
  • add the PR to the next release milestone
• Close the release issue

[BigLep]

The issue description has been updated with the updated set of items that are required for the 2023-06-01 RC.

[BigLep]

Issue description has been updated, but we should update to the latest go-libp2p (which is now go-libp2p 0.27.5, rather than 0.27.4).

[BigLep]

Concerning go-libp2p related releases...

We're not going to depend on the upcoming go-libp2p 0.28 release. go-libp2p v0.28.0 will introduce smart dialling. Smart dialling prioritizes LAN addresses when possible so if you know about a LAN address of someone else, you will first ONLY try those. Unfortunately we announce LAN addresses in the DHT, so those dials habr to fail (or a timer has to expire) before we actually try the public IPs to avoid a performance hit on all connection to nodes that are not on your LAN. As a result, we need libp2p/go-libp2p-kad-dht#839 which removes the private IPs from the public DHT. However if we add libp2p/go-libp2p-kad-dht#839 we are breaking the cloud VPC usecase by removing the ability to use private IPs on the DHT. Arguably relying on private IPs over public DHT was a bug, so we should fix it by giving users control over their private DHT too with #9884.

The Kubo release is already falling behind and I want to minimize the scope creep. We need to stick to the required items above. As a result, I think the actions are:

1. Kubo 0.21 depends on go-libp2p 0.27.5 (not 0.28)
2. Kubo 0.21 depends on a go-libp2p-kad-dht release that has fix: don't add unresponsive DHT servers to the Routing Table libp2p/go-libp2p-kad-dht#820 but not filter local addresses (for WAN) and localhost addresses (for LAN) libp2p/go-libp2p-kad-dht#839

Cc @marten-seemann and @guillaumemichel so they're aware

[BigLep]

Update concerning the go-libp2p release per Slack thread...

Kubo can update to go-libp2p 0.28 if it's ready since it won't incure extra work on the maintainers. This is because for go-libp2p 0.28 smart dialing local addresses and public addresses are dialed in parallel. For non-relay addresses, ranking and delaying dials happens within the address groups, not between address groups. It will not first dial localhost & LAN then public then relay. Instead go-libp2p v0.28 will first dial in parallel the best LAN, the best public v4 and the best public v6. There are docs about this in libp2p/go-libp2p#2336.

If go-libp2p 0.28 is ready when we start the Kubo release, we'll pick it up.

[BigLep]

2023-06-08 maintainer conversation: we aren't going to do the RC until 2023-06-12 because we're waiting on libp2p/go-libp2p-kad-dht#820. Per that PR, we don't expect that to land until 2023-06-12.

[hacdias]

Early testers ping for v0.21.0-rc1 testing 😄.

• Charity Engine (@Rytiss, @tristanolive)
• Fission (@bmann)
• Infura (@MichaelMure)
• OrbitDB (@aphelionz)
• pacman.store (@RubenKelevra)
• Pinata (@obo20)
• PL EngRes bifrost (@gmasgras)
• RTrade (@postables)
• Siderus (@koalalorenzo)
• Textile (@sanderpick)

You're getting this message because you're listed here. Please update this list if you no longer want to be included.

[BigLep]

There has been some discussion amongst Kubo and go-libp2p maintainers on whether the 0.21 release should hold up for the recent 0.28 go-libp2p release.

Kubo 0.21 with ship with go-libp2p 0.27.6 (not the newly release 0.28.0).

Given that I held up the 0.21 release for other things, why won't I hold it up for a new go-libp2p?

The Kubo release mostly slipped for the operational item fixes that we had discussed or drafted 3+ months ago (item 1, item 2). They had been spoken about for multiple months, and I didn't want to go into Q3 with those still open. They would have slipped to Q3 if 0.21 had shipped without them (since 0.22 won't happen until July). This is a judgement call, but for me it boils down to psychological, following through with public commitment to complete these changes by end of the quarter, and emphasizing that operational items are important (i.e., the train will depart to miss out on features, but we will cause disruption for security and operational items).

I don't want to take on go-libp2p 0.28 right now because:

1. It does take some extra time from @hacdias to do another RC (and a new minor release of go-libp2p should always require an RC).
2. It reduces the chances that we can complete the 0.21 release by EOD Thursday 2023-06-15 (which is then only one week behind our original date).
3. Internally to PL EngRes, we've got enough other stuff happening this month with H2 planning and perf that want to avoid unnecessary disruptions.

There is a lot of great functionality in go-libp2p 0.28, and I look forward it to landing in master/main soon and going out with Kubo 0.22, with a current RC targeting 2023-07-13.

How close do Kubo maintainers hit their release dates?

• 0.19 - 2 weeks late
• 0.20 - 2 weeks late (but IPFS Thing and an additional RC was also in there)
• 0.21 - currently targeting to be 1 week late.

[BigLep]

@hacdias : a couple of things about the release:

1. Can you please copy / paste in the checklist from https://github.com/ipfs/kubo/blob/master/docs/RELEASE_ISSUE_TEMPLATE.md so it's clear where we're at ? Alternatively or in addition, let's paste in the kubo release output for visibility.
2. Probably an item for the "improve the release process" PR, but it would be great if the PRs that are getting cut related to the release referenced this issue so that the two-way links show up in the audit trail.
3. I would like us to push towards doing the final release on Thursday, 2023-06-15. The key data point we need is the Thunderdome testing. Can you please start on that on Tuesday (I believe @Jorropo shadowed Gus on this last time), and @iand is a good source of help as well.

Thank you!

[SgtPooki]

Any chance we can get the latest webui in the this release? #9940

[hacdias]

@SgtPooki it will, but the interop tests with the WebUI are failing. Please check the comment I wrote on #9940. I was hoping you could take a look at it, because it seems something might've changed on the Web UI side.

[BigLep]

2023-06-13 maintainer conversation:

Things for RC2:

• go-libp2p update (0.27.6)
• fix for whatever is the problem with go routine leak found during Thunderdome testing

[p-shahi]

libp2p go routine leak is being tracked here libp2p/go-libp2p-kad-dht#849
@BigLep is there a link to the Thuderdome test results?

[BigLep]

I believe the Thunderdome results will be published to https://www.notion.so/kubo-prerelease-21-93caf901a408490480438b48974d8a07

[BigLep]

I updated the issue description with the items I believe need to make it into RC2

I forgot to save (doh!), but have done so now.

[hacdias]

Early testers ping for v0.21.0-rc2 testing 😄.

• Charity Engine (@Rytiss, @tristanolive)
• Fission (@bmann)
• Infura (@MichaelMure)
• OrbitDB (@aphelionz)
• pacman.store (@RubenKelevra)
• Pinata (@obo20)
• PL EngRes bifrost (@gmasgras)
• RTrade (@postables)
• Siderus (@koalalorenzo)
• Textile (@sanderpick)

You're getting this message because you're listed here. Please update this list if you no longer want to be included.

[BigLep]

In terms of release date, this will be 2023-06-19 at the earlier as need a successful thunderdome test run of RC2. I currently don't see one: https://protocollabs.grafana.net/d/GE2JD7ZVz/experiment-timeline?from=now-7d&to=now&var-experiment=kubo-prerelease-21&var-timeframe=6h&orgId=1

[BigLep]

(I'm recalling more details. I believe there will be a RC3 with a new go-libp2p 0.27.x patchh release. We'd then Thunderdome test that and confirm results before release.). Please correct me if I have any of that wrong @Jorropo .

[lidel]

Found a bug during RC2 testing, we need to include ipfs/boxo#358 in Kubo 0.21 (release blocker)

[hacdias]

Boxo patch release is up and Kubo is updated.

[hacdias]

Early testers ping for v0.21.0-rc3 testing 😄.

• Charity Engine (@Rytiss, @tristanolive)
• Fission (@bmann)
• Infura (@MichaelMure)
• OrbitDB (@aphelionz)
• pacman.store (@RubenKelevra)
• Pinata (@obo20)
• PL EngRes bifrost (@gmasgras)
• RTrade (@postables)
• Siderus (@koalalorenzo)
• Textile (@sanderpick)

You're getting this message because you're listed here. Please update this list if you no longer want to be included.

[BigLep]

In terms of next steps for completing the release, @Jorropo has confirmed in Thunderdome testing that goroutine leak and deadlock problems have been addressed. We now want to be confident that TTFB hasn't regressed. @Jorropo will capture his analysis in https://www.notion.so/pl-strflt/kubo-prerelease-21-93caf901a408490480438b48974d8a07. Once that is good, we'll release. We expect to do so by EOD Thursday, 2023-06-22, assuming no issues are found. Internal performance review work has reduced our bandwidth for getting this over the line quicker.

[BigLep]

2023-06-22 maintainer conversation:
Open things to fix:

• Migration issue with accelerated DHT (@Jorropo has context)
• Another migration issue (@Jorropo has context)
• webui: CORS issue on Peers page breaks geoip #9983
• Thunderdome testing (ensure it isn't a Kubo problem of why things stall after 6 hours)

[iand]

Every Thunderdome experiment has a configured duration, specified in minutes in when thunderdome deploy is run. After that time the experiment is terminated and all instances halted and deprovisioned. 6 hours is the recommended duration of a useful Kubo experiment, but I note that the example in the release docs uses 120 minutes.

[BigLep]

@Jorropo : can you please provide an update on how your three things from the 2023-06-22 conversation are going?

[Jorropo]

The fix for the migration and config issues are here #9995

[hacdias]

Quick comment here that we will want to include some Boxo fixes in the 0.21:

• fix(gateway): include CORS on subdomain redirects boxo#395
• fix(gateway): ensure 'X-Ipfs-Root' header is valid boxo#337

I think it is safe to release a Boxo 0.11 (there were breaking changes in boxo/ipns public API) and also include 5156f21 here. If there are any concerns, I can also work on releasing a cherry-picked patch release for Boxo with only the fixes above.

[marten-seemann]

@BigLep What's the timeline for this release? The release timeline seems to be quite outdated:

• Expected RC date: 2023-06-01 2023-06-08 2023-06-12
• 🚢 Expected final release date: 2023-06-08 2023-06-15

[hacdias]

@marten-seemann this week if Thunderdome testing results are good to go.

[marten-seemann]

Thank you @hacdias!
It would be really helpful if you could keep the issue up to date, so people can see what the current plan is 🙏

[Jorropo]

From thunderdome testing all kubo rc4 are still running with no memory leak, they keep memory very cool (~6GiB) compared to v0.20 (has a memory leak ~32GiB then crashes).
TTFB is noisy, but low connections dropped so that means the loadbalancer is able to get requests through, I think it's fine but what do I know ?

That a 🟢 on release for me on the thunderdom side.

[hacdias]

Release is scheduled for Monday 2023-07-04 2023-07-03.

[hacdias]

🎉 Kubo v0.21.0 is out!

[hacdias]

The release is essentially done, so I'm closing 😄 There are still one or two things left that will be taken care async.