v1.6.1

Overview

We're happy to announce the Boost v1.6.1 release. This release contains several fixes and improvements for Boost 1.6.0, users are highly encouraged to upgrade.

See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

🌟 Highlights

🚀 Improved performance for retrievals over graphsync

We have simplified how unpaid retrievals over graphsync work. This results in significantly less calls to the Lotus API, and legacy code paths, which have improved download speeds and overall retrieval performance. All paid deals, or custom voucher types, will fallback to the old retrieval path.

A highlight of some of the key improvements:

• Resolved a potential memory leak that could occur from improperly terminated retrievals in the old code path.
• Reduced the frequency of timeout errors in popular retrieval clients, such as Lassie, by up to 30x. Improvements were more pronounced with higher parallelism.
• Reduced total memory utilization of retrieval by 40%
• While there is significant variance in retrieval performance due to latency, block size of the data being retrieved, size of retrieval, and number of parallel retrievals, we saw significant gains in performance over the existing code:
  • On a production SP with 30ms of latency we saw a minimum 20% improvement in retrieval speed
  • The benefit was less pronounced for higher latency, and larger retrievals, seeing on average an improvement of 5% with 120ms of latency
  • In a 0 latency test environment, retrieval speeds increased more than 35x
• Unpaid graphsync retrievals now use the same lightweight cache as booster-bitswap, which will improve performance for spikes in retrieval for the same content. This should help with performance when serving to retrieval clusters. Note, you don't need to be running booster-bitswap for this, as the cacheing layer is shared via boostd process.

💾 Online backup

Boost now supports online backups, so you no longer have to stop your node first.

boostd backup <backup directory>

See the backup and restore docs for more details.

What's Changed

• feat: make deal over json-rpc by @LexLuthr in #1262
• fix: set cancel log to debug level by @LexLuthr in #1275
• chore(deps): bump webpack from 5.72.1 to 5.76.1 in /react by @dependabot in #1278
• Simple graphsync retrieval server by @dirkmc in #1236
• async contract monitor by @nonsense in #1282
• feat: Online backup by @LexLuthr in #1265
• Dont log errors when the transfer event queue is full by @dirkmc in #1291
• fatally fail contract deals on publish errors by @nonsense in #1294
• better logs for deal id by @nonsense in #1298
• timer for eth_call by @nonsense in #1299
• print protos as part of boostx stats by @nonsense in #1300
• fixed "unknown actor code" constantly appearing in the boost logs via lotus upgrade @jacobheun in #1293
• Set user agent from boost build version instead of lotus by @dirkmc in #1305
• boostx stats: output agent version and retrieval protocols by @dirkmc in #1304
• fixed a potential panic in go-graphsync by @dirkmc in #1313
• fixed an issue where free storage space could be miscalculated after a config change by @LexLuthr in #1315

New Contributors

• @brossetti1 made their first contribution in #1302

Full Changelog: v1.6.0...v1.6.1

v1.6.1-rc1

Overview

We're happy to announce the Boost v1.6.1-rc1 release.

See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

Please make sure to upgrade to test the new features shipped in this release. See our upgrade guide for details.

🌟 Highlights

• This release contains multiple fixes and improvements for contract deals
• SP's can now create deals over to the RPC api, which should help improve deal flows when acting on behalf of your clients, as you no longer need to use libp2p
• Fixed an issue causing excessive unknown actor code logs

Improved performance for retrievals over graphsync

We have simplified how unpaid retrievals over graphsync work. This results in significantly less calls to the Lotus API, and legacy code paths, which should result in increased download speeds and overall retrieval performance. All paid deals, or custom voucher types, will fallback to the old retrieval path. We'll be posting more information about our performance testing as part of the official release.

What's Changed

• feat: make deals over json-rpc by @LexLuthr in #1262
• fix: set cancel log to debug level by @LexLuthr in #1275
• Simple graphsync retrieval server by @dirkmc in #1236
• async contract monitor by @nonsense in #1282
• feat: Online backup by @LexLuthr in #1265
• Disregard unseal price for unpaid retrieval by @dirkmc in #1289
• Dont log errors when the transfer event queue is full by @dirkmc in #1291
• fatally fail contract deals on publish errors by @nonsense in #1294
• better logs for deal id by @nonsense in #1298
• timer for eth_call by @nonsense in #1299
• print protos as part of boostx stats by @nonsense in #1300

Full Changelog: v1.6.0...v1.6.1-rc1

v1.6.0

Overview

We're happy to announce the Boost v1.6.0 release.

This release is compatible with Lotus v1.20.0 release. You should only upgrade to this release if you are also upgrading Lotus to v1.20.0 or higher for NV18 network upgrade. See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

Please make sure to upgrade to test the new features shipped in this release. See our upgrade guide for details.

Boost v1.5.x Adopters

If you have been running the v1.5.x and want to update to this release for lotus compatibility, you can do so as long as you are also upgrading Lotus to v1.20.0 or later. This PR contains the latest 1.5.3 codebase and we will continue to update it with any subsequent releases.

🌟 Highlights

Includes updates previously noted in RC builds of this release

FVM Contract Deal Support

• With the release of FVM, it is now possible for smart contracts to make deal proposals on-chain. This is made possible though the DealProposal FRC.
• DataDAOs, as well as other clients who want to store data on Filecoin, can now deploy a smart contract on the FVM which adheres to the DealProposal FRC, and make deal proposals that are visible to every storage provider running Boost who monitors the chain.
• In order to enable DealProposal FRC, you have to edit your config.toml and enable contract deal monitoring. By default it is disabled. Here is an example configuration:

[ContractDeals]
  Enabled = true
  AllowlistContracts = []
  From = "0x0000000000000000000000000000000000000000"

Cancelation of stalled retrievals

Boost now runs a background process that will cancel retrievals that have not been updated in the past 30minutes by default. You can configure this by setting the new DealMaking.StalledRetrievalTimeout property in the Boost config.

The following config will cancel all retrievals that have not changed state, or transferred bytes in the past 10minutes.

[Dealmaking]
  ...
  RetrievalLogDuration = "24h0m0s"
  StalledRetrievalTimeout = "10m0s"

Note: For efficiency of the job, Boost leverages the Retrieval Logs DB, so your RetrievalLogDuration should always exceed the StalledRetrievalTimeout.

Updated deal filter

Boost now exposes the following additional deal variables to the external deal filters like CIDgravity. This should allow more extensive filter logic for deal acceptance.

• IsOffline
• SkipIPNIAnnounce
• RemoveUnsealedCopy

🤩 Upcoming Features

These features will be part of the upcoming releases.

🆕 Graphsync Retrieval Server

• This is a simpler version of current graphsync server for retrievals
• It should allow much faster retrievals
• Only free retrievals will be handled by this code path. Paid retrievals will still go via the old code path

Online backup 💾

• Boost will support online backups along with the current offline version

What's Changed

• move markets code from lotus to boost by @dirkmc in #1186
• move some utilities to extern/boostd-data/shared by @dirkmc in #1195
• Port filecoin-project/lotus#9627 to Boost by @LexLuthr in #1202
• update go-bitswap dependency by @LexLuthr in #1204
• fix: BasicDealFilter logic by @LexLuthr in #1210
• feat: cache sealing pipeline status by @LexLuthr in #1209
• initial boost / fvm contract deals support by @nonsense in #1183
• port #1213 to main by @LexLuthr in #1214
• fix: bug bypassing basic filters by @LexLuthr in #1217
• add start-epoch-head-offset by @anjor in #1223
• Update types.go by @RobQuistNL in #1226
• fix homedir expansion by @nonsense in #1231
• feat: add job to cancel stalled retrievals by @jacobheun in #1233
• clarify cid type by @anjor in #1234
• test: devnet quality of life updates by @jacobheun in #1245
• Simplify docker build tasks by @airenas in #1230
• remove MessageConfidence from boost build params by @nonsense in #1246
• fix docker image tagging by @nonsense in #1247
• FRC DealProposal integration by @nonsense in #1216
• feat(ui): display end of peerid instead of front by @jacobheun in #1252
• remove defer statement for boost retrieve by @nonsense in #1254
• add recover to contract monitor by @nonsense in #1256

New Contributors

• @anjor made their first contribution in #1223
• @RobQuistNL made their first contribution in #1226

Full Changelog: v1.5.3...v1.6.0

v1.6.0-rc3 - Lotus v1.20.0 Compatibility

Overview

We're happy to announce the Boost v1.6.0-rc3 release.

This is a compatibility update for the Lotus v1.20.0 release. You should only upgrade to this release if you are also upgrading Lotus to v1.20.0. See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

Please make sure to upgrade to test the new features shipped in this release. See our upgrade guide for details.

Boost v1.5.x Adopters

If you have been running the v1.5.x and want to update to this release for lotus compatibility, you can do so as long as you are also upgrading Lotus to v1.20.0. This PR contains the latest 1.5.3 codebase and we will continue to update it with any subsequent releases.

🌟 Highlights

Includes updates previously noted in RC builds of this release

FVM Contract Deal Support

• With the release of FVM, it is now possible for smart contracts to make deal proposals on-chain. This is made possible though the DealProposal FRC.
• DataDAOs, as well as other clients who want to store data on Filecoin, can now deploy a smart contract on the FVM which adheres to the DealProposal FRC, and make deal proposals that are visible to every storage provider running Boost who monitors the chain.
• In order to enable DealProposal FRC, you have to edit your config.toml and enable contract deal monitoring. By default it is disabled. Here is an example configuration:

[ContractDeals]
  Enabled = true
  AllowlistContracts = []
  From = "0x0000000000000000000000000000000000000000"

Cancelation of stalled retrievals

Boost now runs a background process that will cancel retrievals that have not been updated in the past 30minutes by default. You can configure this by setting the new DealMaking.StalledRetrievalTimeout property in the Boost config.

The following config will cancel all retrievals that have not changed state, or transferred bytes in the past 10minutes.

[Dealmaking]
  ...
  RetrievalLogDuration = "24h0m0s"
  StalledRetrievalTimeout = "10m0s"

Note: For efficiency of the job, Boost leverages the Retrieval Logs DB, so your RetrievalLogDuration should always exceed the StalledRetrievalTimeout.

Updated deal filter

Boost now exposes the following additional deal variables to the external deal filters like CIDgravity. This should allow more extensive filter logic for deal acceptance.

• IsOffline
• SkipIPNIAnnounce
• RemoveUnsealedCopy

What's Changed

• move markets code from lotus to boost by @dirkmc in #1186
• move some utilities to extern/boostd-data/shared by @dirkmc in #1195
• Port filecoin-project/lotus#9627 to Boost by @LexLuthr in #1202
• update go-bitswap dependency by @LexLuthr in #1204
• fix: BasicDealFilter logic by @LexLuthr in #1210
• feat: cache sealing pipeline status by @LexLuthr in #1209
• initial boost / fvm contract deals support by @nonsense in #1183
• port #1213 to main by @LexLuthr in #1214
• fix: bug bypassing basic filters by @LexLuthr in #1217
• add start-epoch-head-offset by @anjor in #1223
• Update types.go by @RobQuistNL in #1226
• fix homedir expansion by @nonsense in #1231
• feat: add job to cancel stalled retrievals by @jacobheun in #1233
• clarify cid type by @anjor in #1234
• test: devnet quality of life updates by @jacobheun in #1245
• Simplify docker build tasks by @airenas in #1230
• remove MessageConfidence from boost build params by @nonsense in #1246
• fix docker image tagging by @nonsense in #1247
• FRC DealProposal integration by @nonsense in #1216
• feat(ui): display end of peerid instead of front by @jacobheun in #1252
• remove defer statement for boost retrieve by @nonsense in #1254
• add recover to contract monitor by @nonsense in #1256

New Contributors

• @anjor made their first contribution in #1223
• @RobQuistNL made their first contribution in #1226

Full Changelog: v1.5.3...v1.6.0-rc3

v1.5.3

Overview

We're happy to announce Boost 1.5.3 which includes automatic cancelation of stalled retrievals.

Lotus 1.20.0 Users: If you have already upgraded Lotus to 1.20.0 you should be using Boost 1.6.0-rc1 or later. See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

An official release of 1.6 is scheduled for March 8th.

Highlights

Cancelation of stalled retrievals

Boost now runs a background process that will cancel retrievals that have not been updated in the past 30minutes by default. You can configure this by setting the new DealMaking.StalledRetrievalTimeout property in the Boost config.

The following config will cancel all retrievals that have not changed state, or transferred bytes in the past 10minutes.

[Dealmaking]
  ...
  RetrievalLogDuration = "24h0m0s"
  StalledRetrievalTimeout = "10m0s"

Note: For efficiency of the job, Boost leverages the Retrieval Logs DB, so your RetrievalLogDuration should always exceed the StalledRetrievalTimeout.

What's Changed

• feat: back port add job to cancel stalled retrievals (#1233) by @jacobheun in #1239

Full Changelog: v1.5.2...v1.5.3

v1.6.0-rc2 - Lotus v1.20.0-rcX Compatibility

Overview

We're happy to announce the Boost v1.6.0-rc2 release.

This is a patch release to fix slow offline deal processing by Boost. You should upgrade to this release if you are currently running Boost v.1.6.0-rc1. See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

Please make sure to upgrade to test the new features shipped in this release. See our upgrade guide for details.

What's Changed

• feat: cache sealing pipeline status by @LexLuthr in #1209
• fix: BasicDealFilter logic by @LexLuthr in #1210
• fix: send more deal params to deal filters by @LexLuthr in #1213
• fix: bug bypassing basic filters by @LexLuthr in #1217

Full Changelog: v1.6.0-rc1...v1.6.0-rc2

v1.5.2

Overview

We're happy to announce the Boost v1.5.2 release.

This is a patch release to fix slow offline deal processing by Boost. You should upgrade to this release if you are currently running Boost v.1.5.1. See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

Please make sure to upgrade to test the new features shipped in this release. See our upgrade guide for details.

What's Changed

• fix: offline deal slow acceptance and ConsiderOnlineStorageDeals rejection by @LexLuthr in #1211
• fix: send more deal params to deal filters by @LexLuthr in #1213
• fix: basic deal filters no longer skipped #1218

Full Changelog: v1.5.1...v1.5.2

v1.6.0-rc1 - Lotus v1.20.0-rc1 Compatibility

Overview

We're happy to announce the Boost v1.6.0-rc1 release.

This is a compatibility update for the Lotus v1.20.0-rc1 release. You should only upgrade to this release if you are also upgrading Lotus to v1.20.0-rc1. See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

Please make sure to upgrade to test the new features shipped in this release. See our upgrade guide for details.

Boost v1.5.1 RC Adopters

If you have been running the v1.5.1 RC and want to update to this release for lotus compatibility, you can do so as long as you are also upgrading Lotus to v1.20.0-rc1. This PR contains the latest 1.5.1 codebase and we will continue to update it with any subsequent releases.

What's Changed

• Update support for lotus v1.20.0-rc1 by @jacobheun in #1188
• feat: add flags for bitswap server internals by @jacobheun in #1182
• Introduce deal proposal protocol v1.2.1 by @dirkmc in #1185
• Devnet images using Lotus from DockerHub by @airenas in #1157

Full Changelog: v1.5.1-rc5...v1.6.0-rc1

v1.5.1 - Bitswap support and more!

Overview

We're happy to announce the Boost v1.5.1 stable release, with several new features and fixes!
See our upgrade guide for details on upgrading.

Please review the Boost and Lotus compatibility matrix to plan your upgrade.

🌟 Highlights

Includes updates previously noted in RC builds of this release

IPFS retrieval support is here with Booster-bitswap!

• booster-bitswap is a binary that runs alongside the boostd process, to serve retrievals over the Bitswap protocol, which is the data transfer protocol used in IPFS. This release provides tools for managing a bitswap service for serving retrievals.
• Note that there is currently no payment method mechanism in booster-bitswap; this endpoint is intended to serve free content.
• Check out the docs on how to start serving Bitswap retrievals

Default Grafana dashboards and monitoring

• Boost now includes configured Grafana dashboards that SP's can enable with monitoring. See the monitoring tutorial in the docs for an example on how this is set up via docker. We plan to add more dashboards over time to improve visibility into the systems performance. Reach out in #boost-help on Slack if you have any questions!

Boostd process overview dashboard	Booster-bitswap retrieval dashboard

New storage deal proposal flags

• The new storage deal proposal flags adds configurability options to indexer announcements and storing unsealed copies of data. The current default storage deal protocol behaviors remain the same.
• Check out the details in the Propose Storage Deal Protocol docs

HTTP Retrieval Updates

• HTTP retrieval URLs have been redesigned to support piece retrievals for Spade (previously slingshot-evergreen).
  • We've simplified the /piece endpoint for ease of use, and optimized for existing user needs
  • BREAKING CHANGE: We removed support for getting the piece by its PayloadCid and the ability to retrieve the payload. We may revisit adding support back if there is clear demand.
• We've added a new /info endpoint that includes versioning of booster-http to make it easier to track changes
• You can see the updated endpoints in the docs - https://boost.filecoin.io/experimental-features/http-retrieval.

And more!

• You can now use the boost client to retrieve data similar to lotus' retrieve command, example:
  boost retrieve --provider=t01000 bafk2bzacedsnewk7clxwab2wgwyoi7u5tzdhldx7fkxpqdq7unrxz2zoy4d2g
• Boostx now includes a utility command for unsealing a sector, example:
  boostx sector unseal --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO 2

What's Changed

• Add wallet sign to boost cli by @jsmetana in #915
• add throttler to libp2p http server for estuary by @nonsense in #927
• Periodic logsDB cleanup based on configured duration by @LexLuthr in #929
• feat: add transparent gzip support for booster-http response by @LexLuthr in #932
• feat: retry offline deal after commp errors by @LexLuthr in #899
• feat: show datacap in boost wallet list by @LexLuthr in #959
• Add meaningful default for start-epoch by @parkan in #982
• feat: introduce http api version by @LexLuthr in #972
• Surface retrieval logs in Web UI by @dirkmc in #961
• Inspect by non-root cid by @dirkmc in #998
• feat: client flag to not announce deals by @LexLuthr in #1051
• Sort sealing worker list by date by @dirkmc in #1074
• Add bitswap bytes served grafana dashboard by @dirkmc in #1117
• add overview dashboard by @nonsense in #1122
• Add the ability to filter certain properties in the deal list by @jacobheun in #1097
• Add more storage deal filter parameters by @dirkmc in #497
• port boost retrieve to main branch by @dirkmc in #1163
• Port unseal command to main by @dirkmc in #1155
• feat: add flags for bitswap server internals by @jacobheun in #1182
• Introduce deal proposal protocol v1.2.1 by @dirkmc in #1185

New Contributors

• @jsmetana made their first contribution in #915
• @parkan made their first contribution in #982
• @ribasushi made their first contribution in #1152
• @ischasny made their first contribution in #1159

Full Changelog: v1.5.0...v1.5.1

v1.5.1-rc5

We're happy to announce the v1.5.1-rc5 release. This is the final planned RC before release.

Highlights

• Fixes an issue with booster-bitswap request caches not getting pruned. This should improve memory usage overtime. See #1167 for config options for managing the cache size and expiry time.
• Improves resiliency of deny filter downloads for booster-bitswap
• You can now use the boost client to retrieve data similar to lotus' retrieve command, example:
  boost retrieve --provider=t01000 bafk2bzacedsnewk7clxwab2wgwyoi7u5tzdhldx7fkxpqdq7unrxz2zoy4d2g
• Boostx now includes a utility command for unsealing a sector, example:
  boostx sector unseal --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO 2

What's Changed

• Add more storage deal filter parameters by @dirkmc in #497
• Fetch bitswap deny filters from remote if local cache is invalid by @ischasny in #1159
• port boost retrieve to RC by @dirkmc in #1163
• Port unseal command to RC @dirkmc in #1155
• feat: use an expiry lru cache for the index backed blockstore by @jacobheun in #1167

New Contributors

• @ribasushi made their first contribution in #1152
• @ischasny made their first contribution in #1159

Full Changelog: v1.5.1-rc4...v1.5.1-rc5