fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568

schomatis · 2021-11-25T14:25:04Z

@lidel's blurb for Release Notes

This PR requires refactor: BlockPutSettings.CidPrefix interface-go-ipfs-core#80 to land first. Go there first.

BREAKING CHANGES

ipfs block commands
- ipfs block put now produces CIDv1 with raw codec by default now
  - ipfs block put --cid-codec makes block put return CID with alternative codec
    - this impacts only the returned CID, it does not trigger any validation or data transformation
    - codec names are validated against tables from https://github.com/multiformats/go-multicodec
  - ipfs block put --format is deprecated. It used incorrect codec names and should be avoided for new deployments. Use it only if you need the old, invalid behavior, namely:
    - ipfs block put --format=v0 will produce CIDv0 (implicit dag-pb)
    - ipfs block put --format=cbor will produce CIDv1 with dag-cbor (!)
    - ipfs block put --format=protobuf will produce CIDv1 with dag-pb (!)
ipfs cid codecs
- it now lists codecs from https://github.com/multiformats/go-multicodec
- ipfs cid codecs --supported can be passed to only show codecs supported in various go-ipfs commands

Depends on:

refactor: BlockPutSettings.CidPrefix interface-go-ipfs-core#80 (need to be merged and released before we can merge this PR)
tidy code generation, add KnownCodes, add Code.Tag method multiformats/go-multicodec#59
~~refactor: remove invalid string2code mappings go-cid#137~~
- not a dependency, we can use old version for now, only aciton here was to remove use of maps that will be removed when the go-cid PR is merged.

TODO:

Add ~~--store-codec~~ --cid-codec sharness tests.
update block and cid command helptext
Update github.com/ipfs/interface-go-ipfs-core dependency with version.
~~Update github.com/ipfs/go-cid dependency with version that includes fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568~~
- this was descoped, as noted in "Depends on" above
Update github.com/multiformats/go-multicodec dependency with version that includes tidy code generation, add KnownCodes, add Code.Tag method multiformats/go-multicodec#59
Fix sharness test "can set multihash type and length on block put without format" once interface-go-ipfs-core is reviewed and the --format=protobuf backward-compatibility is added back.

Closes #8471

BigLep · 2022-03-29T16:09:19Z

@schomatis : @lidel is going to take this over as it is being used to validate his work in ipfs/go-cid#137 . The issue has been reeassigned.

lidel · 2022-04-11T22:24:20Z

(wip rebase, need to rebase ipfs/interface-go-ipfs-core#80 too and switch this PR to that)

- switches to ipfs/interface-go-ipfs-core#80 (comment) - updates helptext of block commands to reflect the fact we now use CIDs everywhere and document the defaults - add tests for new --cid-codec and old --format behavior

we use it in IPNS-related commands

lidel

Ready for review together with ipfs/interface-go-ipfs-core#80

CI is green.
Added TLDR for release notes in the top comment fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568 (comment)

lidel · 2022-04-13T00:38:53Z

core/commands/block.go

-			if mhtval != mh.SHA2_256 || (mhlen != -1 && mhlen != 32) {
-				format = "protobuf"
-			} else {
-				format = "v0"
-			}


ℹ️ this now lives in https://github.com/ipfs/interface-go-ipfs-core/pull/80/files#diff-756c89773c36d0e2b7019845646b5754c2923b05807fa08ff91512a9b60c7369R93

lidel · 2022-04-19T16:35:54Z

TODO: see if we can error if no format or cid-codec is provided over HTTP RPC (or if the default from CLI applies everywhere).

lidel · 2022-04-19T18:41:44Z

core/commands/block.go

-		cmds.StringOption(mhtypeOptionName, "multihash hash function").WithDefault("sha2-256"),
-		cmds.IntOption(mhlenOptionName, "multihash hash length").WithDefault(-1),
-		cmds.BoolOption(pinOptionName, "pin added blocks recursively").WithDefault(false),
+		cmds.StringOption(blockCidCodecOptionName, "Multicodec to use in returned CID").WithDefault("raw"),


@aschmahmann I did some tests around this and the good news is that I was wrong, there is a way to have unified handling of default opts that works the same in ipfs block put CLI and raw HTTP POST to /api/v0/block/put.

If we keep the .WithDefault("raw") here, then this implicit default is also applied when people use curl against /api/v0/block/put and forget to pass cid-codec. It will also produce good docs at https://docs.ipfs.io/reference/http/api/#api-v0-block-put.

If we want to be extra strict and return error, I'd have to remove WithDefault and add some glue code that detects cidCodec == "" and throws error, but that feels hacky and annoying because it would make cid-codec a mandatory parameter, making it bad UX for the common use case where people are fine with codec being raw.

My vote is to keep WithDefault("raw").

Yep, the only breaking change is:

ipfs block put now produces CIDv1 with raw codec by default now

Which only effects the output CID, but ipfs block put <some-dagpb-stuff>; ipfs block get QmDagPBv0Thing will still work fine. This seems like a not too bad silent breakage, although I'm generally averse to silent breakages. WDYT?

sgtm. Added 0.13 breaking changes section to CHANGELOG.md in 2f5b8a4

https://github.com/ipfs/interface-go-ipfs-core/releases/tag/v0.7.0

lidel · 2022-04-21T15:57:47Z

Switched to interface-go-ipfs-core@v0.7.0.
Added breaking changes to CHANGELOG.md section for 0.13 release

As per triage verbal with Adin today, will merge as soon CI is green.

Applies necessary changes to ensure 'block/put' works and is backward-compatible. Context: ipfs/kubo#8568

* chore: interop with go-ipfs 0.13 Applies necessary changes to ensure 'block/put' works and is backward-compatible. Context: ipfs/kubo#8568 * chore: 0.3.1 bumping as patch because we bumped to 0.3.0 recently, as part of other (unreleased) go-ipfs 0.13 work

This ensures cid-codec introduced in ipfs/kubo#8568 gets correctly passed (+ we maintain backward-compatibility with CIDv0)

* chore: interop with go-ipfs 0.13 Applies necessary changes to ensure 'block/put' works and is backward-compatible. Context: #8568 * chore: 0.3.1 bumping as patch because we bumped to 0.3.0 recently, as part of other (unreleased) go-ipfs 0.13 work This commit was moved from ipfs/go-ipfs-http-client@ecf364c

This ensures cid-codec introduced in #8568 gets correctly passed (+ we maintain backward-compatibility with CIDv0) This commit was moved from ipfs/go-ipfs-http-client@9c9f43f

…cs' (#8568) BREAKING CHANGES: - see ipfs/kubo#8568 (comment) Co-authored-by: Marcin Rataj <lidel@lidel.org>

schomatis requested a review from aschmahmann November 25, 2021 14:25

schomatis self-assigned this Nov 25, 2021

schomatis mentioned this pull request Nov 27, 2021

Use standard IPLD codec names across the CLI/HTTP API #8471

Closed

BigLep added this to the Best Effort Track milestone Mar 3, 2022

This was referenced Mar 23, 2022

Create IPIP with Gateway spec for partial CAR exports ipfs/specs#348

Closed

feat: serve CBOR encoded DAG nodes from the gateway #8037

Closed

refactor: remove invalid string2code mappings ipfs/go-cid#137

Merged

BigLep modified the milestones: Best Effort Track, go-ipfs 0.13 Mar 29, 2022

BigLep assigned lidel and unassigned schomatis Mar 29, 2022

BigLep removed the request for review from aschmahmann March 29, 2022 16:10

lidel force-pushed the schomatis/fix/use-ipld-multicodecs branch from 6c90cda to 1b7c968 Compare April 11, 2022 22:20

lidel mentioned this pull request Apr 12, 2022

refactor: BlockPutSettings.CidPrefix ipfs/interface-go-ipfs-core#80

Merged

schomatis and others added 3 commits April 13, 2022 00:43

fix(cmds): use correct ipld multicodecs in cmds

4448de5

refactor: block put --cid-codec

e348b59

- switches to ipfs/interface-go-ipfs-core#80 (comment) - updates helptext of block commands to reflect the fact we now use CIDs everywhere and document the defaults - add tests for new --cid-codec and old --format behavior

chore: go-multicodec v0.4.1 + helptext fixes

70c13cf

lidel force-pushed the schomatis/fix/use-ipld-multicodecs branch from 1b7c968 to 70c13cf Compare April 13, 2022 00:37

lidel added 4 commits April 13, 2022 02:49

fix: add libp2p-key to supported codecs

a33ede2

we use it in IPNS-related commands

fix(test): t0110-gateway.sh

2119931

test: interface-go-ipfs-core CIDv1 raw and dag-pb

44cc485

fix(test): t0290-cid.sh

02aff74

lidel approved these changes Apr 13, 2022

View reviewed changes

lidel requested a review from aschmahmann April 13, 2022 02:09

lidel marked this pull request as ready for review April 13, 2022 02:12

test: use of both 'format' and 'cid-codec'

c34fe9c

lidel reviewed Apr 19, 2022

View reviewed changes

lidel added 2 commits April 21, 2022 17:54

docs(changelog): document breaking changes

2f5b8a4

chore: interface-go-ipfs-core v0.7.0

1f4902b

https://github.com/ipfs/interface-go-ipfs-core/releases/tag/v0.7.0

lidel merged commit 7b5fe80 into master Apr 21, 2022

lidel deleted the schomatis/fix/use-ipld-multicodecs branch April 21, 2022 16:19

lidel added a commit to ipfs/go-ipfs-http-client that referenced this pull request Apr 21, 2022

chore: interop with go-ipfs 0.13

df31b56

Applies necessary changes to ensure 'block/put' works and is backward-compatible. Context: ipfs/kubo#8568

lidel mentioned this pull request Apr 21, 2022

fix: interop with 'block put' from go-ipfs 0.13 ipfs/go-ipfs-http-client#158

Merged

lidel changed the title ~~fix(cmds): use ipld multicodecs for cmds block put and cid~~ fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ Apr 21, 2022

lidel added a commit to ipfs/go-ipfs-http-client that referenced this pull request Jun 3, 2022

feat: improved interop with go-ipfs 0.13

7b91563

This ensures cid-codec introduced in ipfs/kubo#8568 gets correctly passed (+ we maintain backward-compatibility with CIDv0)

lidel added a commit to ipfs/go-ipfs-http-client that referenced this pull request Jun 3, 2022

feat: improved interop with go-ipfs 0.13

93a5b2a

This ensures cid-codec introduced in ipfs/kubo#8568 gets correctly passed (+ we maintain backward-compatibility with CIDv0)

lidel mentioned this pull request Jun 3, 2022

feat: improved interop with go-ipfs 0.13 ipfs/go-ipfs-http-client#160

Merged

5 tasks

lidel added a commit to ipfs/go-ipfs-http-client that referenced this pull request Jun 3, 2022

feat: improved interop with go-ipfs 0.13

5d280d8

This ensures cid-codec introduced in ipfs/kubo#8568 gets correctly passed (+ we maintain backward-compatibility with CIDv0)

lidel added a commit to ipfs/go-ipfs-http-client that referenced this pull request Jun 23, 2022

fix: interop with go-ipfs 0.13 (#160)

9c9f43f

This ensures cid-codec introduced in ipfs/kubo#8568 gets correctly passed (+ we maintain backward-compatibility with CIDv0)

BigLep mentioned this pull request Jul 26, 2022

Release v0.14 #9032

Closed

68 tasks

Jorropo pushed a commit that referenced this pull request May 30, 2023

fix: interop with go-ipfs 0.13 (#160)

1cdb9ad

This ensures cid-codec introduced in #8568 gets correctly passed (+ we maintain backward-compatibility with CIDv0) This commit was moved from ipfs/go-ipfs-http-client@9c9f43f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568

fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568

schomatis commented Nov 25, 2021 •

edited by lidel

Loading

BigLep commented Mar 29, 2022

lidel commented Apr 11, 2022

lidel left a comment •

edited

Loading

lidel Apr 13, 2022

lidel commented Apr 19, 2022 •

edited

Loading

lidel Apr 19, 2022 •

edited

Loading

aschmahmann Apr 19, 2022 •

edited

Loading

lidel Apr 21, 2022 •

edited

Loading

lidel commented Apr 21, 2022

fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568

fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568

Conversation

schomatis commented Nov 25, 2021 • edited by lidel Loading

@lidel's blurb for Release Notes

BREAKING CHANGES

BigLep commented Mar 29, 2022

lidel commented Apr 11, 2022

lidel left a comment • edited Loading

Choose a reason for hiding this comment

lidel Apr 13, 2022

Choose a reason for hiding this comment

lidel commented Apr 19, 2022 • edited Loading

lidel Apr 19, 2022 • edited Loading

Choose a reason for hiding this comment

aschmahmann Apr 19, 2022 • edited Loading

Choose a reason for hiding this comment

lidel Apr 21, 2022 • edited Loading

Choose a reason for hiding this comment

lidel commented Apr 21, 2022

schomatis commented Nov 25, 2021 •

edited by lidel

Loading

lidel left a comment •

edited

Loading

lidel commented Apr 19, 2022 •

edited

Loading

lidel Apr 19, 2022 •

edited

Loading

aschmahmann Apr 19, 2022 •

edited

Loading

lidel Apr 21, 2022 •

edited

Loading