Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

feat: update DAG API to match go-ipfs@0.10 changes #3917

Merged
merged 20 commits into from
Dec 3, 2021
Merged

feat: update DAG API to match go-ipfs@0.10 changes #3917

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
b7b0b3f
feat!: update http-client DAG API to match go-ipfs@0.10
rvagg Oct 11, 2021
d558a60
feat!: update DAG API options types
rvagg Oct 11, 2021
5d1cdfa
chore: update go-ipfs dependency
rvagg Oct 11, 2021
24626cb
feat!: update http-server DAG API to match go-ipfs@0.10
rvagg Oct 11, 2021
80b0343
feat!: update core DAG API to match go-ipfs@0.10
rvagg Oct 11, 2021
374349d
feat!: update cli DAG API to match go-ipfs@0.10
rvagg Oct 11, 2021
c0286df
feat!: update interface-ipfs-core DAG API to match go-ipfs@0.10
rvagg Oct 14, 2021
3907671
feat!: update docs for DAG API to match go-ipfs@0.10
rvagg Oct 14, 2021
8899e40
feat!: update ipfs for DAG GET API to match go-ipfs@0.10
rvagg Oct 14, 2021
f8fa2f9
chore(test): add sharness t0053 tests equivalent to go-ipfs
rvagg Oct 18, 2021
f4f42a2
fix: update runnable tests to match go-ipfs
rvagg Oct 18, 2021
4d77f1d
feat: object API returns CIDv1 by default
rvagg Nov 15, 2021
a960d28
TEMPORARY - remove custom ipld formats example test
rvagg Nov 16, 2021
abd817a
Merge branch 'master' into rvagg/dag-api-0.10
lidel Dec 2, 2021
c6816a4
fix(ci): fail-fast: false
lidel Dec 2, 2021
50cbc1f
fix(ci): make tests use less workers
lidel Dec 2, 2021
566f10a
Merge remote-tracking branch 'origin/master' into rvagg/dag-api-0.10
achingbrain Dec 3, 2021
0a2f12c
chore: disable ci concurrency control
achingbrain Dec 3, 2021
ce58e59
chore: remove fail fast and fix block.put arg
achingbrain Dec 3, 2021
376d27d
Merge remote-tracking branch 'origin/master' into rvagg/dag-api-0.10
achingbrain Dec 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 20 additions & 6 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,12 @@ jobs:
name: Unit tests ${{ matrix.project }} node ${{ matrix.node }} ${{ matrix.os }}
needs: build
runs-on: ${{ matrix.os }}
concurrency: js-ipfs-test-${{ github.ref }}
achingbrain marked this conversation as resolved.
Show resolved Hide resolved
strategy:
matrix:
os: [windows-latest, ubuntu-latest, macos-latest]
node: [16]
fail-fast: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK gh orgs only have a certain number of workers available for actions - this may end up hogging resources and preventing other builds from running when we know a given CI run will fail.

Copy link
Member

@lidel lidel Dec 2, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we truly care about not using too many workers then fail-fast does not buy us much.
We should limit the number of concurrent jobs with concurrency – did a poc in 50cbc1f where we limit number of test workers consumed per github revision, which makes PRs independent of each other.

Bit confused why CI stopped passing without any package.json changes – is lerna setup fuzzy like this by design?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, this wasn't meant to be here for the final version - I turned this off because of the examples failures, which are currently commented out and need to be addressed before merge but I'm not sure how to do that dance when they depend on eachother being fixed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we truly care about not using too many workers

We should be optimistic that builds will complete successfully, so they should consume as many resources as necessary to do that expediently (within reason). If they fail, they should not stand in the way of other builds completing, so should fail fast.

Not sure what reasonable is yet but so far so good.

Bit confused why CI stopped passing without any package.json changes

Right now it's failing because the types in iso-random-stream are broken - needs hugomrdias/iso-random-stream#91

I'm not sure how to do that dance when they depend on each other being fixed

We can check out branches of the example tests with fixes applied which should get the build passing.

e.g. in .github/workflows/test.yml

- name: ipfs browser video streaming
  repo: https://github.com/ipfs-examples/js-ipfs-browser-video-streaming.git
  deps: ipfs-core@$PWD/packages/ipfs-core/dist

would become:

- name: ipfs browser video streaming
  repo: https://github.com/ipfs-examples/js-ipfs-browser-video-streaming.git#fix/this-is-a-branch-with-a-fix
  deps: ipfs-core@$PWD/packages/ipfs-core/dist

If that's not practical, I think it's ok to merge with the example tests disabled, then fix the examples and re-enable them in a follow up PR. As long as there isn't a massive lag between the two.

steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -101,6 +103,7 @@ jobs:
name: Unit tests ${{ matrix.project }} ${{ matrix.browser }} ${{ matrix.type }}
needs: build
runs-on: ubuntu-latest
concurrency: js-ipfs-test-${{ github.ref }}
strategy:
matrix:
browser:
Expand All @@ -109,6 +112,7 @@ jobs:
type:
- browser
- webworker
fail-fast: false
steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -139,6 +143,7 @@ jobs:
name: Unit tests electron-main
needs: build
runs-on: ubuntu-latest
concurrency: js-ipfs-test-${{ github.ref }}
steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -171,6 +176,7 @@ jobs:
name: Unit tests electron-renderer
needs: build
runs-on: ubuntu-latest
concurrency: js-ipfs-test-${{ github.ref }}
steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -203,13 +209,15 @@ jobs:
name: Interop tests ${{ matrix.project }} ${{ matrix.type }}
needs: build
runs-on: ubuntu-latest
concurrency: js-ipfs-test-${{ github.ref }}
strategy:
matrix:
type:
- node
- browser
#- electron-main
#- electron-renderer
fail-fast: false
steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -240,6 +248,7 @@ jobs:
name: Interface tests ${{ matrix.suite }} ${{ matrix.type }}
needs: build
runs-on: ubuntu-latest
concurrency: js-ipfs-test-${{ github.ref }}
strategy:
matrix:
type:
Expand All @@ -252,6 +261,7 @@ jobs:
- test:interface:client
- test:interface:http-go
- test:interface:http-js
fail-fast: false
steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -282,6 +292,7 @@ jobs:
name: Interface tests test:interface:message-port-client browser
needs: build
runs-on: ubuntu-latest
concurrency: js-ipfs-test-${{ github.ref }}
steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -312,6 +323,7 @@ jobs:
name: Test example ${{ matrix.example.name }}
needs: build
runs-on: ubuntu-latest
concurrency: js-ipfs-test-${{ github.ref }}
strategy:
matrix:
example:
Expand Down Expand Up @@ -366,9 +378,9 @@ jobs:
- name: ipfs custom ipfs repo
repo: https://github.com/ipfs-examples/js-ipfs-custom-ipfs-repo.git
deps: ipfs-core@$PWD/packages/ipfs-core/dist
- name: ipfs custom ipld formats
repo: https://github.com/ipfs-examples/js-ipfs-custom-ipld-formats.git
deps: ipfs-core@$PWD/packages/ipfs-core/dist,ipfs-daemon@$PWD/packages/ipfs-daemon/dist,ipfs-http-client@$PWD/packages/ipfs-http-client/dist
#- name: ipfs custom ipld formats
# repo: https://github.com/ipfs-examples/js-ipfs-custom-ipld-formats.git
# deps: ipfs-core@$PWD/packages/ipfs-core/dist,ipfs-daemon@$PWD/packages/ipfs-daemon/dist,ipfs-http-client@$PWD/packages/ipfs-http-client/dist
- name: ipfs custom libp2p
repo: https://github.com/ipfs-examples/js-ipfs-custom-libp2p.git
deps: ipfs-core@$PWD/packages/ipfs-core/dist
Expand Down Expand Up @@ -396,9 +408,9 @@ jobs:
- name: ipfs running multiple nodes
repo: https://github.com/ipfs-examples/js-ipfs-running-multiple-nodes.git
deps: ipfs@$PWD/packages/ipfs/dist
- name: ipfs traverse ipld graphs
repo: https://github.com/ipfs-examples/js-ipfs-traverse-ipld-graphs.git
deps: ipfs-core@$PWD/packages/ipfs-core/dist
#- name: ipfs traverse ipld graphs
# repo: https://github.com/ipfs-examples/js-ipfs-traverse-ipld-graphs.git
# deps: ipfs-core@$PWD/packages/ipfs-core/dist
- name: types with typescript
repo: https://github.com/ipfs-examples/js-ipfs-types-use-ipfs-from-ts.git
deps: ipfs-core@$PWD/packages/ipfs-core/dist
Expand Down Expand Up @@ -436,6 +448,7 @@ jobs:
name: Test external ${{ matrix.external.name }}
needs: build
runs-on: ubuntu-latest
concurrency: js-ipfs-test-${{ github.ref }}
strategy:
matrix:
external:
Expand All @@ -451,6 +464,7 @@ jobs:
- name: ipfs-log
repo: https://github.com/orbitdb/ipfs-log.git
deps: ipfs@$PWD/packages/ipfs/dist,orbit-db-io@next
fail-fast: false
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
Expand Down
11 changes: 6 additions & 5 deletions docs/core-api/DAG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,12 +88,13 @@ An optional object which may have the following keys:

| Name | Type | Default | Description |
| ---- | ---- | ------- | ----------- |
| format | `String` | `'dag-cbor'` | The IPLD format multicodec |
| storeCodec | `String` | `'dag-cbor'` | The codec that the stored object will be encoded with |
| inputCodec | `String` | `undefined` | If an already encoded object is provided (as a `Uint8Array`), the codec that the object is encoded with, otherwise it is assumed the `dagNode` argument is an object to be encoded |
| hashAlg | `String` | `'sha2-256'` | The hash algorithm to be used over the serialized DAG node |
| cid | [CID][] | `'dag-cbor'` | The IPLD format multicodec |
| pin | `boolean` | `false` | Pin this node when adding to the blockstore |
| timeout | `Number` | `undefined` | A timeout in ms |
| signal | [AbortSignal][] | `undefined` | Can be used to cancel any long running requests started as a result of this call |
| signal | [AbortSignal][] | `undefined` | Can be used to cancel any long running requests started as a result of this call |

**Note**: You should pass `cid` or the `format` & `hashAlg` pair but _not both_.

Expand All @@ -107,7 +108,7 @@ An optional object which may have the following keys:

```JavaScript
const obj = { simple: 'object' }
const cid = await ipfs.dag.put(obj, { format: 'dag-cbor', hashAlg: 'sha2-512' })
const cid = await ipfs.dag.put(obj, { storeCodec: 'dag-cbor', hashAlg: 'sha2-512' })

console.log(cid.toString())
// zBwWX9ecx5F4X54WAjmFLErnBT6ByfNxStr5ovowTL7AhaUR98RWvXPS1V3HqV1qs3r5Ec5ocv7eCdbqYQREXNUfYNuKG
Expand Down Expand Up @@ -160,7 +161,7 @@ const obj = {
}
}

const cid = await ipfs.dag.put(obj, { format: 'dag-cbor', hashAlg: 'sha2-256' })
const cid = await ipfs.dag.put(obj, { storeCodec: 'dag-cbor', hashAlg: 'sha2-256' })
console.log(cid.toString())
// zdpuAmtur968yprkhG9N5Zxn6MFVoqAWBbhUAkNLJs2UtkTq5

Expand Down Expand Up @@ -272,7 +273,7 @@ const obj = {
}
}

const cid = await ipfs.dag.put(obj, { format: 'dag-cbor', hashAlg: 'sha2-256' })
const cid = await ipfs.dag.put(obj, { storeCodec: 'dag-cbor', hashAlg: 'sha2-256' })
console.log(cid.toString())
// bafyreicyer3d34cutdzlsbe2nqu5ye62mesuhwkcnl2ypdwpccrsecfmjq

Expand Down
16 changes: 8 additions & 8 deletions packages/interface-ipfs-core/src/block/rm.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ export function testRm (factory, options) {

it('should remove by CID object', async () => {
const cid = await ipfs.dag.put(uint8ArrayFromString(nanoid()), {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
})

Expand All @@ -61,15 +61,15 @@ export function testRm (factory, options) {
it('should remove multiple CIDs', async () => {
const cids = await Promise.all([
ipfs.dag.put(uint8ArrayFromString(nanoid()), {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
}),
ipfs.dag.put(uint8ArrayFromString(nanoid()), {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
}),
ipfs.dag.put(uint8ArrayFromString(nanoid()), {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
})
])
Expand All @@ -86,7 +86,7 @@ export function testRm (factory, options) {

it('should error when removing non-existent blocks', async () => {
const cid = await ipfs.dag.put(uint8ArrayFromString(nanoid()), {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
})

Expand All @@ -102,7 +102,7 @@ export function testRm (factory, options) {

it('should not error when force removing non-existent blocks', async () => {
const cid = await ipfs.dag.put(uint8ArrayFromString(nanoid()), {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
})

Expand All @@ -119,7 +119,7 @@ export function testRm (factory, options) {

it('should return empty output when removing blocks quietly', async () => {
const cid = await ipfs.dag.put(uint8ArrayFromString(nanoid()), {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
})
const result = await all(ipfs.block.rm(cid, { quiet: true }))
Expand All @@ -129,7 +129,7 @@ export function testRm (factory, options) {

it('should error when removing pinned blocks', async () => {
const cid = await ipfs.dag.put(uint8ArrayFromString(nanoid()), {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
})
await ipfs.pin.add(cid)
Expand Down
22 changes: 11 additions & 11 deletions packages/interface-ipfs-core/src/dag/get.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,16 +73,16 @@ export function testGet (factory, options) {
Data: uint8ArrayFromString('I am inside a Protobuf'),
Links: []
}
cidPb = CID.createV0(await sha256.digest(dagPB.encode(nodePb)))
cidPb = CID.createV1(dagPB.code, await sha256.digest(dagPB.encode(nodePb)))
nodeCbor = {
someData: 'I am inside a Cbor object',
pb: cidPb
}

cidCbor = CID.createV1(dagCBOR.code, await sha256.digest(dagCBOR.encode(nodeCbor)))

await ipfs.dag.put(nodePb, { format: 'dag-pb', hashAlg: 'sha2-256' })
await ipfs.dag.put(nodeCbor, { format: 'dag-cbor', hashAlg: 'sha2-256' })
await ipfs.dag.put(nodePb, { storeCodec: 'dag-pb', hashAlg: 'sha2-256' })
await ipfs.dag.put(nodeCbor, { storeCodec: 'dag-cbor', hashAlg: 'sha2-256' })
})

it('should respect timeout option when getting a DAG node', () => {
Expand All @@ -93,7 +93,7 @@ export function testGet (factory, options) {

it('should get a dag-pb node', async () => {
const cid = await ipfs.dag.put(pbNode, {
format: 'dag-pb',
storeCodec: 'dag-pb',
hashAlg: 'sha2-256'
})

Expand All @@ -105,7 +105,7 @@ export function testGet (factory, options) {

it('should get a dag-cbor node', async () => {
const cid = await ipfs.dag.put(cborNode, {
format: 'dag-cbor',
storeCodec: 'dag-cbor',
hashAlg: 'sha2-256'
})

Expand All @@ -122,7 +122,7 @@ export function testGet (factory, options) {

const node = result.value

const cid = CID.createV0(await sha256.digest(dagPB.encode(node)))
const cid = CID.createV1(dagPB.code, await sha256.digest(dagPB.encode(node)))
expect(cid.equals(cidPb)).to.be.true()
})

Expand Down Expand Up @@ -192,7 +192,7 @@ export function testGet (factory, options) {
}

const cid = await ipfs.dag.put(node, {
format: 'dag-pb',
storeCodec: 'dag-pb',
hashAlg: 'sha2-256',
version: 0
})
Expand Down Expand Up @@ -226,7 +226,7 @@ export function testGet (factory, options) {
foo: 'dag-cbor-bar'
}

const cid = await ipfs.dag.put(cbor, { format: 'dag-cbor', hashAlg: 'sha2-256' })
const cid = await ipfs.dag.put(cbor, { storeCodec: 'dag-cbor', hashAlg: 'sha2-256' })
expect(cid.code).to.equal(dagCBOR.code)
expect(cid.toString(base32)).to.equal('bafyreic6f672hnponukaacmk2mmt7vs324zkagvu4hcww6yba6kby25zce')

Expand All @@ -241,10 +241,10 @@ export function testGet (factory, options) {
foo: 'dag-cbor-bar'
}

const cid1 = await ipfs.dag.put(cbor1, { format: 'dag-cbor', hashAlg: 'sha2-256' })
const cid1 = await ipfs.dag.put(cbor1, { storeCodec: 'dag-cbor', hashAlg: 'sha2-256' })
const cbor2 = { other: cid1 }

const cid2 = await ipfs.dag.put(cbor2, { format: 'dag-cbor', hashAlg: 'sha2-256' })
const cid2 = await ipfs.dag.put(cbor2, { storeCodec: 'dag-cbor', hashAlg: 'sha2-256' })

const result = await ipfs.dag.get(cid2, {
path: 'other/foo'
Expand All @@ -256,7 +256,7 @@ export function testGet (factory, options) {
const buf = Uint8Array.from([0, 1, 2, 3])

const cid = await ipfs.dag.put(buf, {
format: 'raw',
storeCodec: 'raw',
hashAlg: 'sha2-256'
})

Expand Down
4 changes: 3 additions & 1 deletion packages/interface-ipfs-core/src/dag/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,15 @@ import { testGet } from './get.js'
import { testPut } from './put.js'
import { testImport } from './import.js'
import { testResolve } from './resolve.js'
import { testDagSharnessT0053 } from './sharness-t0053-dag.js'

const tests = {
export: testExport,
get: testGet,
put: testPut,
import: testImport,
resolve: testResolve
resolve: testResolve,
dagSharnessT0053: testDagSharnessT0053
}

export default createSuite(tests)
12 changes: 6 additions & 6 deletions packages/interface-ipfs-core/src/dag/put.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,35 +37,35 @@ export function testPut (factory, options) {

it('should put dag-pb with default hash func (sha2-256)', () => {
return ipfs.dag.put(pbNode, {
format: 'dag-pb',
storeCodec: 'dag-pb',
hashAlg: 'sha2-256'
})
})

it('should put dag-pb with non-default hash func (sha2-512)', () => {
return ipfs.dag.put(pbNode, {
format: 'dag-pb',
storeCodec: 'dag-pb',
hashAlg: 'sha2-512'
})
})

it('should put dag-cbor with default hash func (sha2-256)', () => {
return ipfs.dag.put(cborNode, {
format: 'dag-cbor',
storeCodec: 'dag-cbor',
hashAlg: 'sha2-256'
})
})

it('should put dag-cbor with non-default hash func (sha2-512)', () => {
return ipfs.dag.put(cborNode, {
format: 'dag-cbor',
storeCodec: 'dag-cbor',
hashAlg: 'sha2-512'
})
})

it('should return the cid', async () => {
const cid = await ipfs.dag.put(cborNode, {
format: 'dag-cbor',
storeCodec: 'dag-cbor',
hashAlg: 'sha2-256'
})
expect(cid).to.exist()
Expand All @@ -90,7 +90,7 @@ export function testPut (factory, options) {

it('should override hash algorithm default and resolve with it', async () => {
const cid = await ipfs.dag.put(cborNode, {
format: 'dag-cbor',
storeCodec: 'dag-cbor',
hashAlg: 'sha2-512'
})
expect(cid.code).to.equal(dagCBOR.code)
Expand Down
Loading