fix: write to balanced layout at width

[alanshaw]

Fixes a bug that occurs when writing chunks to a balanced layout when the chunks to be written cause the number of leaves to fall on the width boundary.

I was seeing this file link come out when adding a 1GiB file:

{
  cid: CID(bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku),
  contentByteLength: 0,
  dagByteLength: 0
}

When using settings:

const settings = UnixFS.configure({
  fileChunkEncoder: raw,
  smallFileEncoder: raw,
  chunker: withMaxChunkSize(1024 * 1024),
  fileLayout: withWidth(1024),
})

Note that widths that are multiples of the chunk size also exhibit the behaviour e.g. 512, 128 etc. NOT 2048 though as this does not trigger the layout to do any work to rebalance.

For completeness, this is what is expected:

{
  cid: CID(bafybeigpyaqbbvedzatazfw6jf5daalt4tcegmqsmi32nlhdvvrw72ybea),
  contentByteLength: 1073741824,
  dagByteLength: 1073793035
}

Test code:

import fs from 'node:fs'
import { Readable } from 'node:stream'
import * as UnixFS from '@ipld/unixfs'
import * as raw from 'multiformats/codecs/raw'
import { withMaxChunkSize } from '@ipld/unixfs/file/chunker/fixed'
import { withWidth } from '@ipld/unixfs/file/layout/balanced'

const settings = UnixFS.configure({
  fileChunkEncoder: raw,
  smallFileEncoder: raw,
  chunker: withMaxChunkSize(1024 * 1024),
  fileLayout: withWidth(1024),
})

const queuingStrategy = UnixFS.withCapacity()

async function main () {
  console.log('start')
  const stream = Readable.toWeb(fs.createReadStream('./1GiB.bin'))
  const { readable, writable } = new TransformStream(undefined, queuingStrategy)
  ;(async () => {
    await readable.pipeTo(new WritableStream())
  })()
  const writer = UnixFS.createWriter({ writable, settings })
  const file = UnixFS.createFileWriter(writer)
  await stream.pipeTo(new WritableStream({
    async write (chunk) {
      await file.write(chunk)
    }
  }))
  const link = await file.close()
  console.log('file', link)

  const dir = UnixFS.createDirectoryWriter(writer)
  dir.set('1GiB.bin', link)
  const dirLink = await dir.close()
  console.log('dir', dirLink)

  console.log('end')
}

main()

[Gozala]

Thanks for addressing this, do you mind creating test vector with kubo so we can have assurance behavior is the same. Can be a followup or maybe an issue so I’ll get to it eventually

thanks

[alanshaw]

Thanks for looking. I'm going to merge and release so we get a fix available ASAP but I'd appreciate it if you could take another quick look when you're feeling 💯 as I'm not familiar with this code at all...

I'll open an issue for test vectoring and will try and remember to get to it on Monday (I have run out of time today!).