find an alternative to btrfsutil

[sbrudenell]

btrfsutil is not published in pypi, it must be installed from an os repo or compiled from source. this dependency causes some problems:

• virtual envs require --system-site-packages
  • this leads to atypical (thus untested) combinations of package versions. python developers are cavalier about specifying dependency versions. the assumption is that a package is just always installed in a newly created venv, where all the package versions were installed at the same time, and this is the only case tested. I've already run into trouble running tests on ubuntu-20.04.
  • pipenv is overkill to fix this problem. it throws out the baby with the bath water. semantic versioning is sufficient to adopt bug fixes without worrying about incompatibility.
  • the best defense against version chaos is pip install -I, which is easy to forget
• btrfs2s3 can only run on a single python version for a given installation (the one for which btrfsutil is compiled)

some approaches I can think of to work around btrfsutil, in order of preference:

• use ctypes to make ioctl calls directly
  • seems viable after brief research
  • should be future proof given kernel interfaces are required to be stable
  • even avoids dependency on btrfs-progs
  • introduces potential segfaults
  • potentially have to duplicate complex code from libbtrfsutil
• use cffi to call libbtrfsutil
  • seems to require a setup.py in API mode, which needs distribution as binary wheels or source dist
  • says ABI mode is "slow". is this equivalent to using ctypes?
• call btrfs-progs and parse the output
  • the CLI output isn't stable
  • there is an option to output formatted JSON, but it's very new. last I checked it wasn't present in the latest Alpine release
  • we need info (ctime, ctransid) from a large number of subvolumes. I think this would require one subprocess call for each

[sbrudenell]

reading further: it looks like libbtrfsutil is in most cases a really thin wrapper around ioctl.

in a few cases, it uses ioctl to look at some very low level stuff from the btrfs tree.

they have some very strict guidelines that kernel types and constants shouldn't be exposed in libbtrfsutil. they go so far as defining structs in the library interface that are identical to the kernel ones, and the library function just copies data around.

the goal of this isn't clear to me. there's some licensing mumbo jumbo. perhaps they wanted to avoid clients coupling logic to the exact tree structure that can be accessed with ioctl. in any case they do guarantee that versions of kernel and btrfs-progs can be mixed with no trouble, so the kernel interface should be stable as expected, and it should be fine to query the tree the same way the library does.

[sbrudenell]

it looks like btrfs receive parses the stream in user space and invokes the various syscalls. unless we want to re implement the whole thing, we'll still depend on btrfs-progs at runtime, at least for restore.