Command-line interface (CLI) with plugins

[hammer]

The discussion in sgkit-dev/sgkit-plink#8 made it clear it would be nice to have a CLI with plugins to support IO and other operations.

[hammer]

click-contrib/click-plugins looks to be one way to accomplish this task.

[hammer]

Also it's not necessary but if we are going to invest in the CLI as an interface we may want to make use of a library like willmcgugan/rich to ensure we make effective use of all of the capabilities of the shell/terminal/console.

[jeromekelleher]

Re plugins, I think we should make the interface more tightly defined than plugging in some script to the CLI. Each format plugin should provide:

1. function to convert an input path in that format into an sgkit dataset
2. a function to convert an sgkit dataset into that format and write to a path
3. a function that "sniffs" a path to see if it's a valid file in that format.

The only extra thing we'd need apart from these functions in order to make a functioning CLI like sgkit import would be some way of pulling out the documentation for the extra kwargs on the functions (1) and (2), so that we can push this into the CLI help text.

[eric-czech]

re PLINK conversion CLI:

While these are out of scope for what a simple CLI would likely want to support, I wanted to share the edge cases that came up in importing UKB PLINK for future reference. This could be good "first issue" work in future CLI enhancements:

• Many of the fam fields are irrelevant, so options to choose a projection for conversion would be helpful
• Sample ids happen to be ints rather than strings as usual so it may also be helpful to pair projected fields with a preferred dtype
• The files are split by contig and there is a little bit of extra merging logic necessary to make that work
  • Namely, our contig index will be 0 for all files so that has to be overwritten
• It would be helpful to choose remote destinations for output Zarr (GCS, S3, etc.)
• Compression and filter options would also be good parameters, though hopefully we can standardize those enough that it may not be necessary

My preamble before Zarr writes looks like this:

path = osp.join(data_dir, f'ukb_chr{contig_name}.zarr')
store = gcsfs.GCSMap(path, gcs=gcs, check=False, create=True)
compressor = zarr.Blosc(cname='zstd', clevel=3, shuffle=2)
encoding = {v: {'compressor': compressor} for v in ds}
logger.info(f'Writing dataset for contig {contig} to {path}')
with dask.config.set(scheduler='processes'), ProgressBar():
    ds.to_zarr(store=store, mode='w', consolidated=True, encoding=encoding)

which could be fairly easy to parameterize.

[jeromekelleher]

Resurrecting this, I think a CLI to do basic conversion and dataset inspection/summary would be really handy.

Is anyone dead-against the idea of this, or should I create a batch of issues to track?

[jeromekelleher]

We discussed this on the dev call today, and the consensus was that a basic CLI with to do inspection and data format conversion would be useful. Things have moved on since this issue, so I'm going to close this and open some new ones.