Manage large vectors of bits types in Julia. A thin wrapper for mmapped binary data, with a few sanity checks and convenience functions.
For each dataset, the columns (vectors of equal length) and metadata are stored in a directory like this:
dir/
├── layout.jld2
├── meta/
│ └ ...
├── 1.bin
├── 2.bin
├── ...
├── ...
└── ...
The file layout.jld2 specifies the number and types of columns (using
JLD2.jl, and the total number of
elements. The $i.bin files contain the data for each column, which
can be memory mapped.
Additional metadata can be saved as in files in the directory
meta. This is ignored by this library; use the function meta_path
to calculate paths relative to dir/meta.
Two interfaces are provided. Use SinkColumns for an ex ante
unknown number of elements, written sequentially. This is useful for
ingesting data.
MmappedColumns is useful when the number of records is known and
fixed.
Types for the columns are specified as Tuples. See the docstrings
for both interfaces and the unit tests for examples.
Work on this library was supported by the Austrian National Bank Jubiläumsfonds grant #17378.