Features | Installation | Dependencies | Documentation | Attribution | Contributions | Issues | Acknowledgements
Providing large testing and demo data alongside your package releases is challenging for two reasons. First, code repositories have strict limits on file sizes. Second, you don't want your users to wait forever to download your cool package because you've included large data files. If you're a python developer and have hit these issues then DataRig is for you. DataRig allows you to move data from web-based repositories into your user's local directories post-installation. This "just-in-time" data fetching is perfect for users to test or run your package's demos.
DataRig can be installed into your projects environment using pip:
- Activate the virtual or conda environment of your package
$ source <YOUR_ENV>/bin/activate # python virtual environment$ conda activate <YOUR_ENV>- Install DataRig to your active environment
(<YOUR_ENV>)$ pip install datarigDataRig is super lightweight requiring just Python ≥ 3.9 and the request library available here:
| package | pypi | conda |
|---|---|---|
| requests | https://pypi.org/project/requests/ | ✓ |
Using DataRig to access a repository is simple. Just build a Record instance and all the data will be at your fingertips. Here's how to do it for a sample Zenodo repository:
$ ipython>>> from datarig import Zenodo
>>> # set the url to the api endpoint url for the record id 7868945
>>> url = 'http://zenodo.org/api/records/7868945'
>>> record = Zenodo(url)This record contains all of the repositories information stored as attributes. To see everything at once, just print the record.
>>> print(record)You will see a datasets attribute with a list of Dataset objects. These Datasets contain the name, url link, size and file type of the datasets that can be downloaded from the repository record. Let's print each of them.
>>> for dset in record.datasets:
... print(dset)Notice that a Dataset instance describes the data but does not contain the actual data. To get the data to your machine, you call call the records 'download' method. Let's get help for this method before calling it.
>>> help(record.download)To call this method we need a directory to place the downloaded data, the name of the dataset to download, the amount of memory to use during downloading (chunksize) and a boolean of whether the download should be streamed to disk. Streaming is usually the right choice since the files you will download are likely large. Let's download the "sample_arr.npy" file from this record into your current working dir.
>>> from pathlib import Path
>>> record.download(directory=None, name='sample_arr.npy')That's it! You've just downloaded a dataset from a Zenodo record 😎
If you find DataRig useful, please cite the Zenodo archive of this repository.
If you really like DataRig, you can also star the repository ⭐!
Contributions are what makes open-source fun and we would love for you to contribute. Please check out our contribution guide to get started.
DataRig provides custom issue templates for filing bugs, requesting feature enhancements, suggesting documentation changes, or just asking questions. Ready to discuss? File an issue here.
This work is generously supported through the Ting Tsung and Wei Fong Chao Foundation and the National Institute of Neurological Disorders and Stroke (Grant 2R01 NS100738-05A1).