StreamingFast Storage Abstraction

dstore is a simple abstraction on top of Local storage and Cloud storage. It handles commonly used functions to store things (locally, or on cloud storage providers), list files, delete, etc..

It is used by StreamingFast.

Features

It currently supports:

• AWS S3 (s3://[bucket]/path?region=us-east-1, with AWS-specific env vars)
  • Minio (through the S3 interface)
• Google Storage (gs://[bucket]/path, with GOOGLE_APPLICATION_CREDENTIALS env var set)
• Azure Blob Storage (az://[account].[container]/path, with AZURE_STORAGE_KEY env var set)
• Local file systems (including virtual of fused-based) (file:/// prefix)

Testing

The storetests package contains all our integration tests we perform on our store implementation. Some of the store implementations can be tested directly while few others, from Cloud Providers essentially, requires some extra environment variables to run. They are skip if the correct environment variables for the provider is not set.

To run the full test suite, you will need to perform the following steps.

First, you will need to have locally a few dependencies:

• minio

Then, start minio server:

mkdir -p /tmp/minio-tests/store-tests
cd /tmp/minio-tests
minio server .

Ensure you have access to GCP Storage Bucket, S3 Bucket, then run the full test suite:

STORETESTS_GS_STORE_URL="gs://streamingfast-developement-random/store-tests"\
STORETESTS_S3_STORE_URL="s3://streamingfast-customer-outbox/store-tests?region=us-east-2"\
STORETESTS_S3_MINIO_STORE_URL="s3://localhost:9000/store-tests?region=none&insecure=true&access_key_id=minioadmin&secret_access_key=minioadmin"\
go test ./...

Note S3 bucket is not existing anymore, you need to provide your own.

Contributing

Issues and PR in this repo related strictly to the dstore library.

Report any protocol-specific issues in their respective repositories

Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.

License

Apache 2.0