Golang tooling for DataBento's APIs and DBN format
This repository contains Golang bindings to Databento's file format Databento Binary Encoding (DBN), Historical API, and Live API. It also includes tools to interact with these services.
NOTE: This is a new library and is under active development. It is not affiliated with Databento. Please be careful with commands which incur billing. We are not responsible for any charges you incur.
To use this library, import the following package, optionally with Databento Historical or Live support:
import (
dbn "github.com/NimbleMarkets/dbn-go"
dbn_hist "github.com/NimbleMarkets/dbn-go/hist"
dbn_live "github.com/NimbleMarkets/dbn-go/live"
)Most dbn-go types and enums parallel DataBento's libraries. Available messages types are:
Mbp0MsgMboMsgMbp1MsgCbboMsgMbp10MsgOhlcvMsgImbalanceMsgSymbolMappingMsgErrorMsgSystemMsgStatMsg
If you want to read a homogeneous array of DBN records from a file, use the dbn.ReadDBNToSlice generic function. The generic argument dicates which message type to read.
file, _ := os.Open("ohlcv-1s.dbn")
defer file.Close()
records, metadata, err := dbn.ReadDBNToSlice[dbn.OhlcvMsg](file)Alternatively, you can use the DBNScanner to read records one-by-one. Each record can be handled directly, or automatically dispatched to the callback method of a struct that implements the dbn.Visitor interface.
dbnFile, _ := os.Open("ohlcv-1s.dbn")
defer dbnFile.Close()
dbnScanner := dbn.NewDbnScanner(dbnFile)
metadata, err := dbnScanner.Metadata()
if err != nil {
return fmt.Errorf("scanner failed to read metadata: %w", err)
}
for dbnScanner.Next() {
header := dbnScanner.GetLastHeader()
fmt.Printf("rtype: %s ts: %d\n", header.RType, header.TsEvent)
// you could get the raw bytes and size:
lastRawByteArray := dbnScanner.GetLastRecord()
lastSize := dbnScanner.GetLastSize()
// you probably want to use this generic helper to crack the message:
ohlcv, err := dbn.DbnScannerDecode[dbn.OhlcvMsg](dbnScanner)
// or if you are handing multiple message types, dispatch a Visitor:
err = dbnScanner.Visit(visitor)
}
if err := dbnScanner.Error(); err != nil && err != io.EOF {
return fmt.Errorf("scanner error: %w", err)
}If you already have DBN-based JSON text files, you can use the generic dbn.ReadJsonToSlice or dbn.JsonScanner to read them in as dbn-go structs. Similar to the raw DBN, you can handle records manually or use the dbn.Visitor interface.
jsonFile, _ := os.Open("ohlcv-1s.dbn.json")
defer jsonFile.Close()
records, metadata, err := dbn.ReadJsonToSlice[dbn.OhlcvMsg](jsonFile)jsonFile, _ := os.Open("ohlcv-1s.dbn.json")
defer jsonFile.Close()
jsonScanner := dbn.NewJsonScanner(jsonFile)
for jsonScanner.Next() {
if err := jsonScanner.Visit(visitor); err != nil {
return fmt.Errorf("visitor err: %w", err)
}
}
if err := jsonScanner.Error(); err != nil {
fmt.Errorf("scanner err: %w", err)
}Many of the dbn-go structs are annotated with json tags to facilitate JSON serialization and deserialization using json.Marshal and json.Unmarshal. That said, dbn-go uses valyala/fastjson and hand-written extraction code.
Support for the Databento Historical API is available in the /hist folder. Every API method is a function that takes an API key and arguments and returns a response struct and error.
databentoApiKey := os.Getenv("DATABENTO_API_KEY")
schemas, err := dbn_hist.ListSchemas(databentoApiKey, "DBEQ.BASIC")
The source for dbn-go-hist illustrates using this dbn_hist module.
Support for the Databento Live API is available in the /live folder.
The general model is:
- Use
dbn_live.NewLiveClientto create aLiveClientbased on aLiveConfigand connect to a DBN gateway client.Authenticatewith the gateway.client.Subscribeto a stream using one or manySubscriptionRequestMsg.- Begin the stream with
client.Start. - Read the stream using
client.GetDbnScanner.
The source for dbn-go-live illustrates using this dbn_live module.
We include some tools to make our lives easier: dbn-go-hist and dbn-go-live. They are available as:
-
Binaries from the
dbn-goreleases page -
Homebrew packages via:
brew install NimbleMarkets/homebrew-tap/dbn-go
-
Docker multi-architecture images on GitHub's Container Registry at
ghcr.io/nimblemarkets/dbn-go:- Hist query:
docker run -e DATABENTO_API_KEY --rm ghcr.io/nimblemarkets/dbn-go:0.0.12 /usr/local/bin/dbn-go-hist datasets - Simple Live feed handler:
docker run -e DATABENTO_API_KEY -v ${pwd}/dbn --rm ghcr.io/nimblemarkets/dbn-go:0.0.12 /usr/local/bin/dbn-go-live -d DBEQ.BASIC -s ohlcv-1h -o /dbn/foo.dbn -v -t QQQ SPY
- Hist query:
-
Built-from-source to the
./binfolder with the commandtask go-build(install Taskfile).
dbn-go-hist is a command-line tool to interact with the Databento Historical API. You can see an example of exercising it in this script file. It requires your Databento API Key to be set with --key or preferably via the DATABENTO_API_KEY environment variable.
CAUTION: This program may incur billing!
$ dbn-go-hist --help
dbn-go-hist queries the DataBento Historical API.
Usage:
dbn-go-hist [command]
Available Commands:
completion Generate the autocompletion script for the specified shell
cost Queries DataBento Hist for the cost and size of a GetRange query
dataset-condition Queries DataBento Hist for condition of a dataset
dataset-range Queries DataBento Hist for date range of a dataset
datasets Queries DataBento Hist for datasets and prints them
fields Queries DataBento Hist for fields of a schema/encoding and prints them
files Lists files for the given DataBento Hist JobID
get-range Download a range of data from the Hist API
help Help about any command
jobs Lists DataBento Hist jobs
publishers Queries DataBento Hist for publishers and prints them
resolve Resolve symbols via the Databento Symbology API
schemas Queries DataBento Hist for schemas and prints them
submit-job Submit a data request job to the Hist API
unit-prices Queries DataBento Hist for unit prices of a dataset
Flags:
-h, --help help for dbn-go-hist
-k, --key string DataBento API key (or use DATABENT_API_KEY envvar)
Use "dbn-go-hist [command] --help" for more information about a command.
Simple invocation:
$ dbn-go-hist datasets
DBEQ.BASIC
GLBX.MDP3
IFEU.IMPACT
NDEX.IMPACT
OPRA.PILLAR
XNAS.ITCHdbn-go-live is a command-line tool to subscribe to a Live DataBento stream and write it to a file. It requires your Databento API Key to be set with --key or preferably via the DATABENTO_API_KEY environment variable.
CAUTION: This program incurs billing!
$ dbn-go-live --help
usage: dbn-go-live -d <dataset> -s <schema> [opts] symbol1 symbol2 ...
-d, --dataset string Dataset to subscribe to
-e, --encoding dbn.Encoding Encoding of the output ('dbn', 'csv', 'json') (default dbn)
-h, --help Show help
-k, --key string Databento API key (or set 'DATABENTO_API_KEY' envvar)
-o, --out string Output filename for DBN stream ('-' for stdout)
-s, --schema stringArray Schema to subscribe to (multiple allowed)
-i, --sin dbn.SType Input SType of the symbols. One of instrument_id, id, instr, raw_symbol, raw, smart, continuous, parent, nasdaq, cms (default raw_symbol)
-t, --start string Start time to request as ISO 8601 format (default: now)
-v, --verbose Verbose logging
Simple invocation:
$ dbn-go-live -d DBEQ.BASIC -s ohlcv-1h -o foo.dbn -v -t QQQ SPY
Simple Docker invocation:
$ docker run -it --rm \
-e DATABENTO_API_KEY \
-v ${pwd}/dbn \
ghcr.io/nimblemarkets/dbn-go:0.0.11 \
/usr/local/bin/dbn-go-live -d DBEQ.BASIC -s ohlcv-1h -o /dbn/foo.dbn -v -t QQQ SPY
We welcome contributions and feedback. Please adhere to our Code of Conduct when engaging our community.
Released under the Apache License, version 2.0, see LICENSE.txt.
Portions adapted from databento/dbn databendo/databento-rs under the same Apache license.
Copyright (c) 2024 Neomantra Corp.
Made with ❤️ and 🔥 by the team behind Nimble.Markets.