Node.js client library for Apache Pinot 🍷

JavaScript client library for connecting to and querying Apache Pinot 🍷, a realtime distributed OLAP datastore.

GitHub Repo, TypeDoc Reference, npm Package

Features

• Implements a controller-based broker selector that periodically updates the table-to-broker mapping via the controller API.
• Provides TypeScript definitions of broker responses (Pinot query results).
• Has 100% test coverage.

Quick start

Start a development Pinot cluster with baseballStats demo data:

docker run \
    -p 9000:9000 -p 8000:8000 \
    apachepinot/pinot:0.9.3 QuickStart \
    -type batch

Installation

npm install pinot-client

or

yarn add pinot-client

Creating a connection to a Pinot cluster

With a simple broker selector (that chooses randomly between the provided brokers upon each query):

import { ConnectionFactory } from "pinot-client";

const connection = ConnectionFactory.fromHostList(["localhost:8000"]);

With a controller-based broker selector (that maintains a periodically updated table-to-broker mapping obtained via controller API):

import { ConnectionFactory } from "pinot-client";

const connection = await ConnectionFactory.fromController("localhost:9000");

Querying Pinot

const r = await connection.execute(
    "baseballStats", // table name
    "select league, sum(hits) as hits from baseballStats group by league order by hits desc" // SQL query
);

console.log(`Scanned ${r.numDocsScanned} docs in ${r.timeUsedMs}ms`);
console.log("Results:");
console.log(r.resultTable.dataSchema.columnNames.join("\t"));
r.resultTable.rows.forEach((row) => {
    console.log(row.join("\t"));
});

Output:

Scanned 97889 docs in 8ms
league  hits
NL      1890198
AL      1650039
AA      88730
NA      24549
FL      21105
PL      10523
UA      7457

Usage

fromHostList() and fromController() options

ConnectionFactory.fromHostList() may optionally take as a second parameter an object with the following keys:

• logger: a logger instance conforming to the standard Log4j interface w/ .child() method (i.e. pino, winston or log4js)
• brokerReqHeaders: additional HTTP headers (object key: value) to include in broker query API requests
• customHttpClient: a custom HTTP client implementation

on top of that, ConnectionFactory.fromController() options may include two additional keys:

• controllerReqHeaders: additional HTTP headers (object key: value) to include in controller API requests
• brokerUpdateFreqMs: wait time in milliseconds between table-to-broker mapping refreshes

Example usage:

const options = {
    brokerReqHeaders: {
        Authorization: "Basic asdf123",
    },
    controllerReqHeaders: {
        Authorization: "Basic xyz123",
    },
    brokerUpdateFreqMs: 500,
};

const connection = await ConnectionFactory.fromController("localhost:9000", options);

Using a custom logger

// let's use pino (not to be confused with pinot) as an example logger
import * as pino from "pino";

const pinoInstance = pino({ level: "debug" });
const childLogger = pinoInstance.child({ lib: "pinot-client" });

const options = {
    logger: childLogger,
};

const connection = await ConnectionFactory.fromController("localhost:9000", options);

Using a custom HTTP client

By default pinot-client uses undici for performing HTTP requests against Pinot Borkers and Controllers. A custom HttpClient interface implementation (containing POST and GET methods) can be provided instead via the customHttpClient options key:

const myClient: HttpClient = {
    get: async function <T>(url: string, options: { headers: Record<string, string> }) {
        const { statusCode, parsedBody } = await otherHTTPClientLib<T>(url, ...);
        // data is of type T
        return { status: statusCode, data: parsedBody };
    },
    post: async function <T>(url: string, data: object, options: { headers: Record<string, string> }) {
        const { statusCode, parsedBody } = await otherHTTPClientLib<T>(url, ...);
        // data is of type T
        return { status: statusCode, data: parsedBody };
    },
};

const options = {
    customHttpClient: myClient,
}

const c = await ConnectionFactory.fromController("http://localhost:9000", options);