Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: hoist node.js bindings on top of rust sdk #2879

Merged
merged 59 commits into from
Apr 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
3a9352d
chore: rename glaredb -> cli crate
tychoish Mar 20, 2024
0cc08ec
chore: engine constructor
tychoish Mar 26, 2024
401d74e
feat: high level glaredb rust api/sdk
tychoish Mar 21, 2024
379f300
chore: refactor connection, create driver connect
tychoish Mar 27, 2024
f40c9df
Merge remote-tracking branch 'origin/main' into tycho/topic/sdk/0/ren…
tychoish Mar 28, 2024
280b9ce
Merge branch 'tycho/topic/sdk/0/rename-cli-package' into tycho/topic/…
tychoish Mar 28, 2024
64c96cf
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Mar 28, 2024
daa757d
wip
tychoish Mar 28, 2024
ae73d45
fix: merge related build failure
tychoish Mar 28, 2024
7a366fd
Merge branch 'tycho/topic/sdk/0/rename-cli-package' into tycho/topic/…
tychoish Mar 28, 2024
b92c760
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Mar 28, 2024
115d5ef
feat: implementation of the connection methods
tychoish Mar 28, 2024
7e14f4d
fix: unittest build
tychoish Mar 28, 2024
e150dc7
fix: explicit reference
tychoish Mar 28, 2024
c40fc24
Merge branch 'tycho/topic/sdk/0/rename-cli-package' into tycho/topic/…
tychoish Mar 28, 2024
1017c59
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Mar 28, 2024
c101221
fix: binary resolution
tychoish Mar 29, 2024
e1495b2
Merge branch 'tycho/topic/sdk/0/rename-cli-package' into tycho/topic/…
tychoish Mar 29, 2024
854bd8d
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Mar 29, 2024
592bda2
fix: python binding error handling
tychoish Mar 29, 2024
50070c2
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Mar 29, 2024
90f783d
fix: lint error
tychoish Mar 29, 2024
9486f21
feat: prql helpers and sync stream producers
tychoish Mar 29, 2024
f143eea
feat: addtional converter
tychoish Mar 29, 2024
a34f5db
chore: reorganization
tychoish Mar 29, 2024
6439f7e
fix: lint
tychoish Mar 29, 2024
851d7cd
Merge remote-tracking branch 'origin/main' into tycho/topic/sdk/0/ren…
tychoish Apr 2, 2024
a759963
Merge branch 'tycho/topic/sdk/0/rename-cli-package' into tycho/topic/…
tychoish Apr 3, 2024
c92cebd
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Apr 3, 2024
14c7a61
chore: backport driver changes from python integration branch
tychoish Apr 5, 2024
74ef043
fix: backport additional features from binding integration
tychoish Apr 5, 2024
b47b1a6
Revert "chore: backport driver changes from python integration branch"
tychoish Apr 5, 2024
cbeef58
chore: backport from driver branch
tychoish Apr 5, 2024
b994e35
chore: port in sqlexec changes
tychoish Apr 5, 2024
7bebf63
chore: backport crates changes
tychoish Apr 5, 2024
1cfcbad
Merge branch 'main' into tycho/topic/sdk/1/engine-constructor
tychoish Apr 5, 2024
5d46218
fix: comple
tychoish Apr 7, 2024
bd9f621
THIS IS AN EMPTY COMMIT
tychoish Apr 7, 2024
92fa346
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Apr 7, 2024
b9b52b6
fix: non-deterministic http test
tychoish Apr 7, 2024
3402ab3
Merge branch 'tycho/fix-flaky-test' into tycho/topic/sdk/1/engine-con…
tychoish Apr 7, 2024
2903c1b
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Apr 7, 2024
0823e3d
Merge branch 'main' into tycho/topic/sdk/1/engine-constructor
tychoish Apr 7, 2024
28b15a5
Merge branch 'tycho/topic/sdk/1/engine-constructor' into tycho/topic/…
tychoish Apr 7, 2024
a65a831
feat: documentation for Rust SDK crate
tychoish Apr 8, 2024
d0e8a35
chore: hoist nodejs onto rust sdk
tychoish Apr 9, 2024
829c004
Merge remote-tracking branch 'origin/main' into tycho/topic/sdk/7/nod…
tychoish Apr 11, 2024
7a1d83b
chore: clean up integration
tychoish Apr 11, 2024
943af80
build update
tychoish Apr 11, 2024
06faa49
fix: lazy sdk implementation
tychoish Apr 12, 2024
dc16531
Merge branch 'tycho/topic/sdk/8/operation-interface' into tycho/topic…
tychoish Apr 12, 2024
ef89280
Merge remote-tracking branch 'origin/main' into tycho/topic/sdk/7/nod…
tychoish Apr 12, 2024
d0e696c
fix: cleanup calling
tychoish Apr 12, 2024
faeca7c
Merge remote-tracking branch 'origin/main' into tycho/topic/sdk/8/ope…
tychoish Apr 12, 2024
0ecc883
backport method naming
tychoish Apr 12, 2024
1a306e2
copypasta
tychoish Apr 12, 2024
c0a2429
Merge branch 'tycho/topic/sdk/8/operation-interface' into tycho/topic…
tychoish Apr 12, 2024
3f472a5
fix type names
tychoish Apr 12, 2024
6c71863
Merge branch 'main' into tycho/topic/sdk/7/nodejs-integration
tychoish Apr 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 7 additions & 5 deletions bindings/nodejs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,15 @@ crate-type = ["cdylib"]
workspace = true

[dependencies]
arrow_util = { path = "../../crates/arrow_util" }
datafusion_ext = { path = "../../crates/datafusion_ext" }
glaredb = { path = "../../crates/glaredb" }
ioutil = { path = "../../crates/ioutil" }
sqlexec = { path = "../../crates/sqlexec" }
metastore = { path = "../../crates/metastore" }
telemetry = { path = "../../crates/telemetry" }
pgsrv = { path = "../../crates/pgsrv" }
pgrepr = { path = "../../crates/pgrepr" }
datafusion_ext = { path = "../../crates/datafusion_ext" }
arrow_util = { path = "../../crates/arrow_util" }
pgsrv = { path = "../../crates/pgsrv" }
sqlexec = { path = "../../crates/sqlexec" }
telemetry = { path = "../../crates/telemetry" }
terminal_util = { path = "../../crates/terminal_util" }
futures = { workspace = true }
datafusion = { workspace = true }
Expand All @@ -32,6 +33,7 @@ napi = { version = "2.16.2", default-features = false, features = ["full"] }
napi-derive = "2.16.2"
once_cell = "1.19.0"
bytes = "1.6.0"
async-once-cell = "0.5.3"

[build-dependencies]
napi-build = "2.1.2"
2 changes: 1 addition & 1 deletion bindings/nodejs/glaredb.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ const glaredb = require('./index.js')
// Some of methods can't be performed through `n-api`
// So we need to monkey patch them here
// The methods should still be defined in rust so we can keep a consistent `index.d.ts` file.
Object.assign(glaredb.JsLogicalPlan.prototype, {
Object.assign(glaredb.JsExecutionOutput.prototype, {
async toPolars() {
try {
const pl = require("nodejs-polars")
Expand Down
16 changes: 8 additions & 8 deletions bindings/nodejs/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ export function connect(dataDirOrCloudUrl?: string | undefined | null, options?:
/** A connected session to a GlareDB database. */
export class Connection {
/**
* Returns a default connection to the global in-memory database.
* Returns the default connection to a global in-memory database.
*
* The database is only initialized once, and all subsequent calls
* will return the same connection, and therefore have access to the
* same data and database.
* The database is only initialized once, and all subsequent
* calls will return the same connection object and therefore
* have access to the same data.
*/
static defaultInMemory(): Promise<Connection>
/**
Expand Down Expand Up @@ -61,7 +61,7 @@ export class Connection {
* await con.sql('create table my_table (a int)').then(cursor => cursor.execute())
* ```
*/
sql(query: string): Promise<JsLogicalPlan>
sql(query: string): Promise<JsExecutionOutput>
/**
* Run a PRQL query against a GlareDB database. Does not change
* the state or dialect of the connection object.
Expand All @@ -70,14 +70,14 @@ export class Connection {
* import glaredb from "@glaredb/glaredb"
*
* let con = glaredb.connect()
* let cursor = await con.sql('from my_table | take 1');
* let cursor = await con.prql('from my_table | take 1');
* await cursor.show()
* ```
*
* All operations execute lazily when their results are
* processed.
*/
prql(query: string): Promise<JsLogicalPlan>
prql(query: string): Promise<JsExecutionOutput>
/**
* Execute a query.
*
Expand All @@ -96,7 +96,7 @@ export class Connection {
/** Close the current session. */
close(): Promise<void>
}
export class JsLogicalPlan {
export class JsExecutionOutput {
toString(): string
show(): Promise<void>
execute(): Promise<void>
Expand Down
4 changes: 2 additions & 2 deletions bindings/nodejs/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -252,8 +252,8 @@ if (!nativeBinding) {
throw new Error(`Failed to load native binding`)
}

const { connect, Connection, JsLogicalPlan } = nativeBinding
const { connect, Connection, JsExecutionOutput } = nativeBinding

module.exports.connect = connect
module.exports.Connection = Connection
module.exports.JsLogicalPlan = JsLogicalPlan
module.exports.JsExecutionOutput = JsExecutionOutput
60 changes: 29 additions & 31 deletions bindings/nodejs/src/connect.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,12 @@
//! queries.

use std::collections::HashMap;
use std::sync::Arc;

use sqlexec::remote::client::RemoteClientType;

use crate::connection::Connection;
use crate::error::JsGlareDbError;

#[napi(object)]
#[derive(Default)]
Expand All @@ -17,44 +21,38 @@ pub struct ConnectOptions {
pub storage_options: Option<HashMap<String, String>>,
}

impl ConnectOptions {
fn spill_path(&mut self) -> Option<String> {
self.spill_path.take()
}

fn disable_tls(&self) -> bool {
self.disable_tls.unwrap_or(false)
}

fn cloud_addr(&self) -> String {
self.cloud_addr
.clone()
.unwrap_or(String::from("https://console.glaredb.com"))
}

fn location(&mut self) -> Option<String> {
self.location.take()
}

fn storage_options(&mut self) -> Option<HashMap<String, String>> {
self.storage_options.take()
impl TryFrom<ConnectOptions> for glaredb::ConnectOptions {
type Error = glaredb::ConnectOptionsBuilderError;

fn try_from(
val: ConnectOptions,
) -> Result<glaredb::ConnectOptions, glaredb::ConnectOptionsBuilderError> {
glaredb::ConnectOptionsBuilder::default()
.spill_path(val.spill_path)
.disable_tls_opt(val.disable_tls)
.cloud_addr_opt(val.cloud_addr)
.location(val.location)
.storage_options_opt(val.storage_options)
.client_type(RemoteClientType::Node)
.build()
}
}


/// Connect to a GlareDB database.
#[napi(catch_unwind)]
pub async fn connect(
data_dir_or_cloud_url: Option<String>,
options: Option<ConnectOptions>,
) -> napi::Result<Connection> {
let mut options = options.unwrap_or_default();
Connection::connect(
data_dir_or_cloud_url,
options.spill_path(),
options.disable_tls(),
options.cloud_addr(),
options.location(),
options.storage_options(),
)
.await
let mut options: glaredb::ConnectOptions = options
.unwrap_or_default()
.try_into()
.map_err(JsGlareDbError::from)?;

options.connection_target = data_dir_or_cloud_url;

Ok(Connection {
inner: Arc::new(options.connect().await.map_err(JsGlareDbError::from)?),
})
}
Loading