feat(datasource): add support for cassandra/scylla

[tempbottle]

Description

As title.

[tychoish]

This looks cool! Would love to hear more about what you're trying to build and how you expect to use Cassandra/Scylla data in GlareDB particularly with regards to handling the schema translation. Also it would be cool to know more about the environment that you're working in so we can make sure we test it appropriately.

Thanks for your interest and attention!

[tempbottle]

We just have many big tables in cassandra or may migrate to scylladb.
And we want to analyze some data with cassandra tables and mysql tables.
There are too many join logic from the two datasource. And the implementation of join is by hand write code in java application. But this java code look like very easier to implement by SQL.

[tychoish]

Awesome this makes sense!

Which version of Cassandra are you using?

Would you be interested in/able to? talking/corresponding more with us next week about the queries you'd need to run on the Cassandra side?

There are a few parts of any integration between GlareDB and external data sources:

• translating Cassandra data types and structures into GlareDB and connecting to Cassandra servers.
• translating SQL queries to Cassandra (CQL) queries so that GlareDB can push down queries to Cassandra (typically we do this just for projections and simple predicates at first)
• writing data to the external data source.

I trust we can drop the write path (for now, certainly), and the first part could be relatively straightforward, but we have some opportunity to control the scope for the second, which could help us get something to you sooner.

[universalmind303]

initial functionality is implemented in #2344

follow up issues

• feat(cassandra): EXTERNAL DATABASE for cassandra #2346
• feat(cassandra): EXTERNAL TABLE support for cassandra #2345
• cassandra: support additional dtypes #2347
• cassandra: support write operations #2348
• cassandra: support predicate pushdowns #2349

[tempbottle]

Awesome this makes sense!

Which version of Cassandra are you using?

Would you be interested in/able to? talking/corresponding more with us next week about the queries you'd need to run on the Cassandra side?

There are a few parts of any integration between GlareDB and external data sources:

* translating Cassandra data types and structures into GlareDB and connecting to Cassandra servers.

* translating SQL queries to Cassandra (CQL) queries so that GlareDB can push down queries to Cassandra (typically we do this just for projections and simple predicates at first)

* writing data to the external data source.

I trust we can drop the write path (for now, certainly), and the first part could be relatively straightforward, but we have some opportunity to control the scope for the second, which could help us get something to you sooner.

Sorry to reply slowly.
Thanks for your attention.
We have two main Cassandra versions(3.9 and 4.1) in the data center. And we plan to migrate to 4.1 in the future.
We don't have very complex SQL in Cassandra now. There are mostly simple query like this:

Select * from table1 where user_city="xxxxxx" and regist_date<="20231222" and regist_date>="20231001" and last_logon_date>"20231210"

Hope this info is helpful.

[tychoish]

@tempbottle, just wanted to follow up with the current state of this:

Cassnadra/scylla datasources (tables and entire databases) exists in the codebase today, and in the latest release. However, the initial implementation missed authentication, and we have a PR open to add that, which should be in by the end of the week and we can cut a point-release with this too.

Wanted to give you an update and see if the auth-free example works locally, and also that we'll have a more complete version out really soon.

(Writes and more pushdowns coming soon, particularly if, from a high level what we have so far works for you.)

Cheers,
sam

[greyscaled]

Adding to this, we have a few examples of what's possible in our release notes: https://github.com/GlareDB/glaredb/releases/tag/v0.8.0

Cheers,
Grey