SplinterDB Integration Investigations

SplinterDB - Postgres Integration

This work was performed in Fall 2022. Following is the Abstract of a paper submitted to VLDB 2023 Industrial Track, but was not accepted for publication. However, the results are encouraging.

---

Relational databases make extensive use of B-tree indexes to accelerate query performance. B-Trees offer excellent point-query performance but inserting data into a large B-tree can be costly, especially for a workload of random inserts. In fact, high-velocity tables are often not indexed at all due to the performance limitations of B-tree insertions.

This paper describes how SplinterDB, a novel, open-source, write-optimized key-value store developed at VMware, can be used as an index within a relational database, such as PostgreSQL, to increase ingestion speed without sacrificing query performance. In our performance evaluation, we show that a SplinterDBindexed table can ingest data up to 24× faster than a B-tree-indexed table. In fact, SplinterDB ingests data half as fast as a table with no index at all, whereas B-tree indexes can impose a 36× slowdown compared to an unindexed table. Furthermore, SplinterDB matched the point-query performance of Postgres’s B-tree and outperformed the builtin B-tree by 50% on large-range queries, although it was about 3× slower on short-range queries.

Our results suggest that a SplinterDB-based index could become a compelling alternative to B-tree indexes for supporting higher ingestion rates like those needed for real-time applications in relational databases like PostgreSQL.

The full technical paper can be found here, SplinterDB-Postgres-vFinal.pdf