SurrealDB x OpenAI: A Chat Playground

Welcome to a project that explores the synergy between SurrealDB and OpenAI.

surreal-chat-demo.mp4

Project Overview

This project aims to demonstrate the potential of integrating SurrealDB with OpenAI by importing 25,000 Wikipedia articles, complete with vector embeddings generated by OpenAI. The ultimate goal is to develop a Retrieval-Augmented Generation (RAG) question-answering system.

The backend is powered by a FastAPI server, with Jinja2 handling template rendering and htmx enriching the frontend chat interface.

Getting Started

Prerequisites

1. SurrealDB: Ensure that SurrealDB is installed and configured on your machine. Installation instructions are available here.

2. Python 3.11: We use uv to manage dependencies.

3. SurrealDB Python SDK: The SurrealDB Python SDK is currently in beta. To install the dependency, please ensure you have rust installed on your system.

Installation

1. Clone the repository.

2. You will need an OpenAI API token for this project. If you don't have one, you can obtain it by following the OpenAI Developer Quickstart. Rename the .env.example file to .env and replace the placeholder with your key.

Note: This project is intended for experimental purposes only. Ensure that your OpenAI API key is kept secure.

Setting Up SurrealDB

To initialise SurrealDB, use the following make commands:

1. Start SurrealDB with on-disk persistence:

   make surreal-start

2. Initialise the database with the required tables and functions:

   make surreal-init

3. If you need to reset the database:

   make surreal-remove

Python Environment

With uv installed, run:

uv sync

Activate the Python virtual environment:

source .venv/bin/activate

Downloading the Dataset

Download the Simple English Wikipedia dataset, which includes vector embeddings (~700MB compressed, ~1.7GB CSV):

download-data

Populating SurrealDB

Insert the dataset into SurrealDB:

surreal-insert

Starting the Chat Application

To start the chat server:

make server-start

Support

If you find this project helpful, consider supporting the development: