Skip to content

FPGA-Research-Manchester/OrkhestraFPGAStream

Repository files navigation

OrkhestraFPGAStream - Accelerated database operations on an FPGA with a dynamic stream processing interface

License Build & Test Standard

OrkhestraFPGAStream is a middleware for orchestrating a dynamic stream processing pipeline. Currently, it only supports DBMS data streams with DBMStoDSPI, which uses FPGA modules with our Dynamic Stream Processing Interface (DSPI). This software stack is meant to be coupled with a SQL parser or a DBMS. Then the software stack would execute the given query plan with an available FPGA using the available accelerator module library. How the data flows through the FPGA stack is shown in the graph below:

DBMStoDSPI source layout

As the image shows, the input would be a query plan graph from a DBMS to the software stack, which will use FOS and the modules described below to accelerate the query. The image is slightly out of date, but it gives a rough overview.

Currently supported data types:

  • 32 bit integer
  • Fixed length string
  • SQL Date
  • 64 bit SQL Decimal(15,2)
  • Raw binary data
  • NULL

Currently supported modules:

The DMA is a compulsory module which will set up the interface. The sort modules are meant to be used together. The join module is for sort-merge join operations. So it will only join sorted streams. The addition, multiplication and sum modules work with 64 bit decimal values.

The interface specification is better described in the thesis by Kristiyan Manev.

How does it work?

To run the program use the following command:

dbmstodspi -i INPUT_DEF.json -c CONFIG.ini

The first file defines the query plan. The second file contains json filepaths to define which bitstreams are available with their respective constraints. Examples can be found here:

For the interactive mode for demos or debugging use the -x flag without the -i flag and the INPUT_DEF file.

More info on the how the data is mapped to the interface and how the data is processed can be found here.

How to use the source?

The documentation for this project is created using CMake with Doxygen. To turn the documentation build option on you need to change the main CMakeLists.txt file and then you can find the output in the build directory under the doc_doxygen folder. The FOS library and test library aren't included in the generated documentation.

Formatting is done according to the clang-tidy options you can find here.

Testing for this code is done using googletest and is automatically downloaded when the code is built with testing options turned on.

Quick steps to running the benchmark from zero:

  • Get a ZCU102 SD card image here or build it yourself
  • Install the OS to an 16GB SD card. Further explanation here or just use the following command:
dd if=<fos_image.img> of=<SD card image> bs=4M status=progress

About

Private repository for accelerating DBMS SQL queries with FPGAs

Topics

Resources

License

Stars

Watchers

Forks