DuckDB HTTP Server Extension

This very experimental extension spawns an HTTP Server from within DuckDB serving query requests.
The extension goal is to replace the functionality currently offered by quackpipe

Features

• Turn any DuckDB instance into an HTTP OLAP API Server
• Use the embedded Play User Interface to query and visualize data
• Pair with chsql extension for ClickHouse flavoured SQL
• Work with local and remote datasets including MotherDuck 🐤
• 100% Opensource, ready to use and extend by the Community!

Extension Functions

• httpserve_start(host, port, auth): starts the server using provided parameters
• httpserve_stop(): stops the server thread

📦 Installation

INSTALL httpserver FROM community;
LOAD httpserver;

🔌 Usage

Start the HTTP server providing the host, port and auth parameters.

• If you want no authentication, just pass an empty string as parameter.
  
• If you want the API run in foreground set DUCKDB_HTTPSERVER_FOREGROUND=1

Basic Auth

D SELECT httpserve_start('localhost', 9999, 'user:pass');

┌───────────────────────────────────────────────┐
│ httpserve_start('0.0.0.0', 9999, 'user:pass') │
│                    varchar                    │
├───────────────────────────────────────────────┤
│ HTTP server started on 0.0.0.0:9999           │
└───────────────────────────────────────────────┘

curl -X POST -d "SELECT 'hello', version()" "http://user:pass@localhost:9999/"

Token Auth

SELECT httpserve_start('localhost', 9999, 'supersecretkey');

┌───────────────────────────────────────────────┐
│ httpserve_start('0.0.0.0', 9999, 'secretkey') │
│                    varchar                    │
├───────────────────────────────────────────────┤
│ HTTP server started on 0.0.0.0:9999           │
└───────────────────────────────────────────────┘

Query your endpoint using the X-API-Key token:

curl -X POST --header "X-API-Key: secretkey" -d "SELECT 'hello', version()" "http://localhost:9999/"

You can perform the same action from DuckDB using HTTP extra_http_headers:

D CREATE SECRET extra_http_headers (
      TYPE HTTP,
      EXTRA_HTTP_HEADERS MAP{
          'X-API-Key': 'secret'
      }
  );

D SELECT * FROM duck_flock('SELECT version()', ['http://localhost:9999']);
┌─────────────┐
│ "version"() │
│   varchar   │
├─────────────┤
│ v1.1.1      │
└─────────────┘

👉 QUERY UI

Browse to your endpoint and use the built-in quackplay interface (experimental)

👉 QUERY API

Query your API endpoint using curl GET/POST requests

curl -X POST -d "SELECT 'hello', version()" "http://localhost:9999/?default_format=JSONCompact

{
  "meta": [
    {
      "name": "'hello'",
      "type": "String"
    },
    {
      "name": "\"version\"()",
      "type": "String"
    }
  ],
  "data": [
    [
      "hello",
      "v1.1.1"
    ]
  ],
  "rows": 1,
  "statistics": {
    "elapsed": 0.01,
    "rows_read": 1,
    "bytes_read": 0
  }
}

👉 CROSS-OVER EXAMPLES

You can now have DuckDB instances query each other and... themselves!

D LOAD json;
D LOAD httpfs;
D SELECT httpserve_start('0.0.0.0', 9999);
┌─────────────────────────────────────┐
│  httpserve_start('0.0.0.0', 9999)   │
│               varchar               │
├─────────────────────────────────────┤
│ HTTP server started on 0.0.0.0:9999 │
└─────────────────────────────────────┘
D SELECT * FROM read_json_auto('http://localhost:9999/?q=SELECT version()');
┌─────────────┐
│ "version"() │
│   varchar   │
├─────────────┤
│ v1.1.1      │
└─────────────┘

Flock Macro by @carlopi

Check out this flocking macro from fellow Italo-Amsterdammer @carlopi @ DuckDB Labs

• a DuckDB CLI, running httpserver extension
• a DuckDB from Python, running httpserver extension
• a DuckDB from the Web, querying all 3 DuckDB at the same time

---

API Documentation

Endpoints Overview

Endpoint	Methods	Description
/	GET, POST	Query API endpoint
/ping	GET	Health check endpoint

Detailed Endpoint Specifications

Query API

Methods: GET, POST

Parameters:

Parameter	Description	Supported Values
default_format	Specifies the output format	JSONEachRow, JSONCompact
query	The DuckDB SQL query to execute	Any valid DuckDB SQL query

Notes

• Ensure that your queries are properly formatted and escaped when sending them as part of the request.
• The root endpoint (/) supports both GET and POST methods, but POST is recommended for complex queries or when the query length exceeds URL length limitations.
• Always specify the default_format parameter to ensure consistent output formatting.

🃏 Disclaimers

Footnotes

1. DuckDB ® is a trademark of DuckDB Foundation. All rights reserved by their respective owners. ¹ ↩

2. ClickHouse ® is a trademark of ClickHouse Inc. No direct affiliation or endorsement. ² ↩

3. Released under the MIT license. See LICENSE for details. All rights reserved by their respective owners. ³ ↩