fast-http

This is a fast HTTP request/response protocol parser for Common Lisp.

Features

• Parses both HTTP requests and responses.
• Handles persistent streams (keep-alive).
• Decodes chunked encoding.
• Exports low-level APIs which don't any memory allocations during parsing.

API differences from http-parse

The API is quite similar to http-parse, although there's some differences.

• http, http-request and http-response are structure classes, not standard classes.
• http doesn't have :force-stream option. (always streaming)
• http doesn't have :store-body option because it can consume much memory.
• body-callback for make-parser doesn't take a flag body-complete-p.
  • Use finish-callback to know if the parsing is finished.
• body-callback for make-parser takes pointers start and end.
• multipart-callback for make-parser has been deleted.
  • Use make-multipart-parser and body-callback by yourself.
• :callback of make-multipart-parser takes a stream, not a body octet vector at the 4th argument.
• Raises errors aggressively while parsing.
  • Handle fast-http-error as you needed.
• Doesn't use a property list as a representation of HTTP headers. (See issue #1)

APIs

[Structure] http

Base structure class extended by http-request and http-response.

NOTE: Don't use this class directly unless you're intended to use low-level APIs of fast-http.

(make-http)
;=> #S(FAST-HTTP.HTTP:HTTP
;     :METHOD NIL
;     :MAJOR-VERSION 0
;     :MINOR-VERSION 9
;     :STATUS 0
;     :CONTENT-LENGTH NIL
;     :CHUNKED-P NIL
;     :UPGRADE-P NIL
;     :HEADERS NIL
;     :HEADER-READ 0
;     :MARK -1
;     :STATE 0)

Methods

• http-method: Returns a HTTP request method in a keyword (such like :GET, :POST or :HEAD).
• http-major-version: Returns a HTTP protocol major version in an integer (such like 1 or 0).
• http-minor-version: Returns a HTTP protocol minor version in an integer (such like 1 or 0).
• http-version: Returns a HTTP protocol version in a float (such like 1.0 or 1.1).
• http-status: Returns a HTTP response status code in an integer (such like 200 or 302).
• http-content-length: Returns a value of Content-Length header in an integer. If the header doesn't exist, it returns NIL.
• http-chunked-p: Returns T if the value of Transfer-Encoding header is chunked. If the header doesn't exist, it returns NIL.
• http-upgrade-p: Returns T if Upgrade header exists.
• http-headers: Returns a hash-table which represents HTTP headers. Note all hash keys are lower-cased and all values are string except Set-Cookie header, whose value is a list of strings. (Content-Length -> "content-length").

[Structure] http-request (extends http)

Structure class holds values specific to an HTTP request.

(make-http-request)
;=> #S(FAST-HTTP.HTTP:HTTP-REQUEST
;     :METHOD NIL
;     :MAJOR-VERSION 0
;     :MINOR-VERSION 9
;     :STATUS 0
;     :CONTENT-LENGTH NIL
;     :CHUNKED-P NIL
;     :UPGRADE-P NIL
;     :HEADERS NIL
;     :HEADER-READ 0
;     :MARK -1
;     :STATE 0
;     :RESOURCE NIL)

Methods

• http-resource: Returns an URI string.

[Structure] http-response (extends http)

Structure class holds values specific to an HTTP response.

(make-http-response)
;=> #S(FAST-HTTP.HTTP:HTTP-RESPONSE
;     :METHOD NIL
;     :MAJOR-VERSION 0
;     :MINOR-VERSION 9
;     :STATUS 0
;     :CONTENT-LENGTH NIL
;     :CHUNKED-P NIL
;     :UPGRADE-P NIL
;     :HEADERS NIL
;     :HEADER-READ 0
;     :MARK -1
;     :STATE 0
;     :STATUS-TEXT NIL)

Methods

• http-status-text: Returns an response status text (such like Continue, OK or Bad Request).

[Function] make-parser (http &key first-line-callback header-callback body-callback finish-callback)

Makes a parser closure and returns it.

(let ((http (make-http-request)))
  (make-parser http
               :body-callback
               (lambda (data start end)
                 (write-to-buffer data start end))
               :finish-callback
               (lambda ()
                 (handle-response http))))
;=> #<CLOSURE (LAMBDA (DATA &KEY (START 0) END)
;              :IN
;              FAST-HTTP.PARSER:MAKE-PARSER) {10090BDD0B}>

The closure takes one required argument data, that is a simple byte vector and two keyword arguments start and end.

Callbacks

• first-line-callback (): This callback function will be called when the first line is parsed.
• header-callback (headers-hash-table): This callback function will be called when the header lines are parsed. This function is the same object to the http object holds.
• body-callback (data-byte-vector): This callback function will be called whenever it gets a chunk of HTTP body. Which means this can be called multiple times.
• finish-callback (): This callback function will be called when the HTTP message ends.

NOTE: If the HTTP request/response has multiple messages (like HTTP/1.1 pipelining), all these functions can be called multiple times.

[Function] make-multipart-parser (content-type callback)

Makes a multipart/form-data parser closure and returns it.

This takes 2 arguments, content-type (such like "multipart/form-data; boundary=--AsB03x") and callback. The callback is a function which takes exact 4 arguments -- a field name, field headers, field meta data and body bytes.

Low-level APIs

The following functions are intended to be used for internally. These APIs are likely to change in the future.

Most of functions are declared as (optimize (speed 3) (safety 0)) which means it won't check the type of arguments.

[Structure] callbacks

Structure class holds callback functions. The callbacks are similar to make-parser's, but don't correspond to them directly.

Slots

• message-begin (http): This will be called when a new HTTP message begins.
• url (http data start end): This will be called when an URL part of the HTTP request parsed.
• first-line (http): This will be called when the first line of the HTTP request/response parsed.
• status (http data start end): This will be called when the status text (not code) of the HTTP response parsed.
• header-field (http data start end): This will be called when a header field parsed.
• header-value (http data start end): This will be called when a header value parsed. This function can be called multiple times when the header value is folded onto multiple lines.
• headers-complete (http): This will be called when all headers parsed.
• body (http data start end): This will be called whenever the parser gets a chunk of HTTP body.
• message-complete (http): This will be called when the HTTP message ends.

[Function] parse-request (http callbacks data &key (start 0) end)

Parses data as an HTTP request, sets values to http and invokes callbacks in callbacks.

This takes a http object, a callbacks object, and a simple byte vector data and two pointers -- start and end. If end is nil, the length of data will be used.

[Function] parse-response (http callbacks data &key (start 0) end)

Parses data as an HTTP response, sets values to http and invokes callbacks in callbacks.

Takes a http object, a callbacks object, and a simple byte vector data and two pointers -- start and end. If end is nil, the length of data will be used.

[Condition] eof

Will be raised when the data ends in the middle of parsing.

Installation

(ql:quickload :fast-http)

Running tests

(asdf:test-system :fast-http)

Benchmark

• Parsing an HTTP request header 100000 times.

In this benchmark, fast-http is 1.25 times faster than http-parser, a C equivalent.

http-parser (C)	fast-http
0.108s	0.086s

Environment

• Travis CI
• SBCL 1.2.6

You can see the latest result at Travis CI.

fast-http (Common Lisp)

(ql:quickload :fast-http-test)
(fast-http-test.benchmark:run-ll-benchmark)

Evaluation took:
  0.086 seconds of real time
  0.085897 seconds of total run time (0.084763 user, 0.001134 system)
  100.00% CPU
  257,140,751 processor cycles
  0 bytes consed

http-parser (C)

#include "http_parser.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>
#include <time.h>

static http_parser *parser;

static http_parser_settings settings_null =
  {.on_message_begin = 0
  ,.on_header_field = 0
  ,.on_header_value = 0
  ,.on_url = 0
  ,.on_status = 0
  ,.on_body = 0
  ,.on_headers_complete = 0
  ,.on_message_complete = 0
  };

int
main (void)
{
  const char *buf;
  int i;
  float start, end;
  size_t parsed;

  parser = malloc(sizeof(http_parser));

  buf = "GET /cookies HTTP/1.1\r\nHost: 127.0.0.1:8090\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17\r\nAccept-Encoding: gzip,deflate,sdch\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\nCookie: name=wookie\r\n\r\n";

  start = (float)clock()/CLOCKS_PER_SEC;
  for (i = 0; i < 100000; i++) {
    http_parser_init(parser, HTTP_REQUEST);
    parsed = http_parser_execute(parser, &settings_null, buf, strlen(buf));
  }
  end = (float)clock()/CLOCKS_PER_SEC;

  free(parser);
  parser = NULL;

  printf("Elapsed %f seconds.\n", (end - start));

  return 0;
}

$ make http_parser.o
$ gcc -Wall -Wextra -Werror -Wno-error=unused-but-set-variable -O3 http_parser.o mybench.c -o mybench
$ mybench
Elapsed 0.108815 seconds.

Author

• Eitaro Fukamachi (e.arrows@gmail.com)

Copyright

Copyright (c) 2014 Eitaro Fukamachi

License

Licensed under the MIT License.