Internals: introduction

This page is the "Home Page" for documentation about jq internals -- the guts of jq. The primary audience is anyone who wants to contribute to the project.

If you want to learn to use jq, please head to the jq website.

Still here? Right, grab a shovel.

First and foremost, you should learn how the Docs and Tests work. Then, figure out how to

• write built-in functions
• use jv, jq's internal JSON representation
• understand how the interpreter works, and why
• understand the compiler and linker

If you have a shiny new jq feature to add, it is almost certainly better to add it as a builtin function than to add it as new syntax. Builtins have names and you can search the docs for them, syntax is generally difficult to make intuitive.

Plus, if you are adding a new builtin, you don't need to understand how the parser/compiler/interpreter fits together. If you can write your builtin in jq it's a one-line change (plus docs and tests). If it requires new functionality written in C, you'll need to figure out how to use jv first.

Source layout

Here's a rough map of the source (warning: this may go out of date quite quickly):

parser.y, lexer.l

The parser for jq programs. Uses Flex and Bison.

src/jv*.h, src/jv*.c

The internal JSON library. The general API is in src/jv.c, while src/jv_aux.c has some higher-level operations which are implemented in terms of the ones in src/jv.c. For instance, src/jv.c contains jv_object_get and jv_array_get which look up references into objects (hashes) and array, while src/jv_aux.h contains jv_get which handles both arrays and objects, calling the appropriate function for each.

src/jv_parse.c and src/jv_print.c contain the JSON parser and printer, respectively, which jq uses to parse input and render output (these are different from the parser in src/parser.y, which is used to parse jq programs).

src/builtin.jq

This contains the jq-coded builtins.

src/compile.[ch], src/linker.c, src/bytecode.[ch], src/execute.c, and others

The compiler lives in compile.[ch]. These functions are called by the parser to build an internal representation (block form), which gets linked by the linker, and finally gets compiled into bytecode. The bytecode then interpreted by the interpreter, which lives in execute.c.

The interpreter has a supporting cast that handle various runtime aspects of jq, including forkable_stack.h (the guts of jq's stack, used to implement backtracking) and builtin.h (which defines built-in functions).

src/main.c

Handles command-line arguments, stdin, stdout, and not much else.