Skip to content

Commit

Permalink
uri: Add URI API documentation and improve README
Browse files Browse the repository at this point in the history
  • Loading branch information
@resilar
resilar committed Apr 24, 2019
1 parent 9de5a14 commit 3fc3fc4
Showing 1 changed file with 147 additions and 82 deletions.
229 changes: 147 additions & 82 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@
- [Compiling](#compiling)
- [Cryptography buzzwords](#cryptography-buzzwords)
- [Example](#example)
- [SQLite3 encryption API](#sqlite3-encryption-api)
- [Android support](#android-support)
- [Remarks](#remarks)
- [Library API](#library-api)
- [C programming interface](#c-programming-interface)
- [URI configuration interface](#uri-configuration-interface)
- [Android/iOS support](#android-ios-support)
- [Versioning scheme](#versioning-scheme)
- [License](#license)

Expand All @@ -23,33 +24,33 @@ SQLite3 shell with sqleet encryption support can be compiled as follows:
% gcc sqleet.c shell.c -o sqleet
```

[Example](#example) illustrates sqleet encryption using the compiled shell. The
C interface of the sqleet encryption extension is described in section [SQLite3
encryption API](#sqlite3-encryption-api).
[Example](#example) illustrates the sqleet encryption using the compiled shell.
[Library API](#library-api) consists of [C programming
interface](#c-programming-interface) and [URI-based configuration
interface](#uri-configuration-interface).

To use sqleet as a library, the recommended way is to download a preconfigured
[release package](https://github.com/resilar/sqleet/releases/latest) instead of
cloning the source repository. Contained `sqleet.c` and `sqleet.h` files are
drop-in replacements for the official `sqlite3.c` amalgamation and `sqlite3.h`
header. Alternatively, `sqleet.c` and `sqleet.h` of the source repository can
be used directly if all sqleet source files are available at compile time.
drop-in replacements for the original `sqlite3.c` amalgamation and `sqlite3.h`
header. Alternatively, `sqleet.c` and `sqleet.h` of the source repository can be
used directly if all sqleet source files are available at compile time.

To produce a custom release version of sqleet, run `./script/amalgamate.sh
<sqleet.c >release.c` to create amalgamation of SQLite3 with sqleet encryption
support. Likewise, `./script/amalgamate.sh <sqleet.h >release.h` to amalgamate
the header.
<sqleet.c >release.c` to create an amalgamation of SQLite3 with sqleet
encryption support. Similarly, run `./script/amalgamate.sh <sqleet.h >release.h`
to amalgamate the header.


Cryptography buzzwords
----------------------

- PBKDF2-HMAC-SHA256 key derivation algorithm with a 16-byte random salt and
12345 iterations.
- PBKDF2-HMAC-SHA256 key derivation with a 16-byte salt and 12345 iterations.
- ChaCha20 stream cipher with one-time keys.
- Poly1305 authentication tags.

A low-level description of the encryption scheme is available in
[sqleet.c:145](sqleet.c#L145).
[sqleet.c:258](sqleet.c#L258).


Example
Expand All @@ -58,35 +59,36 @@ Example
Encrypting a database with a password "swordfish".

```
[sqleet]% hexdump -C hello.db
[sqleet]% hexdump -C hello.db
00000000 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 |SQLite format 3.|
00000010 10 00 01 01 00 40 20 20 00 00 00 02 00 00 00 02 |.....@ ........|
00000010 10 00 01 01 00 40 20 20 00 00 00 01 00 00 00 02 |.....@ ........|
*
00000fd0 00 00 00 2b 01 06 17 17 17 01 37 74 61 62 6c 65 |...+......7table|
00000fe0 68 65 6c 6c 6f 68 65 6c 6c 6f 02 43 52 45 41 54 |hellohello.CREAT|
00000ff0 45 20 54 41 42 4c 45 20 68 65 6c 6c 6f 28 78 29 |E TABLE hello(x)|
*
00001fe0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0f |................|
00001ff0 01 02 27 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 |..'Hello, world!|
[sqleet]% ./sqleet hello.db
SQLite version 3.20.1 2017-08-24 16:21:36
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> PRAGMA rekey='swordfish';
sqlite> .quit
[sqleet]% hexdump -C hello.db
00000000 f5 85 5b cf b4 91 d1 28 f8 5c 0e da ee 7f 66 d1 |..[....(.\....f.|
00000010 55 4e 9f 71 a8 e0 8d f0 52 d8 5c 17 63 9f cc 71 |UN.q....R.\.c..q|
00000020 b3 69 9d c0 ef d1 31 5c 52 fa a3 64 47 be 65 98 |.i....1\R..dG.e.|
00000030 58 53 9c 2e db 3a ce 66 a4 d1 22 bd d2 c8 13 1b |XS...:.f..".....|
[sqleet]% hexdump -C hello.db
00000000 4e 61 0c 1a 25 3f 77 1e 20 50 f4 56 61 c6 b3 37 |Na..%?w. P.Va..7|
00000010 eb aa d5 59 37 0d e6 41 1d d1 69 c8 8e 9a f5 eb |...Y7..A..i.....|
*
00001fe0 07 79 a0 3b f1 cc 9f 7b b2 72 11 21 28 15 71 ce |.y.;...{.r.!(.q.|
00001ff0 e5 ad 4a cd 75 af 8e 8a e2 79 f3 d9 2e 21 e8 4b |..J.u....y...!.K|
```

The database can only be read with the correct password.
The encrypted database is accessible only with the correct password.

```
[sqleet]% ./sqleet hello.db
SQLite version 3.20.1 2017-08-24 16:21:36
[sqleet]% ./sqleet hello.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> .dump
sqlite> .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
/**** ERROR: (26) file is not a database *****/
Expand All @@ -100,16 +102,54 @@ INSERT INTO hello VALUES('Hello, world!');
COMMIT;
```

If the target database has a non-default page size (i.e., other than 4096),
then `page_size` must be initialized accordingly with `PRAGMA` before setting
the encryption key. See [Remarks](#remarks) for more information.
The password can also be provided via [SQLite3 URI
filenames](https://www.sqlite.org/uri.html).

```
[sqleet]% ./sqleet "file:hello.db?key=swordfish"
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> SELECT * FROM hello;
Hello, world!
```

SQLite3 encryption API
----------------------
**Note**: \
The contents of an encrypted database file are indistinguishable from random
data of the same length. This is a conscious design decision made in sqleet, but
as a drawback, database settings cannot be read directly from the database file.
Thus, it is the user's responsibility to guarantee that the settings are
initialized properly before accessing the database. Most importantly, if the
database page size differs from the default 4096, then opening the database will
fail regardless of correct key unless the user explicitly sets `page_size` to
the proper value using, for example, `PRAGMA` command or URI API.

sqleet defines `SQLITE_HAS_CODEC` at the compile time to expose SQLite3's
`sqlite3_key()` and `sqlite3_rekey()` functions for managing encryption keys.
In contrast, the official [SQLite Encryption Extension
(SEE)](https://www.sqlite.org/see) leaves bytes 16..23 of the database header
unencrypted so that certain information, including the page size, can be read
from encrypted databases - with the obvious cost of making database files
distinguishable from random. sqleet can optionally be compiled with
`-DSKIP_HEADER_BYTES=24` flag to get the same default behavior (bytes 0..15
contain the KDF salt so only the bytes 16..23 are actually skipped). URI
parameter `skip=n` overrides the value of `SKIP_HEADER_BYTES` with `n`.


Library API
-----------

The public sqleet API consists of [C programming
interface](#c-programming-interface) and [URI configuration
interface](#uri-configuration-interface).


### C programming interface

sqleet defines `SQLITE_HAS_CODEC` compile-time option to expose SQLite3
encryption API, i.e., C functions `sqlite3_key()` and `sqlite3_rekey()` for
managing database encryption keys. These functions can be called directly from C
code, while other programming languages need to call the C functions via
[FFI](https://en.wikipedia.org/wiki/Foreign_function_interface) mechanism.
Another way to invoke the functions is with PRAGMAs `key` and `rekey` (see
[Example](#example)).

```c
SQLITE_API int sqlite3_key( /* Invoked by PRAGMA key='x' */
Expand All @@ -119,11 +159,11 @@ SQLITE_API int sqlite3_key( /* Invoked by PRAGMA key='x' */
```

`sqlite3_key()` is typically called immediately after `sqlite3_open()` to
specify an encryption key for the opened database. The function returns
specify an encryption key (password) for the opened database. Return value is
`SQLITE_OK` if the given key was correct; otherwise, a non-zero SQLite3 error
code is returned and subsequent attempts to read or write the database will
fail. Note that the first page of the database is read from the disk in order
to validate the key.
fail. The function validates the key by reading & decrypting the first page of
the database from disk.

```c
SQLITE_API int sqlite3_rekey( /* Invoked by PRAGMA rekey='x' */
Expand All @@ -133,37 +173,54 @@ SQLITE_API int sqlite3_rekey( /* Invoked by PRAGMA rekey='x' */
```

`sqlite3_rekey()` changes the database encryption key. This includes encrypting
the database the first time, decrypting the database (if nKey == 0), as well as
re-encrypting it with a new key. Internally, `sqlite3_rekey()` performs a
`VACUUM` to encrypt/decrypt all pages of the database. The return value is
`SQLITE_OK` on success and a SQLite3 error code on failure.
the database the first time, decrypting the database (if `nKey == 0`), as well
as re-encrypting it with a new key. Internally, `sqlite3_rekey()` runs `VACUUM`
command to encrypt/decrypt all pages of the database, whereas re-encryption is
performed directly by processing each page sequentially. The return value is
`SQLITE_OK` on success and an SQLite3 error code on failure.

In addition, there are `sqlite3_key_v2()` and `sqlite3_rekey_v2()` functions
that accept the target database name as the second parameter. By default, the
main database is used.
accepting name of the target database as the second parameter.


#### Raw keys
### URI configuration interface

**Disclaimer**: The current interface is experimental and likely to change in
future versions (see issue #13 for discussion). Use at your own risk!
**Disclaimer**: URI interface is experimental and subject to changes in future
versions. Use at your own risk!

The encryption functions pass the provided key string (password) to a key
derivation algorithm (i.e., PBKDF2-HMAC-SHA256 with a 16-byte salt and 12345
iterations). Optionally, the user can bypass the key derivation by specifying a
raw key in format `raw:K` where `K` is a 32-byte binary string or a 64-digit
hex-encoded string. This is useful in programs that use sqleet as a library and
want to handle key derivation by themselves. Additionally, the raw key string
can also be followed by a 16-byte (or 32-hexdigit) salt which is stored in the
beginning of the database file (otherwise a random salt is generated).
**Warning:** In any way erroneous raw key (e.g., unsupported length or invalid
hex-encoding) results in the key being handled as a normal key including the
`raw:` prefix. Moreover, specifying a salt makes sense only when creating a new
database or re-encrypting an existing database, because otherwise the specified
salt is overridden by the salt stored in the database file.
Run-time configuration of sqleet encryption is implemented based on [SQLite3 URI
filenames](https://www.sqlite.org/uri.html) which allow defining parameters when
opening a database. List of URI parameters supported by sqleet:

| Parameter | Description |
| :--- | :--- |
| `key` | Encryption key for `sqlite3_key()` after opening the database |
| `salt` | 16-byte salt for the key derivation function (KDF) |
| `header` | 16-byte header overwriting the database magic header |
| `kdf` | Key derivation function (only `none` supported for now) |
| `skip` | Run-time setting overriding compile-time SKIP_HEADER_BYTES |
| `page_size` | Equivalent to `page_size` PRAGMA |

Android support
---------------
Parameters `key`, `salt` and `header` have corresponding hex-prefixed versions.

Parameters `salt` and `header` expect 16-byte strings as values (shorter strings
are zero-padded to 16 bytes). The KDF salt is stored in the first 16 bytes of
the database file if `header` is undefined. Otherwise the value of `header`
overwrites the first 16 bytes, effectively hiding the KDF salt from the database
file.

When the default PBKDF2-HMAC-SHA256 KDF is disabled with `kdf=none`, URI
parameter `key` (in addition to PRAGMAs `key` and `rekey`) accepts a 32-byte
string that becomes the *master* encryption key which is normally derived by the
KDF from the given key. This allows the users of the library to take full
control of the key derivation process if needed.

Erroneus parameters (e.g., unsupported value length or otherwise bad value)
cause the opening of the database to fail with a non-zero SQLite3 error code.


Android/iOS support
-------------------

sqleet does not have an out-of-the-box support for Android. However, [SQLite
Android Bindings](https://www.sqlite.org/android/doc/trunk/www/index.wiki)
Expand All @@ -174,28 +231,36 @@ In particular, see [Using The SQLite Encryption
Extension](https://www.sqlite.org/android/doc/trunk/www/see.wiki) page for
build & usage instructions.

Likewise, sqleet does not offer a specialized version for iOS either, but
compiling a custom SQLite3 with sqleet encryption support for iOS is a
straightforward task (e.g., compile
[switflyfalling/SQLiteLib](https://github.com/swiftlyfalling/SQLiteLib) using
sqleet release amalgamation instead of the SQLite3 amalgamation). Moreover, iOS
apps get terminated when sent to the background if they use an *encrypted*
WAL-journaled SQLite3 database located in a shared data container (see
[sqlcipher/sqlcipher#255](https://github.com/sqlcipher/sqlcipher/issues/255),
[TN2408](https://developer.apple.com/library/archive/technotes/tn2408/_index.html)
and
[TN2151](https://developer.apple.com/library/archive/technotes/tn2151/_index.html)
for more information). One workaround is to leave the first 32 bytes of the
database file unencrypted so that iOS recognizes the file as a SQLite3 database
in WAL journaling mode. Thus, an iOS-compatible sqleet database can be created
in the following fashion:

Remarks
-------

The contents of an encrypted database file are indistinguishable from random
data of the same length. This is a conscious design decision made in sqleet,
but as a drawback, database settings cannot be read directly from the database
file. Thus, it is the user's responsibility to guarantee that the settings are
initialized properly before accessing the database. Most importantly, if the
database page size differs from the default value of 4096, then the user must
explicitly set `page_size` to the actual value (using, e.g., `PRAGMA` command)
or otherwise opening the database will fail regardless of correct key.

In contrast, the official [SQLite Encryption Extension
(SEE)](https://www.sqlite.org/see) leaves the bytes 16 through 23 of the
database header unencrypted so that specific information, including the page
size, can be read from encrypted databases - with the obvious cost of making
database files distinguishable from random. sqleet can optionally be compiled
with the same behavior by giving `-DSKIP_HEADER_BYTES=24` flag at compile time
(the value 24 only skips the encryption of the bytes 16 through to 23 because
the first 16 bytes contain a plaintext salt anyway).

```
[sqleet]% rm -f secrets.db
[sqleet]% ./sqleet "file:secrets.db?key=swordfish&salt=SodiumChloride42&header=SQLite%20format%203&skip=32"
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> CREATE TABLE f(x,y);
sqlite> .quit
[sqleet]% xxd secrets.db | head -n5
00000000: 5351 4c69 7465 2066 6f72 6d61 7420 3300 SQLite format 3.
00000010: 1000 0101 2040 2020 0000 0001 0000 0002 .... @ ........
00000020: 4640 824c 703e 3f72 dffc 3a19 23a6 c964 F@.Lp>?r..:.#..d
00000030: a1b3 abf0 8f3c 996f 0eb8 c665 afe1 0d72 .....<.o...e...r
00000040: b864 57f7 2492 8c31 6398 61d0 5d49 5a28 .dW.$..1c.a.]IZ(
```

Versioning scheme
-----------------
Expand Down

0 comments on commit 3fc3fc4

Please sign in to comment.