/***************************************************************************
 *  _____
 * |\    | >                   VESmail
 * | \   | >  ___       ___    Email Encryption made Convenient and Reliable
 * |  \  | > /   \     /   \                               https://vesmail.email
 * |  /  | > \__ /     \ __/
 * | /   | >    \\     //        - RFC5322 MIME Stream Encryption & Decryption
 * |/____| >     \\   //         - IMAP4rev1 Transparent Proxy Server
 *       ___      \\_//          - ESMTP Transparent Proxy Server
 *      /   \     /   \          - VES Encryption Key Exchange & Recovery
 *      \__ /     \ __/
 *         \\     //    _____                     ______________by______________
 *          \\   //  > |\    |
 *           \\_//   > | \   |                    VESvault
 *           /   \   > |  \  |                    Encrypt Everything
 *           \___/   > |  /  |                    without fear of losing the Key
 *                   > | /   |                              https://vesvault.com
 *                   > |/____|                                  https://ves.host
 *
 * (c) 2020 VESvault Corp
 * Jim Zubov <jz@vesvault.com>
 *
 * GNU General Public License v3
 * You may opt to use, copy, modify, merge, publish, distribute and/or sell
 * copies of the Software, and permit persons to whom the Software is
 * furnished to do so, under the terms of the COPYING file.
 *
 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
 * KIND, either express or implied.
 *
 ***************************************************************************/


VESmail is a transparent end-to-end email encryption solution.
This distribution consists of:
- Email stream encryption/decryption engine
- Transparent IMAP and SMTP proxy servers
- Daemonizer for running the proxies in any compatible environment
VESmail uses VES end-to-end encryption and public key infrastructure
[https://vesvault.com].
A VES account is needed to use email encryption and decryption.
An end-to-end encrypted VESmail profile that stores the authentication
credentials for IMAP and SMTP servers is required to use IMAP and SMTP proxies.
[https://vesmail.email]


DISTRIBUTION CONTENTS:

lib/: Email stream encryption / decryption library components
srv/: General proxy server components
imap/: IMAP4rev1 proxy components
smtp/: ESMTP proxy components
now/: HTTP mini server for instant access to spooled encrypted emails
cli/: Command line utility components
app/: Components for local app builds
snif/: SNIF VPN connector (https://snif.host)

etc/vesmail/: Sample config and template files
etc/xinetd.d/: A sample external daemonizer conf for IMAP & SMTP proxies
etc/systemd/: A sample internal daemonizer conf


REQUIRED LIBRARIES:

  libVES (https://github.com/vesvault/libVES.c)
    libVES
    libVES.h
    libVES/*.h
    jVar.h
  OpenSSL (https://www.openssl.org/source/)
    libcrypto
    libssl
    openssl/*.h
  cURL (https://curl.haxx.se/download.html)
    libcurl
    curl/*.h


GUI LOCAL PROXY APP SOURCES:

See app/README


EMAIL ENCRYPTION & DECRYPTION:

Message-ID: (RFC 2392) is used as a unique identifier for the message
encryption key. If a message without Message-ID is passed to the encryptor,
VESmail creates a new one. The message encryption key is randomly generated by
the encryptor, encrypted with each recipient's public key that is retrieved
from the VES repository, and then each end-to-end encrypted copy of the message
encryption key is deposited into the VES repository. Each recipient can decrypt
the message encryption key using their own VES keychain.

Most of the message RFC822 and MIME headers and all body parts are encrypted
with the message encryption key. Each encrypted header is passed as
X-VESmail-Header: base64 encoded ciphertext, usually wrapped multiline.
The MIME structure of the message is mostly preserved. Each content body part
is replaced with an application/vnd.ves.encrypted base64 encoded ciphertext.
A multipart/alternative container is injected by the encryptor if the message
didn't already have one. Plaintext banner parts are injected, and become the
main message content when the encrypted message is viewed in an email client.
The encryptor injects certain X-VESmail-*: into the message and MIME headers
to be handled by the decryptor.

The decryptor reverts each encrypted header and content part to their original
state, preserving all essential formatting and encoding. All injected headers,
banners and containers are removed. The decrypted message is mostly identical
to the original one.

If the original message was not properly RFC822 formatted, the encryptor treats
the content after the last well-formed RFC822 header as the body and encrypts
the parts accordingly.

Note that the lead-in part of a multipart container (preceding the first
boundary), as well as the lead-out part (after the terminal boundary), are
never expected to contain any sensitive information and are passed as an
unencrypted plaintext.


IMAP & SMTP PROXIES:

Both IMAP and SMTP VESmail proxies connect to the email provider's server, pass
the requests and responses between the email client and the provider's server,
and perform necessary transformations to transparently encrypt/decrypt the
email messages.

A user needs a VESmail profile to connect to IMAP/SMTP proxy. The VESmail
Profile is an end-to-end encrypted JSON object that is stored in VES repository
and is decryptable by the account owner's VES keychain. The profile contains
the settings and credentials for the provider's IMAP and SMTP servers, and a
few settings specific to VESmail.

The user's email client connects to the VESmail IMAP/SMTP proxy instead of the
provider's server, and authenticates with specific username and password that
are provided by the VESmail Profile management webpage.

The username contains the information about the email address and the VESmail
profile, in many cases it is simply the user's email address. The password is
the user's VES App Key for VESmail that unlocks all encrypted information
pertaining to the user's VESmail. Considering that, the password / App Key
should be kept secret. In case of it being compromised, the user can rekey
the vault through [https://vesvault.com].


VESMAIL PROXY AUTHENTICATION
(LOGIN or SASL for IMAP, SASL for SMTP):

_ LOGIN acme@example.com MySecret
    ; VES account:     ves://vesmail/acme@example.com/
    ; VESmail AppKey:  MySecret
    ; VESmail Profile: ves://vesmail/acme@example.com!

_ LOGIN acme@example.com! MySecret
    ; VES account:     ves://vesmail/acme@example.com/
    ; VESmail AppKey:  MySecret
    ; VESmail Profile: ves://vesmail/acme@example.com!

_ LOGIN acme@example.com!profile1 MySecret
    ; VES account:     ves://vesmail/acme@example.com/
    ; VESmail AppKey:  MySecret
    ; VESmail Profile: ves://vesmail/acme@example.com!profile1

_ LOGIN acme@example.com#profile1 MySecret
    ; VES account:     ves://vesmail/acme@example.com/
    ; VESmail AppKey:  MySecret
    ; VESmail Profile: ves://vesmail/acme@example.com%23profile1

_ LOGIN acme@example.com##profile1 MySecret
    ; VES account:     ves://vesmail/acme@example.com/
    ; VESmail AppKey:  MySecret
    ; VESmail Profile: ves://vesmail/profile1


IMAP PROXY OPERATION:

The IMAP proxy intercepts FETCH and APPEND requests and responses. For each
FETCH request or response, the proxy may issue additional FETCH requests to the
provider's IMAP server and consume the responses to identify VESmail encrypted
messages and collect the information necessary for successful decryption.
Once the decryption is possible or is determined unnecessary for each requested
message, the proxy returns properly formed FETCH responses to the client.

Collecting the information involves buffering the responses and intermediate
data. To avoid memory overusage, VESmail has a limit on a size of an IMAP
literal object that is allowed to be buffered. For object that exceed the
allowed size VESmail invokes synchronous transformation, when the response
is either transformed as a stream and passed to the client or dropped, but not
buffered.

Due to the nature of IMAP literals, the size must be known before the
transmission starts. Therefore, synchronous transformation usually involves
multiple fetches of large objects from the provider's IMAP server, with
intermediate responses being consumed by VESmail to collect the size and other
info, and causes increased latency and traffic as the cost of reducing the
memory profile.

In case when an encrypted email was sent to a user who is not currently on VES,
libVES might temporarily be not able to decrypt the message encryption key
until the VES key exchange process is complete. RFC3501 does not explicitly
specify how to handle FETCH requests when the requested message is temporarily
unavailable. VESmail silently drops all responses for the messages that cannot
be currently decrypted, which may cause confusion with some email clients.
All clients that have been tested so far either hide an unavailable message or
display an empty message placeholder, and are able to download the message
later when the key becomes available.

Certain IMAP servers, notably Yahoo mail and AOL, do not fully comply with MIME
specs, and their FETCH BODY[*.MIME] responses drop all headers that do not
start with Content-*. This is a major problem with VESmail since it passes some
critical information in X-VESmail-* headers in MIME body parts. The problem
is referred as the MIME Bug in the VESmail code. VESmail implements a mechanism
for detecting and handling the MIME bug, at the expense of some extra traffic
and latency for concerned IMAP providers.

Because of the streamed nature of VESmail encryption, the proxy does not know
the exact size of a message or its parts until it's fully decrypted. By
default, VESmail sends an estimate of bytes and lines in a FETCH BODY /
BODYSTRUCTURE response, unless the message have been already fully processed
in the current IMAP session.

Certain IMAP clients, notably Apple native software, fail to properly handle a
case when the estimated size exceeds the later determined accurate size. Such
clients fall into an infinite loop of ranged FETCH beyond the end of the
content. VESmail detects such requests, referred as OOR in the code. Once
repetitive OOR requests have been detected, VESmail sends a blank padding in
response to pacify the client, and switches to CALC mode, when all sizes are
calculated by fully consuming and decrypting each encrypted message.

The CALC option can be explicitly set in the VESmail profile to deal with
problematic clients, at the expence of traffic and latency.

APPEND requests are also intercepted by the proxy to encrypt all messages being
saved to folders. The buffering limit for synchronous transformation applies to
APPEND requests too. APPEND is always single-pass, if any appended message
exceeds the threshold size, the synchronous mode is invoked. The encrypted size
is estimated with a margin based on the original size. The appended encrypted
message is padded with blank lines up to the size originally estimated. In an
unlikely case when the encrypted size exceeds the estimate, the saved message
will be truncated. It it extremely unlikely to happen for any realistic email
message, and might be addressed in further versions if ever becomes a concern.


SMTP PROXY OPERATION:

The SMTP proxy is more simple than IMAP. Unlike IMAP, SMTP if fully streamed
doesn't need any special handling for large messages.

The proxy intercepts RCPT TO: requests to collect the email addresses to share
the message encryption key with. When a recipient doesn't have a VES account
yet, the behaviour is determined by the mode that's been selected when setting
up the VESmail profile.

If the FALLBACK mode was selected, the email goes out unencrypted if any of the
recipients in not on VES. Maximum convenience for recipients, at the cost of
lowest security.

In the FAIL mode the proxy returns an error code for any recipient that is not
on VES. May cause unexpected behaviour in some clients, and is generally not
recommended.

The XCHG and HIGH modes will invoke VES temp key onboarding mechanism for new
recipients. The recipient will need to set up their VES PIN, and either go
through VESmail setup process, or in case if the sender was using a VESmail
Enterprise instance with an instant viewer, open the message in the web based
viewer.

The XCHG mode is more convenient than HIGH, but is a little less secure. XCHG
adds VES temp keys to the headers of the encrypted emails for new VES users,
so the recipient is always able to access the email once they've created the
VES PIN. In HIGH mode, once the recipient has their VES PIN set up, libVES will
exchange the end-to-end encrypted temp key next time the sender's email client
accesses VESmail proxy. This is the most secure option, but it may delay the
exchange and make the message inaccessible to the recipient for a while.


DAEMONIZER:

VESmail Proxies can run using either an internal daemonizer, or an external
one such as xinetd. Both IMAP and SMTP proxies support STARTTLS and persistent
TLS. The "sni" block in the conf file sets up a multi-host environment, loading
separate conf file based on a negotiated SNI hostname.

The internal daemonizer starts a thread for each proxy, defined in the conf
file. Each proxy session runs in a separate thread.

SNI conf files are automatically reloaded when modified. Changes to the main
conf require full restart to take effect. Changes to the SSL certificate and
private key files are not automatically tracked, update the mtime on the SNI
conf file to reload a renewed certificate.

On *nix platforms, SIGHUP initiated a graceful termination of all daemons,
SIGTERM invokes a forced termination. A guard option (-G) starts a guard
process and one or more forked workers, the guard keeps the listening sockets
and restarts terminated workers. SIGHUP sent to the guard initiates graceful
restart of all workers, SIGTERM initiates a forced termination.


SECURITY CONSIDERATIONS:

Unless run on localhost, all communications between the client and the proxy
should be handled over TLS.

VESmail proxy process must have read access to the private key matching the SSL
certificate. It is recommended to run VESmail proxy as a dedicated user account
(e.g. "vesmail") that has access to sensitive files like private keys.

The "audit" option in the conf file enables compliance audit which might be
required for corporate instances. An encryption key for every email encrypted
through the VESmail proxy instance will be shared with one or more VESmail
accounts specified in the "audit" array. The specified audit account must
already exist on VES and VESmail, otherwise the encryptor will fail.

Write access to the main vesmail.conf and all SNI conf files should be
appropriately restricted to avoid any kind of unauthorized access to the
content of the encrypted emails.


CLI MODE WITH VES KEYSTORE:

The CLI encrypt and decrypt commands will invoke libVES KeyStore dialog
for the selected VESmail user, if built with libVES >= 1.24:

vesmail encrypt -a user@domail.com
vesmail decrypt -a user@domain.com

To bypass the PIN dialog and enable batch mode for the specific user, use
the following commands:

PIN-less session (for batch encrypt only, no reused Message-Id's):

ves -a //vesmail/user@domain.com/ -E sess,save

PIN-less app key (for batch encrypt & decrypt, use wisely)

ves -a //vesmail/user@domain.com/ -E save

Remove the PIN-less app key and session respectively:

ves -a //vesmail/user@domain.com/ -E forget,nopin
ves -a //vesmail/user@domain.com/ -E forget,sess