Skip to content

Commit

Permalink
Merge pull request #7231 from Icinga/feature/docs-tech-concept-networ…
Browse files Browse the repository at this point in the history 
…k-io

Docs: Update technical concepts for TLS Network IO and Boost Asio, Beast and Coroutines
  • Loading branch information
Michael Friedrich authored Jun 5, 2019
2 parents f312962 + 48f9b24 commit f3283ed
Showing 1 changed file with 94 additions and 167 deletions.
261 changes: 94 additions & 167 deletions doc/19-technical-concepts.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -739,196 +739,123 @@ Icinga 2 v2.9+ adds more performance metrics for these values:

### TLS Connection Handling <a id="technical-concepts-tls-network-io-connection-handling"></a>

TLS-Handshake timeouts occur if the server is busy with reconnect handling and other tasks which run in isolated threads. Icinga 2 uses threads in many ways, e.g. for timers to wake them up, wait for check results, etc.

In terms of the cluster communication, the following flow applies.

#### Master Connects <a id="technical-concepts-tls-network-io-connection-handling-master"></a>

* The master initializes the connection in a loop through all known zones it should connect to, extracting the endpoints and their host/port attribute.
* This calls `AddConnection()` whereas a `Tcp::Connect()` is called to create a TCP socket.
* A new thread is spawned for future connection handling, this binds `ApiListener::NewClientHandler()`.
* On top of the TCP socket, a new TLS stream is created.
* The master performs a `TLS->Handshake()`
* Certificates are verified and the endpoint name is compared to the CN.


#### Clients Processes Connection <a id="technical-concepts-tls-network-io-connection-handling-client"></a>

* The client listens for new incoming connections as 'TCP server' pattern inside `ListenerThreadProc()` with an endless loop.
* Once a new connection is detected, `TCP->Accept()` performs the initial socket establishment.
* A new thread is spawned for future connection handling, this binds `ApiListener::NewClientHandler()`, Role being Server.
* On top of the TCP socket, a new TLS stream is created.
* The client performs a `TLS->Handshake()`.


#### Data Transmission between Server and Client Role <a id="technical-concepts-tls-network-io-connection-handling-data-transmission"></a>

Once the TLS handshake and certificate verification is completed, the role is either `Client` or `Server`.

* Client: Send "Hello" message.
* Server: `TLS->WaitForData()` waits for incoming messages from the remote client.

`Client` in this case is the instance which initiated the connection. If the master is doing this,
the Icinga 2 client/agent acts as "server" which accepts incoming connections.


### Asynchronous Socket IO <a id="technical-concepts-tls-network-io-async-socket-io"></a>
Icinga supports two connection directions, controlled via the `host` attribute
inside the Endpoint objects:

* Outgoing connection attempts
* Incoming connection handling

Once the connection is established, higher layers can exchange JSON-RPC and
HTTP messages. It doesn't matter which direction these message go.

This offers a big advantage over single direction connections, just like
polling via HTTP only. Also, connections are kept alive as long as data
is transmitted.

When the master connects to the child zone member(s), this requires more
resources there. Keep this in mind when endpoints are not reachable, the
TCP timeout blocks other resources. Moving a satellite zone in the middle
between masters and agents/clients helps to split the tasks - the master
processes and stores data, deploys configuration and serves the API. The
satellites schedule the checks, connect to the agents/clients and receive
check results.

Agents/Clients can also connect to the parent endpoints - be it a master or
a satellite. This is the preferred way out of a DMZ, and also reduces the
overhead with connecting to e.g. 2000 agents on the master. You can
benchmark this when TCP connections are broken and timeouts are encountered.

#### Master Processes Incoming Connection <a id="technical-concepts-tls-network-io-connection-handling-incoming"></a>

* The node starts a new ApiListener, this invokes `AddListener()`
* Setup SSL Context
* Initialize global I/O engine and create a TCP acceptor
* Resolve bind host/port (optional)
* Listen on IPv4 and IPv6
* Re-use socket address and port
* Listen on port 5665 with `INT_MAX` possible sockets
* Spawn a new Coroutine which listens for new incoming connections as 'TCP server' pattern
* Accept new connections asynchronously
* Spawn a new Coroutine which handles the new client connection in a different context, Role: Server

#### Master Connects Outgoing <a id="technical-concepts-tls-network-io-connection-handling-outgoing"></a>

* The node starts a timer in a 10 seconds interval with `ApiReconnectTimerHandler()` as callback
* Loop over all configured zones, exclude global zones and not direct parent/child zones
* Get the endpoints configured in the zones, exclude: local endpoint, no 'host' attribute, already connected or in progress
* Call `AddConnection()`
* Spawn a new Coroutine after making the SSL context
* Use the global I/O engine for socket I/O
* Create TLS stream
* Connect to endpoint host/port details
* Handle the client connection, Role: Client

#### TLS Handshake <a id="technical-concepts-tls-network-io-connection-handling-handshake"></a>

* Create a TLS connection in sslConn and perform an asynchronous TLS handshake
* Get the peer certificate
* Verify the presented certificate: `ssl::verify_peer` and `ssl::verify_client_once`
* Get the certificate CN and compare it against the endpoint name - if not matching, return and close the connection

#### Data Exchange <a id="technical-concepts-tls-network-io-connection-data-exchange"></a>

Everything runs through TLS, we don't use any "raw" connections nor plain message handling.

The TLS handshake and further read/write operations are not performed in a synchronous fashion
in the new client's thread. Instead, all clients share an asynchronous "event pool".

The TlsStream constructor registers a new SocketEvent by calling its constructor. It binds the
previously created TCP socket and itself into the created SocketEvent object.

`SocketEvent::InitializeEngine()` takes care of whether to use **epoll** (Linux) or
**poll** (BSD, Unix, Windows) as preferred socket poll engine. epoll has proven to be
faster on Linux systems.

The selected engine is stored as `l_SocketIOEngine` and later `Start()` ensures to do the following:

* Use a fixed number for creating IO threads.
* Create a `dumb_socketpair` which basically is a pipe from `in->out` and multiplexes the TCP socket
into a local Unix socket. This removes the complexity and slowlyness of the kernel dealing with the TCP stack and new events.
* `InitializeThread()` prepares epoll with `epoll_create`, socket descriptors and event mapping for later wakeup.
* Each event FD has its own "worker event thread" which deals with incoming data, called `ThreadProc` as endless loop.

By default, there are 8 of these worker threads.

In the `ThreadProc` loop, the following happens:

* `epoll_wait` gets called and provides an event whether new data is `ready` (via socket IO from the Kernel).
* The event created with `epoll_event` holds the `.fd.data` attribute which references the multiplexed event FD (and therefore tcp socket FD).
* All events in this cycle are stored with their descriptors in a list.
* Once the epoll loop is finished, the collected events are processed and the socketevent descriptor (which is the TlsStream object) calls `OnEvent()`.

#### On Socket Event State Machine <a id="technical-concepts-tls-network-io-async-socket-io-on-event"></a>

`OnEvent` implements the "state machine" depending on the current desired action. By default, this is `TlsActionNone`.

Once `TlsStream->Handshake()` is called, this initializes the current action to
`TlsActionHandshake` and performs `SSL_do_handshake()`. This function returns > 0
when successful, anything below needs to be dealt separately.

If the handshake was successful, the registered condition variable `m_CV` gets signalled
and the thread waiting for the handshake in `TlsStream->Handshake()` wakes up and continues
within the `ApiListener::NewClientHandler()` function.

Once the handshake is completed, current action is changed to either `TlsActionRead` or `TlsActionWrite`.
This happens in the beginning of the state machine when there is no action selected yet.
HTTP and JSON-RPC messages share the same port and API, so additional handling is required.

* **Read**: Received events indicate POLLIN (or POLLERR/POLLHUP as error, but normally mean "read").
* **Write**: The send buffer of the TLS stream is greater 0 bytes, and the received events allow POLLOUT on the event socket.
* Nothing matched: Change the event sockets to POLLIN ("read"), and return, waiting for the next event.
On a new connection and successful TLS handshake, the first byte is read. This either
is a JSON-RPC message in Netstring format starting with a number, or plain HTTP.

This also depends on the returned error codes of the SSL interface functions. Whenever `SSL_WANT_READ` occurs,
the event polling needs be changed to use `POLLIN`, vice versa for `SSL_WANT_WRITE` and `POLLOUT`.

In the scenario where the master actively connects to the clients, the client will wait for data and
change the event sockets to `Read` once there's something coming on the sockets.

Action | Description
---------------|---------------
Read | Calls `SSL_read()` with a fixed buffer size of 64 KB. If rc > 0, the receive buffer of the TLS stream is filled and success indicated. This endless loop continues until a) `SSL_pending()` says no more data from remote b) Maximum bytes are read. If `success` is true, the condition variable notifies the thread in `WaitForData` to wake up.
Write | The send buffer of the TLS stream `Peek()`s the first 64KB and calls `SSL_write()` to send them over the socket. The returned value is the number of bytes written, this is adjusted within the send buffer in the `Read()` call (it also optimizes the memory usage).
Handshake | Calls `SSL_do_handshake()` and if successful, the condition variable wakes up the thread waiting for it in `Handshake()`.

##### TLS Error Handling

TLS error code | Description
-------------------------|-------------------------
`SSL_WANT_READ` | The next event should read again, change events to `POLLIN`.
`SSL_ERROR_WANT_WRITE` | The next event should write, change events to `POLLOUT`.
`SSL_ERROR_ZERO_RETURN` | Nothing was returned, close the TLS stream and immediately return.
default | Extract the error code and log a fancy error for the user. Close the connection.

From this [question](https://stackoverflow.com/questions/3952104/how-to-handle-openssl-ssl-error-want-read-want-write-on-non-blocking-sockets):

```
With non-blocking sockets, SSL_WANT_READ means "wait for the socket to be readable, then call this function again."; conversely, SSL_WANT_WRITE means "wait for the socket to be writeable, then call this function again.". You can get either SSL_WANT_WRITE or SSL_WANT_READ from both an SSL_read() or SSL_write() call.
```
HTTP/1.1

##### Successful TLS Actions

* Initialize the next TLS action to `none`. This re-evaluates the conditions upon next event call.
* If the stream still contains data, adjust the socket events.
* If the send buffer contains data, change events to `POLLIN|POLLOUT`.
* Otherwise `POLLIN` to wait for data.
* Process data when the receive buffer has them available and we are actively handling events.
* If the TLS stream is supposed to shutdown, close everything including the TLS connection.

#### Data Processing <a id="technical-concepts-tls-network-io-async-socket-io-data-processing"></a>

Once a stream has data available, it calls `SignalDataAvailable()`. This holds a condition
variable which wakes up another thread in a handled which was previously registered, e.g.
for JsonRpcConnection, HttpServerConnection or HttpClientConnection objects.

All of them read data from the stream and process the messages. At this point the string is available as JSON already and later decoded (e.g. Icinga data structures, as Dictionary).



### General Design Patterns <a id="technical-concepts-tls-network-io-design-patterns"></a>

Taken from https://www.ibm.com/developerworks/aix/library/au-libev/index.html

2:{}
```
One of the biggest problems facing many server deployments, particularly web server deployments, is the ability to handle a large number of connections. Whether you are building cloud-based services to handle network traffic, distributing your application over IBM Amazon EC instances, or providing a high-performance component for your web site, you need to be able to handle a large number of simultaneous connections.
A good example is the recent move to more dynamic web applications, especially those using AJAX techniques. If you are deploying a system that allows many thousands of clients to update information directly within a web page, such as a system providing live monitoring of an event or issue, then the speed at which you can effectively serve the information is vital. In a grid or cloud situation, you might have permanent open connections from thousands of clients simultaneously, and you need to be able to serve the requests and responses to each client.
Before looking at how libevent and libev are able to handle multiple network connections, let's take a brief look at some of the traditional solutions for handling this type of connectivity.

### Handling multiple clients
Depending on this, `ClientJsonRpc` or `ClientHttp` are assigned.

There are a number of different traditional methods that handle multiple connections, but usually they result in an issue handling large quantities of connections, either because they use too much memory, too much CPU, or they reach an operating system limit of some kind.
JSON-RPC:

The main solutions used are:
* Create a new JsonRpcConnection object
* When the endpoint object is configured, spawn a Coroutine which takes care of syncing the client (file and runtime config, replay log, etc.)
* No endpoint treats this connection as anonymous client, with a configurable limit. This client may send a CSR signing request for example.
* Start the JsonRpcConnection - this spawns Coroutines to HandleIncomingMessages, WriteOutgoingMessages, HandleAndWriteHeartbeats and CheckLiveness

* Round-robin: The early systems use a simple solution of round-robin selection, simply iterating over a list of open network connections and determining whether there is any data to read. This is both slow (especially as the number of connections increases) and inefficient (since other connections may be sending requests and expecting responses while you are servicing the current one). The other connections have to wait while you iterate through each one. If you have 100 connections and only one has data, you still have to work through the other 99 to get to the one that needs servicing.
* poll, epoll, and variations: This uses a modification of the round-robin approach, using a structure to hold an array of each of the connections to be monitored, with a callback mechanism so that when data is identified on a network socket, the handling function is called. The problem with poll is that the size of the structure can be quite large, and modifying the structure as you add new network connections to the list can increase the load and affect performance.
* select: The select() function call uses a static structure, which had previously been hard-coded to a relatively small number (1024 connections), which makes it impractical for very large deployments.
There are other implementations on individual platforms (such as /dev/poll on Solaris, or kqueue on FreeBSD/NetBSD) that may perform better on their chosen OS, but they are not portable and don't necessarily resolve the upper level problems of handling requests.
HTTP:

All of the above solutions use a simple loop to wait and handle requests, before dispatching the request to a separate function to handle the actual network interaction. The key is that the loop and network sockets need a lot of management code to ensure that you are listening, updating, and controlling the different connections and interfaces.
* Create a new HttpServerConnection
* Start the HttpServerConnection - this spawns Coroutines to ProcessMessages and CheckLiveness

An alternative method of handling many different connections is to make use of the multi-threading support in most modern kernels to listen and handle connections, opening a new thread for each connection. This shifts the responsibility back to the operating system directly but implies a relatively large overhead in terms of RAM and CPU, as each thread will need it's own execution space. And if each thread (ergo network connection) is busy, then the context switching to each thread can be significant. Finally, many kernels are not designed to handle such a large number of active threads.
```


### Alternative Implementations and Libraries <a id="technical-concepts-tls-network-io-async-socket-io-alternatives"></a>

While analysing Icinga 2's socket IO event handling, the libraries and implementations
below have been collected too. [This thread](https://www.reddit.com/r/cpp/comments/5xxv61/a_modern_c_network_library_for_developing_high/)
also sheds more light in modern programming techniques.
All the mentioned Coroutines run asynchronously using the global I/O engine's context.
More details on this topic can be found in [this blogpost](https://www.netways.de/blog/2019/04/04/modern-c-programming-coroutines-with-boost/).

Our main "problem" with Icinga 2 are modern compilers supporting the full C++11 feature set.
Recent analysis have proven that gcc on CentOS 6 are too old to use modern
programming techniques or anything which implemens C++14 at least.
The lower levels of context switching and sharing or event polling are
hidden in Boost ASIO, Beast, Coroutine and Context libraries.

Given the below projects, we are also not fans of wrapping C interfaces into
C++ code in case you want to look into possible patches.
#### Data Exchange: Coroutines and I/O Engine <a id="technical-concepts-tls-network-io-connection-data-exchange-coroutines"></a>

One key thing for external code is [license compatibility](http://gplv3.fsf.org/wiki/index.php/Compatible_licenses#GPLv2-compatible_licenses) with GPLv2.
Modified BSD and Boost can be pulled into the `third-party/` directory, best header only and compiled
into the Icinga 2 binary.
Light-weight and fast operations such as connection handling or TLS handshakes
are performed in the default `IoBoundWorkSlot` pool inside the I/O engine.

#### C
The I/O engine has another pool available: `CpuBoundWork`.

* libevent: http://www.wangafu.net/~nickm/libevent-book/TOC.html
* libev: https://www.ibm.com/developerworks/aix/library/au-libev/index.html
* libuv: http://libuv.org
This is used for processing CPU intensive tasks, such as handling a HTTP request.
Depending on the available CPU cores, this is limited to `std::thread::hardware_concurrency() * 3u / 2u`.

#### C++

* Asio (standalone header only or as Boost library): http://think-async.com (the Boost Software license is compatible with GPLv2)
* Poco project: https://github.com/pocoproject/poco
* cpp-netlib: https://github.com/cpp-netlib/cpp-netlib
* evpp: https://github.com/Qihoo360/evpp
```
1 core * 3 / 2 = 1
2 cores * 3 / 2 = 3
8 cores * 3 / 2 = 12
16 cores * 3 / 2 = 24
```

The I/O engine itself is used with all network I/O in Icinga, not only the cluster
and the REST API. Features such as Graphite, InfluxDB, etc. also consume its functionality.

There are 2 * CPU cores threads available which run the event loop
in the I/O engine. This polls the I/O service with `m_IoService.run();`
and triggers an asynchronous event progress for waiting coroutines.

<!--
## REST API <a id="technical-concepts-rest-api"></a>
Expand Down

0 comments on commit f3283ed

Please sign in to comment.