Icinga2 crashs while scan from Saint

[linuxmail]

Hi,

Expected Behavior

Internal security scan should not bring Icinga2 to a broken state

Current Behavior

While scanning, some instances (not all) gets unresponsible and we have to kill the process (because, our Systemd tries to get Icinga2 in a known state back)

Steps to Reproduce (for bugs)

Let Saint scan the API on port 5665

Your Environment

• Version used (icinga2 --version): version: r2.10.2-1
• Operating System and version: Debian Stretch 9.5
• Enabled features (icinga2 feature list): Enabled features: api checker mainlog

debug.txt

Maybe related to: #6559

[Crunsher]

Sounds like it could be a different flavor of #6559 indeed
Can you provide the log of the time around the crash?

[Al2Klimov]

@lippserd What about a v2.10.3 w/ #6817?

[linuxmail]

hi,

I have: attached.
icinga2.log

[linuxmail]

I've found also a 2nd crash, but Systemd was able to "handle" it. Attached.
debug.log.gz

Update: the file has ~11k lines so its compressed and I replaced some internal infos.

[lippserd]

@linuxmail Could you please retest our snapshot packages? We improved the stability of our API since the last release.

[Al2Klimov]

This issue seems to have been addressed by #7005.

[linuxmail]

@linuxmail Could you please retest our snapshot packages? We improved the stability of our API since the last release.

Sorry for the late response. I will try to push them into production.

[linuxmail]

hi,

so, either Systemd was able to get Icinga2 process working fast again, before we (or master) notice it, or it was better working. I had only one, which I had to kill -9 and start again. But the version says r2.8.1-1, but 2.10.2-1.stretch is installed. So I think, while upgrading the process was not killed / stopped before upgrade.

##########################################
Application information:
  Application version: r2.8.1-1
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-6-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: d3c3d2a588bd
Stacktrace:

        (0) libpthread.so.0: <unknown function> (+0x110c0) [0x7f1e84ad40c0]
        (1) libc.so.6: gsignal (+0xcf) [0x7f1e81c9efff]
        (2) libc.so.6: abort (+0x16a) [0x7f1e81ca042a]
        (3) libc.so.6: <unknown function> (+0x2be67) [0x7f1e81c97e67]
        (4) libc.so.6: <unknown function> (+0x2bf12) [0x7f1e81c97f12]
        (5) libremote.so.2.8.1: icinga::HttpServerConnection::ProcessMessageAsync(icinga::HttpRequest&) (+0x1607) [0x7f1e8453a307]
        (6) libbase.so.2.8.1: icinga::WorkQueue::WorkerThreadProc() (+0x592) [0x7f1e83e9ff82]
        (7) libboost_thread.so.1.62.0: <unknown function> (+0x12116) [0x7f1e85aa8116]
        (8) libpthread.so.0: <unknown function> (+0x7494) [0x7f1e84aca494]
        (9) libc.so.6: clone (+0x3f) [0x7f1e81d54acf]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

[dnsmichi]

Hi @linuxmail,

please test the current snapshot packages and run your security scanner against it.

Thanks,
Michael

[dnsmichi]

https://icinga.com/docs/icinga2/snapshot/doc/21-development/#snapshot-packages-nightly-builds

[linuxmail]

hi,

I have a look to do it next week. It's a bit complicated to create a "AdHoc" scan in a PCI environment :-)

[dnsmichi]

I know but it would really help to know whether it survives a scan in your environment. If not, we can fix it prior the release :)

[linuxmail]

hi,

just an update: We try to add the snapshot packages to our Debian Repo (aptly) which is a bit more complicated, because of the required libboost packages.

[dnsmichi]

For Stretch, you'll need the backports repository (likely you already have that because of plugins, etc.).

[linuxmail]

Yeah, I saw it :-) We added a new mirror on aptly with a filter to get only the boost packages and merged them with our local Icinga2 repo. The first test was working ...

==> /var/log/icinga2/icinga2.log <==
...
[2019-04-30 14:56:27 +0200] information/RemoteCheckQueue: items: 0, rate: 0/s (36/min 180/5min 540/15min);
[2019-04-30 14:56:31 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:1028): Error: http request

[2019-04-30 14:56:31 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:1026): Error: unsupported protocol

[2019-04-30 14:56:31 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:1026): Error: version too low

[2019-04-30 14:56:31 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:1026): Error: no shared cipher

[2019-04-30 14:56:31 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:1026): Error: wrong version number

[2019-04-30 14:56:31 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:1026): Error: wrong version number

[2019-04-30 14:56:31 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:1026): Error: wrong version number
...
[2019-04-30 14:57:07 +0200] information/RemoteCheckQueue: items: 0, rate: 0/s (36/min 180/5min 540/15min);
[2019-04-30 14:57:08 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:34524): Error: stream truncated

[2019-04-30 14:57:08 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:16967): Error: stream truncated

I've started a 2nd run with 4 Icinga2 instances. Two with the snapshots and two with the 2.10.2-1.stretch

[dnsmichi]

Cool, many thanks for sharing - also in terms of the error messages, expected with the scanner doing some nasty things.

[linuxmail]

Scan was finished. Unfortunately, all Icinga2 instances survived :-), but the snapshots seems to have a better handling, because I had less error messages . On the stable version, I have hundreds of:

/var/log/icinga2/icinga2.log:[2019-04-30 15:25:49 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:46774): Error: Socket was closed during TLS handshake.                                                      
/var/log/icinga2/icinga2.log:[2019-04-30 15:25:49 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:46776): Error: Socket was closed during TLS handshake.                                                      
/var/log/icinga2/icinga2.log:[2019-04-30 15:25:49 +0200] critical/ApiListener: Client TLS handshake failed (from [192.168.43.77]:46778): Error: Socket was closed during TLS handshake.  

:-)

[dnsmichi]

Error handling has changed quite a lot with the help of Boost ASIO. Good to hear that it works out, checking this is on the global list for #7041. Thanks again, this really helps a lot ❤️

[dnsmichi]

Marking this as solved to clear the view on the open tasks for 2.11 :)