Make existing intermediate/multi-root CA handling perfect

[Al2Klimov]

Status quo

Trusting only external intermediate CA #7719

i.e. making it Icinga's effective root CA.

See #7719 (comment) . In short, one has to cross-self-sign the intermediate CA and to use the result as Icinga's root CA. Also works w/o sharing CA keys with Icinga if all leaf certs are provided manually.

Using intermediate CAs like everyone else

Icinga-owned intermediate CA

i.e. Icinga signs leaf certs by itself with an intermediate CA in /var/lib/icinga2/ca/ca.* while its root CA is still in /var/lib/icinga2/certs/ca.crt.

Doesn't work, yet. External intermediate CAs work (see below), but at least when it comes to CSR handling, Icinga doesn’t take the whole chain into account.

External intermediate CAs

I'm such sort of a fool. 🙈 #8859 (comment)

I thought they don’t work. But they work. Don't ask me why. Probably because of how OpenSSL works:

• client picks up the whole chain in a leaf cert file
• client sends the whole chain
• server validates the whole chain during handshake

Apropos! We validate certs not only during handshakes, but also explicitly. However we never consider chains, not even in CSR handling. That's why we get strange error messages like this:

[2023-06-21 16:54:42 +0000] information/JsonRpcConnection: Received certificate request for CN 'aklimov-intca-2.novalocal' not signed by our CA: unable to get local issuer certificate (code 20)

So it basically works, but it's not perfect.

Test protocol

[root@aklimov-intca-1 ~]# icinga2 feature enable api
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
[root@aklimov-intca-1 ~]# icinga2 daemon -C
[2023-06-21 15:47:28 +0000] information/cli: Icinga application loader (version: r2.13.7-1)
[2023-06-21 15:47:28 +0000] information/cli: Loading configuration file(s).
[2023-06-21 15:47:28 +0000] information/ConfigItem: Committing config item(s).
[2023-06-21 15:47:28 +0000] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/certs//aklimov-intca-1.novalocal.crt': 2147483650, "error:80000002:system library::No such file or directory"
[2023-06-21 15:47:28 +0000] critical/config: Error: Cannot get certificate from cert path: '/var/lib/icinga2/certs//aklimov-intca-1.novalocal.crt'.
Location: in /etc/icinga2/features-enabled/api.conf: 5:1-5:24
/etc/icinga2/features-enabled/api.conf(3):  */
/etc/icinga2/features-enabled/api.conf(4):
/etc/icinga2/features-enabled/api.conf(5): object ApiListener "api" {
                                           ^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/features-enabled/api.conf(6):   //accept_config = false
/etc/icinga2/features-enabled/api.conf(7):   //accept_commands = false
[2023-06-21 15:47:28 +0000] critical/config: 1 error
[2023-06-21 15:47:28 +0000] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
[root@aklimov-intca-1 ~]# openssl req -x509 -newkey rsa:4096 -subj '/CN=Ext. Root CA' -md5 -keyout root.key -out root.crt -nodes
...+.............+......+........+...+...+.......+...+.....+.+..............+.+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+......+............+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+.........+.+........+......+...+...+.......+..+..........+..............+..........+......+.....+....+..+.............+.........+...........+.......+.....+..................+....+...+............+...+.....+.......+..................+..+......+.+...+..+....+......+........+...+.......+.....+.......+...+......+..+.......+........+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
....+.+...+.....+.........+..........+.....+.+..+...+..........+......+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+.....+...+...+............+.............+..+....+.....+...+...+....+...........+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+..+.+..+....+...+.....+......+.+...+...+...+.....+.+.....+...................+..+....+...+.....+.........................+...+............+.....+..........+.....+......+.......+...+....................+.........+..................+...................+...........+....+...........+.........................+..+............+.+....................+...................+.......................+......+.+...+......+......+...........+...+.+......+..................+.....................+..................+.....+.+.....+.+.....+.+.........+.....+...+..........+...+........+..................................+............+......+.....+.............+......+.....+.+.........+......+............+..+.........+......+......+....+.....+.........................+.....+.........+......+....+.....+...............+.+...........+.......+..+...+................+.....+.........+.............+..+.+............+..+.+..+...+.+........................+.....+..........+...........+......+...+...............+..........+..............+..........+..+.+.....+......................+...+..+............+..........+...........+.......+..+.+..+.......+...........+.......+..+..........+.....+.........+.......+...+...........+.........+............+..........+........+......+...+......+.+...+...........+.......+...+...........+...+..........+..+....+..+............+...+...+.+...+..+.+..+......+.+.........+.....+................+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-intca-1 ~]# openssl req -newkey rsa:4096 -subj '/CN=Ext. Intm. CA' -keyout intm.key -out intm.csr -nodes
....+.......+...+.....+.+.....+.+.........+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...................+........+......+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+......+..+......................+.....................+..+.........+.+.........+.....+.+........+....+.....+...+..................+.........+...............+.+......+........+.+...........+....+......+.....+.........+....+..+.+..............+..................+.+..+.........+....+..............+..........+..+...............+...+......+....+...........+.+...........+.........+....+..+............+.+.........+......+.....+...............+.+...............+...............+...........+...+.......+..+...+..................+.+...+.....+......+.+......+...............+...+......+......+..+......+......+.........+......+.....................+..........+.....+...+.+.........+.....+..........+.........+...........+.+...+...+...+..+..........+..+......................+...+.....+...+...+......+...+......+.+.....................+.....+.........+....+.........+.....+.+.........+.....+...+.................................+.+......+.........+...+...+..............+...............+.+.........+........+...+....+...............+......+.............................+................+.....+........................+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
..+...+....+......+.........+.........+...........+.+..............+....+..............+.+..+....+........+...+...+....+.....+.+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....+........+.+.....+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.........+......+..............+...+.....................+..........+.....+.+.....+.........+.+.....................+..............+.+.....+.........+...............+...+..........+...+..+.+...........+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-intca-1 ~]# openssl x509 -req -in intm.csr -sha512 -out intm.crt -CA root.crt -CAkey root.key -CAcreateserial -extensions ext -extfile <(printf '[ext]\nbasicConstraints=critical,CA:TRUE,pathlen:0')
Certificate request self-signature ok
subject=CN = Ext. Intm. CA
[root@aklimov-intca-1 ~]# openssl req -newkey rsa:4096 -subj '/CN=aklimov-intca-1.novalocal' -keyout aklimov-intca-1.novalocal.key -out aklimov-intca-1.novalocal.csr -nodes
.....+.............+.....+.........+.+...+..+.......+..+..........+.....+....+..+..................+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......................+..+.......+........+.+..............................+..+..........+..+.......+...+...........+....+...+..............+.........+............+...+.......+..+...+...+.......+...+...........+...............................+..+......................+.....+....+.....+......+.+.........+...+......+........................+........+...+.+...+........+.......+..+.........+.+.........+.....+......+...+..........+...+...+.........+.................+...+...+.......+.....+.......+..............................+...+..+.........+.......+.....+..........+...+.........+.....+.+..+..........+...+..+....+...+..+.+...+...........+...+.+.....+......+.+........+....+...........+..........+..+....+.........+......+......+.....+.......+......+...........+...+.+.....+....+...........+......+.............+.....+.......+...+..+...+.....................+....+.....+.............+...+.........+...+.........+...+........+.............+..+.........+.........+.........+............+.+...+............+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+.....+.............+...+.....+.......+..+...+...+...+.+......+.....+...+.+.....+.+........+.+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.........+..+.+..+...+.........+.............+..+.+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..........+...+......+..........................+....+.....+.+...........+....+......+...+.....................+.....+.+..+.......+...+.......................+....+.....+...................+.....+.+.....+....+...........+................+...............+.....+.........+.+..+............................+...+.....+...............+...............+.+......+..............+....+..+.......+.....+.+.........+............+...........+...+.......+..+............+.+............+.....+...+....+..............+....+..+...+..........+......+......+...+...+..+......+...............+.........+...+.......+..+.+.........+...+......+...........+...............+....+..+...+............+.......+......+.....+.........+...+.......+...........+.+...+...........+......+..........+........+.+..+...+.......+..+......+......+...+............+....+.........+...+..+.......+.....+....+..+.........+...+.......+...............+........+...+............+.+......+.....+.........+....+...+..+......+......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-intca-1 ~]# openssl x509 -req -in aklimov-intca-1.novalocal.csr -sha512 -out aklimov-intca-1.novalocal.crt -CA intm.crt -CAkey intm.key -CAcreateserial -extensions SAN -extfile <(printf '[SAN]\nsubjectAltName=DNS:aklimov-intca-1.novalocal')
Certificate request self-signature ok
subject=CN = aklimov-intca-1.novalocal
[root@aklimov-intca-1 ~]# cp aklimov-intca-1.novalocal.* /var/lib/icinga2/certs
[root@aklimov-intca-1 ~]# cat intm.crt >> /var/lib/icinga2/certs/aklimov-intca-1.novalocal.crt
[root@aklimov-intca-1 ~]# chown icinga: /var/lib/icinga2/certs/aklimov-intca-1.novalocal.*
[root@aklimov-intca-1 ~]# icinga2 daemon -C
[2023-06-21 15:53:03 +0000] information/cli: Icinga application loader (version: r2.13.7-1)
[2023-06-21 15:53:03 +0000] information/cli: Loading configuration file(s).
[2023-06-21 15:53:03 +0000] information/ConfigItem: Committing config item(s).
[2023-06-21 15:53:03 +0000] information/ApiListener: My API identity: aklimov-intca-1.novalocal
[2023-06-21 15:53:03 +0000] critical/SSL: Error loading and verifying locations in ca key file '/var/lib/icinga2/certs//ca.crt': 2147483650, "error:80000002:system library::No such file or directory"
[2023-06-21 15:53:03 +0000] critical/config: Error: Cannot make SSL context for cert path: '/var/lib/icinga2/certs//aklimov-intca-1.novalocal.crt' key path: '/var/lib/icinga2/certs//aklimov-intca-1.novalocal.key' ca path: '/var/lib/icinga2/certs//ca.crt'.
Location: in /etc/icinga2/features-enabled/api.conf: 5:1-5:24
/etc/icinga2/features-enabled/api.conf(3):  */
/etc/icinga2/features-enabled/api.conf(4):
/etc/icinga2/features-enabled/api.conf(5): object ApiListener "api" {
                                           ^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/features-enabled/api.conf(6):   //accept_config = false
/etc/icinga2/features-enabled/api.conf(7):   //accept_commands = false
[2023-06-21 15:53:03 +0000] critical/config: 1 error
[2023-06-21 15:53:03 +0000] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
[root@aklimov-intca-1 ~]# cp root.crt /var/lib/icinga2/certs/ca.crt
[root@aklimov-intca-1 ~]# icinga2 daemon -C
[2023-06-21 15:54:06 +0000] information/cli: Icinga application loader (version: r2.13.7-1)
[2023-06-21 15:54:06 +0000] information/cli: Loading configuration file(s).
[2023-06-21 15:54:06 +0000] information/ConfigItem: Committing config item(s).
[2023-06-21 15:54:06 +0000] information/ApiListener: My API identity: aklimov-intca-1.novalocal
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 12 Notifications.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 Host.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 3 Zones.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 244 CheckCommands.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 User.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 11 Services.
[2023-06-21 15:54:06 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2023-06-21 15:54:06 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-06-21 15:54:06 +0000] information/cli: Finished validating the configuration file(s).
[root@aklimov-intca-1 ~]#

[root@aklimov-intca-2 ~]# openssl req -newkey rsa:4096 -subj '/CN=aklimov-intca-2.novalocal' -keyout aklimov-intca-2.novalocal.key -out aklimov-intca-2.novalocal.csr -nodes
...........+.....+...+......+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+....+.........+..+..........+.....+...+......+.........+.......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+....+...........+..........+..............+...+.......+...+...........+........................+............+.............+...+.....+...+......+...+......+............+....+...+...+..+................+.................+.......+..+....+......+........+.+..+.........+...+.......+.....+....+...........+......+..........+...+...........+......+...........................+..........+..+..........+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.......+...+.+.........+..+...+....+..+.+........+.........+...+...+....+............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+........+....+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...........+...+.......+........+....+...+..+...............+.+...........+.........+......+......+.............+.................+...+............+.......+.....+.......+...............+...........+.......+...+...............+..+.+..+....+.....+...+.+......+........+.+..+....+..............+.+.....+..........+...+...+........+............+...+......+...................+...............+.........+..+.+..+......+.....................+.+.....+.......+...+...+.....+.........+..................+..........+........+.+........+............+.....................+..................+....+.....+..........+..+...+.+......+......+.........+...+..+.....................+....+...+...........+......+.........+....+........+.............+..+....+...+........+.........+.+........+......+...+..................+...+.+.....+......+..........+..+...+....+........+......+.+...+.....+.......+.....+..................+.+.....+......+................+...............+............+............+......+.....+....+...............+....................+................+.....+.........+.+.........+...............+.........+..+..........+...+.....+..........+..+..........+.........+...+..+.+........+.........+..........+......+..+.......+...........+......+.........+.+.....+.+.....+..........+..+.......+.........+..............................+............+...+...........+.+...+............+..+.............+..+...+....+.........+.........+......+..+....+.....+......+............+.............+...............+........+....+..........................+......+......+............+...+....+........+...+....+.........+......+.........+......+......+..+...+...+....+......+...............+...............+...............+..+...+..........+.....+.+.....+.........+.+.....+.+..+.............+........+.........+.+......+........+...............+...+............+.............+..+...+..............................+.......+..+.+..+.+......+...+......+..+..........+..+.+......+.....+.+........+.+...........+...+......+...+......+.+.....................+..+...+....+.....+......+.............+......+.....+.+..+......+.......+..............+......+..........+.....+.......+..+...+.....................+..........+.........+............+...........+.......+..................+............+..+....+..............+......+.....................+.+....................+.........+..........+..................+...+.....+......................+...+..+...+...+......+..........+.........+.....+......+.............+..+.+.................+.+.........+........+.............+......+...+............+........+.......+...+.....+......+......+....+..+.........+..................+..........+...+..+...............+......+................+...........+....+...+......+.........+..+...+..........+............+.........+..+...+.+...+..+..........+.....+....+............+..............+..................................+...............+...........+..........+.....+....+.........+...+.....+....+......+...+.....+....+...+...+...............+...............+..+.........+.........+......+.+.....+...+.+.....+...+..........+..+.+.....+...+....+......+......+...+.....+...+...+.+......+.....+............+.......+.........+..+.......+..+..........+...+...........+....+.................+.........+.+........+.............+..+............+...+......+....+..+.+............+..+......+...+....+...+...+..+...............+..........+...+..+....+...+...........................+......+........+.+...........+.......+...+.....+..................+...............+.+...+.....+......+.+........+...+.......+........+..........+........+............+.......+..+....+......+...+......+......+.....+...............+.+.........+.....+...+...+....+.........+.................+.............+......+..............+.+..............+.+.....+.+.......................+......+................+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-intca-2 ~]#

(copies aklimov-intca-2.novalocal.csr over)

[root@aklimov-intca-1 ~]# openssl x509 -req -in aklimov-intca-2.novalocal.csr -sha512 -out aklimov-intca-2.novalocal.crt -CA intm.crt -CAkey intm.key -CAcreateserial -extensions SAN -extfile <(printf '[SAN]\nsubjectAltName=DNS:aklimov-intca-2.novalocal')
Certificate request self-signature ok
subject=CN = aklimov-intca-2.novalocal
[root@aklimov-intca-1 ~]# cat intm.crt >> aklimov-intca-2.novalocal.crt
[root@aklimov-intca-1 ~]#

(copies aklimov-intca-2.novalocal.crt and root.crt over)

[root@aklimov-intca-2 ~]# cp aklimov-intca-2.novalocal.* /var/lib/icinga2/certs
[root@aklimov-intca-2 ~]# chown icinga: /var/lib/icinga2/certs/aklimov-intca-2.novalocal.*
[root@aklimov-intca-2 ~]# cp root.crt /var/lib/icinga2/certs/ca.crt
[root@aklimov-intca-2 ~]# icinga2 feature enable api
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
[root@aklimov-intca-2 ~]# icinga2 daemon -C
[2023-06-21 16:17:32 +0000] information/cli: Icinga application loader (version: r2.13.7-1)
[2023-06-21 16:17:32 +0000] information/cli: Loading configuration file(s).
[2023-06-21 16:17:32 +0000] information/ConfigItem: Committing config item(s).
[2023-06-21 16:17:32 +0000] information/ApiListener: My API identity: aklimov-intca-2.novalocal
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 12 Notifications.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 Host.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 3 Zones.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 244 CheckCommands.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 User.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 11 Services.
[2023-06-21 16:17:32 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2023-06-21 16:17:33 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-06-21 16:17:33 +0000] information/cli: Finished validating the configuration file(s).
[root@aklimov-intca-2 ~]# cat << EOF > /etc/icinga2/zones.conf
object Endpoint "aklimov-intca-1.novalocal" {
  host = "10.27.0.143"
}
object Endpoint "aklimov-intca-2.novalocal" {
  host = "10.27.0.184"
}
object Zone "master" {
  endpoints = [ "aklimov-intca-1.novalocal", "aklimov-intca-2.novalocal" ]
}
EOF
[root@aklimov-intca-2 ~]#

[root@aklimov-intca-1 ~]# cat << EOF > /etc/icinga2/zones.conf
object Endpoint "aklimov-intca-1.novalocal" {
  host = "10.27.0.143"
}
object Endpoint "aklimov-intca-2.novalocal" {
  host = "10.27.0.184"
}
object Zone "master" {
  endpoints = [ "aklimov-intca-1.novalocal", "aklimov-intca-2.novalocal" ]
}
EOF
[root@aklimov-intca-1 ~]#

Now, if I fire up the Icingas, they recognise each other. With config acceptance enabled in the API feature they even sync config to each other.

[root@aklimov-intca-2 ~]# cat /var/lib/icinga2/api/zones/master/_etc/my.conf
object Host "demo.kli.mov" {
	check_command = "dummy"
}
[root@aklimov-intca-2 ~]#

And... now what?

@sircubbi, you have already made PRs in this topic and seem to understand it IMAO. Depending on what you need, please do any of the following. The headings are independent of each other, but the bullets per heading should be done in their order. At best even one after the other's merged, to save worktime. Finally (or even before) indicate what you won’t (need and) do.

Intermediate CAs

• Add the fact that (and how) external intermediate CAs already work (see above) to the Icinga docs (force push try to allow certificate-chains #8859)
  • I did in Doc: Distributed Monitoring: add section "External CA/PKI" #9825
• Continue with PRs like process certificate chains presented by the client #9795 with focus on nicely handling fully external intermediate CAs (no CA keys in Icinga dirs)
• In a separate PR make Icinga-owned intermediate CA (see above) working – but I doubt you'll need/do this as int. CAs are likely to be fully external
• If you're really bored after all of the above, feel free to evaluate whether Icinga could do Intermediate CA for client auth does not work #7719 (comment) automagically (i.e. int. CA used as root CA, Icinga auto-cross-self-signs it in memory w/o its private key)

Multiple roots

A little off-topic, but you've already done something on it: #8859 (comment)

• Add the fact that (and how) multiple root CAs already work (see below) to the Icinga docs
  • I did in Doc: Distributed Monitoring: add section "External CA/PKI" #9825
• Test how well multiple root CAs (no intermediates!) already work in /var/lib/icinga2/certs/ca.crt. Either report success or make a new PR.

[Al2Klimov]

Multiple roots

A little off-topic, but you've already done something on it: #8859 (comment)

Test whether multiple root CAs (no intermediates!) already work in /var/lib/icinga2/certs/ca.crt.

80a405a hints that it doesn't work, but it just has the same problem with strange error messages as the intermediate stuff. Actually it seems to work. E.g. even with icinga2 api setup the following config appended to zones.conf won't successfully validate. ("unable to get local issuer")

for (cn in ["demo.kli.mov", "github.com"]) {
        object Endpoint cn {
                host = name
                port = 443
        }

        object Zone cn {
                endpoints = [ name ]
                parent = ZoneName
        }
}

But with the below ones appended to /var/lib/icinga2/certs/ca.crt it purrs like a cat:

[2023-06-22 10:51:44 +0200] information/ConfigItem: Activated all objects.
[2023-06-22 10:51:44 +0200] information/ApiListener: Reconnecting to endpoint 'github.com' via host 'github.com' and port '443'
[2023-06-22 10:51:44 +0200] information/ApiListener: Reconnecting to endpoint 'demo.kli.mov' via host 'demo.kli.mov' and port '443'
[2023-06-22 10:51:44 +0200] information/ApiListener: New client connection for identity 'github.com' to [140.82.121.4]:443
[2023-06-22 10:51:44 +0200] information/ApiListener: Sending config updates for endpoint 'github.com' in zone 'github.com'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished sending config file updates for endpoint 'github.com' in zone 'github.com'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Syncing runtime objects to endpoint 'github.com'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished reconnecting to endpoint 'github.com' via host 'github.com' and port '443'
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'github.com'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'github.com' in zone 'github.com'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Sending replay log for endpoint 'github.com' in zone 'github.com'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished sending replay log for endpoint 'github.com' in zone 'github.com'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished syncing endpoint 'github.com' in zone 'github.com'.
[2023-06-22 10:51:44 +0200] information/ApiListener: New client connection for identity 'demo.kli.mov' to [2a01:4f8:c2c:16ce::1]:443
[2023-06-22 10:51:44 +0200] information/ApiListener: Sending config updates for endpoint 'demo.kli.mov' in zone 'demo.kli.mov'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished reconnecting to endpoint 'demo.kli.mov' via host 'demo.kli.mov' and port '443'
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished sending config file updates for endpoint 'demo.kli.mov' in zone 'demo.kli.mov'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Syncing runtime objects to endpoint 'demo.kli.mov'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'demo.kli.mov'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'demo.kli.mov' in zone 'demo.kli.mov'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Sending replay log for endpoint 'demo.kli.mov' in zone 'demo.kli.mov'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished sending replay log for endpoint 'demo.kli.mov' in zone 'demo.kli.mov'.
[2023-06-22 10:51:44 +0200] information/ApiListener: Finished syncing endpoint 'demo.kli.mov' in zone 'demo.kli.mov'.

-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
QTAeFw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGExCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IENBMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4jvhEXLeqKTTo1eqUKKPC3eQyaKl7hLOllsB
CSDMAZOnTjC3U/dDxGkAV53ijSLdhwZAAIEJzs4bg7/fzTtxRuLWZscFs3YnFo97
nh6Vfe63SKMI2tavegw5BmV/Sl0fvBf4q77uKNd0f3p4mVmFaG5cIzJLv07A6Fpt
43C/dxC//AH2hdmoRBBYMql1GNXRor5H4idq9Joz+EkIYIvUX7Q6hL+hqkpMfT7P
T19sdl6gSzeRntwi5m3OFBqOasv+zbMUZBfHWymeMr/y7vrTC0LUq7dBMtoM1O/4
gdW7jVg/tRvoSSiicNoxBN33shbyTApOB6jtSj1etX+jkMOvJwIDAQABo2MwYTAO
BgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUA95QNVbR
TLtm8KPiGxvDl7I90VUwHwYDVR0jBBgwFoAUA95QNVbRTLtm8KPiGxvDl7I90VUw
DQYJKoZIhvcNAQEFBQADggEBAMucN6pIExIK+t1EnE9SsPTfrgT1eXkIoyQY/Esr
hMAtudXH/vTBH1jLuG2cenTnmCmrEbXjcKChzUyImZOMkXDiqw8cvpOp/2PV5Adg
06O/nVsJ8dWO41P0jmP6P6fbtGbfYmbW0W5BjfIttep3Sp+dWOIrWcBAI+0tKIJF
PnlUkiaY4IBIqDfv8NZ5YBberOgOzW6sRBc4L0na4UU+Krk2U886UAb3LujEV0ls
YSEY1QSteDwsOoBrp+uvFRTp2InBuThs4pFsiv9kuXclVzDAGySj4dzp30d8tbQk
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
-----END CERTIFICATE-----



-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

[Al2Klimov]

Test protocol

1. On both nodes run icinga2 api setup
2. Copy over /var/lib/icinga2/certs/ca.crt so that both nodes contain both root CAs
3. Edit /var/lib/icinga2/certs/ca.crt, prepend and append a 3rd and 4th Icinga root CA
4. mv /var/lib/icinga2/ca{,.bak}
5. Configure zones like in OP
6. Fire up Icinga

Result

[2023-06-22 09:29:34 +0000] information/ApiListener: New client connection for identity 'aklimov-mrca-1.novalocal' from [::ffff:10.27.3.208]:39616 (certificate validation failed: code 7: certificate signature failure)

Conclusion

At least multiple root CAs with the same name don't work.

[Al2Klimov]

Test protocol II

Same as above, but with distinct CA names 🙈

[root@aklimov-mrca-1 ~]# icinga2 feature enable api
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
[root@aklimov-mrca-1 ~]# openssl req -x509 -newkey rsa:4096 -subj '/CN=Ext. Root CA I' -md5 -keyout root1.key -out root1.crt -nodes
............+..+....+.....+.+..+.........................+..+....+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.....+....+......+...+...............+...............+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+............+...+.......................+..........+..+.............+...+......+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.....+....+......+.....................+...+..+......+.......+...+..+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+....+......+..+.......+...+..+.+....................+....+...+..+......+.+.....+.............+..+...............+...+.+......+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.............+.........+...+..................+........+....+...+...+............+........+.+......+..+.............+...+...........+...............+..................+.+......+.....+..........+.......................+.......+......+.....+...+.........+...+..................+.+...+...+........+......................+........+.....................+...............+...+..........+.....+.+..............+.+..+...+.+...+................................................+...+..+...+.........+.+..+.........+....+.....+....+.....+............+.+...............+.....+...+.......+..+...+....+........................+.........+..+....+............+...+....................+......+.+.....+.........+......+.........................+.................+......+.......+..+.+..+....+......+..+.......+.....+.+...+.........+........+.........+.........+.......+...+..+...............+...............+.......+....................+............+.+..+...+.......+.................+.+.........+...+...........+....+.....+.+.....+........................+...+....+......+.........+.....+.+............+........+.........+......+...+.........+.......+...+..................+...+..+....+...............+...+...+..............+..........+.........+..+.............+.........+...+..+.+...........+......+...+.+...+............+.....+.........+....+.....+............+....+............+...+..+..........+........+.........+.......+...........+....+......+...+..+.......+...+....................+...+....+...+...+............+.........+......+.....+..................+..........+.....+....+...+..+..........+..+....+.....+.......+..............+....+......+.....+............................+........+............+...............+.+..+............+..........+..+.+...+.....+............+..................+....+.................+.+.....+............+...+....+........+.....................+.+.....+.............+...+........+.+...........+...+.......+..............+....+.........+...........+.+...+............+.....+...+.............+........+.+..................+...+.....+...................+......+.....+...+.+.....+....+.....+..........+......+........+...............................+...........+....+..+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-mrca-1 ~]# openssl req -newkey rsa:4096 -subj '/CN=aklimov-mrca-1.novalocal' -keyout aklimov-mrca-1.novalocal.key -out aklimov-mrca-1.novalocal.csr -nodes
....+...+..+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*................+...+..........+..............+..........+...+..+.......+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+............+......+.+...............+.....+.+...+......+......+.....+..........+........+....+......+......+.........+.....+.........+...+...+.......+........+.......+..+......+.........+.......+.....+.............+.....................+..+...............+.............+............+...+..+.......+............+...+..+...............+...+.+.........+.....+......+.......+.....+............+.+...........+.........+.+...+.....+...+...+....+........+...+............+.......+.....+.......+....................+.+.............................+...+.......+...........+.......+.....+...+.........+.............+.....+.......+........+.......+........+.+..+...+....+.....+...+.............+.................+.............+......+.........+..+...........................+...+...+.+...........+....+..............+....+..+......+............+......+.+..............+.......+.........+......+..+............+.........+......+.+...+............+...+...............+.....+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+.............+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+............................+.........+............+...+....................+.+......+......+...............+...+...........+......+....+.....+....+.....+....+...+............+..+..................+.+.........+..+.+..+......+......+................+.....+.......+............+...+...+........+......+...+................+......+........+...+.......+..................+.....+............+.+.....+............+...+.+...+...........+.+.........+..+.......+...+........+......+...................+......+...+..+...+.......+........+.+..+..........+........+.+......+........+.......+.....+.........+.............+...+...+...............+..+.......+.....+.+.....+......+..................+.+..+......................+.................+......+.........+..........+...+......+..............+.+......+.....+......+....+......+..+......+......+.+...+..........................+.............+........+....+..............+......+.+..............+...+......+....+...+........+....+...+.....+....+...........+.............+.....+......+....+......+........+...+............+...+..........+.....+.............+......+.....+....+......+...........+.......+.....+............+.+...+............+........+....+..+.+............+...+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-mrca-1 ~]# openssl x509 -req -in aklimov-mrca-1.novalocal.csr -sha512 -out aklimov-mrca-1.novalocal.crt -CA root1.crt -CAkey root1.key -CAcreateserial -extensions SAN -extfile <(printf '[SAN]\nsubjectAltName=DNS:aklimov-mrca-1.novalocal')
Certificate request self-signature ok
subject=CN = aklimov-mrca-1.novalocal
[root@aklimov-mrca-1 ~]# openssl req -x509 -newkey rsa:4096 -subj '/CN=Ext. Root CA III' -md5 -keyout root3.key -out root3.crt -nodes
...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+..+.......+......+............+...+......+.....................+.........+.........+..............+.+..+....+...........+............+...+....+.....+.+..+......+............+....+...+...........+....+.................+...+...+...............+..................+....+...........+...............+...................+........+........................+...................+..+...+..........+...........+.........+.........+...+....+...+...+........+....+.....+.+..+.........+...+...+.......+.....+..........+.....+......+.+.........+.....+.+...........+.............+..+.........+...................+............+.........+..............+.........+............+.+........+...............+.........+.+..+.................................+.+..............+.........+..................+...+.+.........+............+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.+...+..+.............+..+.........+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...........+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........................................+..........+...+...............+..+...+......+...+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-mrca-1 ~]#

[root@aklimov-mrca-2 ~]# icinga2 feature enable api
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
[root@aklimov-mrca-2 ~]# openssl req -x509 -newkey rsa:4096 -subj '/CN=Ext. Root CA II' -md5 -keyout root2.key -out root2.crt -nodes
...+.....+....+......+....................+.+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+...+.......+...+..+......+...+......+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+......+.....+....+.....+.+......+..+...+........................+......+......................+.....+...+...+..........+.....+.+.....................+...+......+.....+..........+.........+..+.......+...........................+...+.....+.+......+........+.+......+.....+.............+.....+............+....+.........+........+....+...........+....+.........+..+....+...............+..................+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+..+.+.....+...+.+......+.....+..........+..+.......+........+.......+...+...+...+......+...+...+.................+.......+...+........+...+......+......+.......+..+.+.....+....+.....+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+......+.+......+...+.....+.+...+.....+...+.......+...........+....+..+.+..+.+......+.....+......+.......+...+...+.....+....+......+....................+....+...+.....+....+...+........+.......+...+.........+.....+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+....+......+..+....+...................................+....+..............+...+.........+.........+.....................+.+..+.+..+....+.........+..+....+..................+...+.........+.....+.........+......+..........+.....+.............+......+......+...+......+.....+......................+...+...........+...+......+.+...+.................+.+...+.....+.......+.................+.............+..+.+..+.......+..+.........+......+...+.+......+.....+.+.........+...........+.......+......+......+........+......+...............+.........+.+............+..+......+......................+...+..+..........+..+.......+.....+...+.......+.........+......+...+...........+.+......+...+......+...+..+......+...............+.......+..+.+..+...+.............+............+........+.+.....................+........+.+....................+......+...+...............................+...+........+...+....+...........+......+...+......+......+..........+......+...+...........+.+...+..............+.+......+...+.....+..........+..+...+.+............+........+.+...........+...+....+..+....+.........+...+.....+..................+....+.....................+.....+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-mrca-2 ~]# openssl req -newkey rsa:4096 -subj '/CN=aklimov-mrca-2.novalocal' -keyout aklimov-mrca-2.novalocal.key -out aklimov-mrca-2.novalocal.csr -nodes
...........+.....+......+.+...+......+...+...+.....+.......+..+....+..+...+......+.+..+.......+.....+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+.....+.+..+.......+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+..........+..................+........+..................+............+...+......+.+..+.+......+.....+...+......+......+...+...+....+...+....................+.......+..............+.........+.+............+.........+......+........+......+.+........+......+.........+............+.............+...........................+.........+.....+......+..................+.+.........+.....+....+........+....+...+......+.........+...........+.........+.+.....+.+........+.......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
......+...+..+.......+......+..+....+.........+..+.......+...+..+...+.........+...+.+.....+.+......+.....+...............+....+...........+...+....+..+...+..........+..+....+.................+...+.+........+.+.................+....+...+..................+........+......+...+.+.........+..+..........+......+.....+.........+.+...+......+.....+.............+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..........+..+.+.....+.+.....+......+..........+.........+...+...+...+.................+.......+.....+.+..+.......+...+..+...+..................+.......+..+.+......+.....+..........+...+...........................+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.................+...+...................+...+............+...+...+..............+.........+......+...+...+.............+............+...+.........+...........+.+......+..+.+......+...+...+.........+........+..................+..........+........+.......+..+.+...............+.....+.+.........+.....+.+...............+.....+...+............................+...+............+..+.......+.........+..............+...+.......+.................................+.........+...+........+............+...............+...+...+....+........+....+...+..+..........+.....+.........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-mrca-2 ~]# openssl x509 -req -in aklimov-mrca-2.novalocal.csr -sha512 -out aklimov-mrca-2.novalocal.crt -CA root2.crt -CAkey root2.key -CAcreateserial -extensions SAN -extfile <(printf '[SAN]\nsubjectAltName=DNS:aklimov-mrca-2.novalocal')
Certificate request self-signature ok
subject=CN = aklimov-mrca-2.novalocal
[root@aklimov-mrca-2 ~]# openssl req -x509 -newkey rsa:4096 -subj '/CN=Ext. Root CA IV' -md5 -keyout root4.key -out root4.crt -nodes
..............+.....+.+.....+......+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....................+...+..+...+.......+............+...+...+............+..+.+...........+...+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.................+.............+.....+..........+........+.........+..................+.+.........+..+...+.............+...+.........+...+............+........+...+.......+...............+..............+...............+.+..+.......+.........+..+...+.........+...+....+.....+......+.............+......+...+..+.+...............+.....+.......+.....+......+..........+.........+..+..........+...+.....+...+...+.......+...+.....+......+.+..+......+.+.....+....+......+....................+.........+................+........................+..+.........+.........+...............+.+........+..........+...............+.....+...............+...+....+......+...+..+.........+..........+.....+....+.....+....+...............+......+..+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
....+...+.....+.+.....+....+...+..+...............+.+..+...+.......+..+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......+.......+.....+...+.+...........+...+.......+.........+.....+...+...+..........+...+.....+.+...+..+....+.....+......+....+..+.......+..+.+.....+....+...........+...+......+.+..+.......+...+.....+..........+........+...+....+..+.+...+......+...............+......+.....+.........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.....+.+..+.+...........+.+..+.+...........+...+............+.............+..+.+...+....................+.......+............+............+..............+......+.+.....+..........+...+.........+......+...........................+........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[root@aklimov-mrca-2 ~]#

(copy over root*.crt)

[root@aklimov-mrca-1 ~]# cp aklimov-mrca-1.novalocal.* /var/lib/icinga2/certs
[root@aklimov-mrca-1 ~]# chown icinga: /var/lib/icinga2/certs/*
[root@aklimov-mrca-1 ~]# cat root3.crt root1.crt root2.crt root4.crt >/var/lib/icinga2/certs/ca.crt
[root@aklimov-mrca-1 ~]#

[root@aklimov-mrca-2 ~]# cp aklimov-mrca-2.novalocal.* /var/lib/icinga2/certs
[root@aklimov-mrca-2 ~]# chown icinga: /var/lib/icinga2/certs/*
[root@aklimov-mrca-2 ~]# cat root3.crt root1.crt root2.crt root4.crt >/var/lib/icinga2/certs/ca.crt
[root@aklimov-mrca-2 ~]#

[root@aklimov-mrca-1 ~]# cat << EOF > /etc/icinga2/zones.conf
object Endpoint "aklimov-mrca-1.novalocal" {
  host = "10.27.3.208"
}
object Endpoint "aklimov-mrca-2.novalocal" {
  host = "10.27.3.83"
}
object Zone "master" {
  endpoints = [ "aklimov-mrca-1.novalocal", "aklimov-mrca-2.novalocal" ]
}
EOF
[root@aklimov-mrca-1 ~]# systemctl start icinga2
[root@aklimov-mrca-1 ~]#

[root@aklimov-mrca-2 ~]# cat << EOF > /etc/icinga2/zones.conf
object Endpoint "aklimov-mrca-1.novalocal" {
  host = "10.27.3.208"
}
object Endpoint "aklimov-mrca-2.novalocal" {
  host = "10.27.3.83"
}
object Zone "master" {
  endpoints = [ "aklimov-mrca-1.novalocal", "aklimov-mrca-2.novalocal" ]
}
EOF
[root@aklimov-mrca-2 ~]# systemctl start icinga2
[root@aklimov-mrca-2 ~]#

Works. Just works.