Latest OVS Windows driver does not seem to be signed

[antoninbas]

Describe the bug
I am trying to install Antrea on Windows Nodes using the latest release (v1.14.1).
I followed the steps to Enable Loading of Test Signed Drivers:

Bcdedit.exe -set TESTSIGNING ON
Restart-Computer

However, when I try to install the OVS Windows driver, it fails:

PS C:\Windows\system32> Invoke-WebRequest -UseBasicParsing https://raw.githubusercontent.com/antrea-io/antrea/v1.14.1/hack/windows/Install-OVS.ps1 -o Install-OVS.ps1
PS C:\Windows\system32> .\Install-OVS.ps1 -InstallUserspace $false
12/15/2023 7:37 PM Installation log location: C:\Windows\system32\install_ovs.log

12/15/2023 7:37 PM Downloading OVS package from https://downloads.antrea.io/ovs/ovs-3.0.5-antrea.0-win64.zip to C:\Windows\system32\ovs-win64.zip

12/15/2023 7:37 PM Download OVS package success.

12/15/2023 7:37 PM Extract C:\Windows\system32\ovs-win64.zip to C:\Windows\system32

12/15/2023 7:37 PM Copying OVS package from C:\Windows\system32\openvswitch to C:\openvswitch

12/15/2023 7:37 PM Installing OVS driver certificate.



   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\TrustedPublisher

Thumbprint                                Subject
----------                                -------
DC433CFE4053A6300813C89BD44B5B4849EA6E86  CN="WDKTestCert appveyor,133304119867243994"


   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\Root

Thumbprint                                Subject
----------                                -------
DC433CFE4053A6300813C89BD44B5B4849EA6E86  CN="WDKTestCert appveyor,133304119867243994"
12/15/2023 7:37 PM Installing OVS kernel driver

12/15/2023 7:37 PM Hyper-V Virtual Machine Management service status: Running


C:\openvswitch\driver>netcfg -l .\ovsext.inf -c s -i OVSExt
Trying to install OVSExt ...

... failed. Error code: 0xe0000247.

Notice how https://downloads.antrea.io/ovs/ovs-3.0.5-antrea.0-win64.zip is used as the OVS package.

To Reproduce
Create a Windows Server 2019 VM, install prerequisites, try to install OVS drivers using .\Install-OVS.ps1 -InstallUserspace $false.

Expected
Driver installation should succeed.

Actual behavior
Notice this failure in the output:

C:\openvswitch\driver>netcfg -l .\ovsext.inf -c s -i OVSExt
Trying to install OVSExt ...

... failed. Error code: 0xe0000247.

Unfortunately the script thinks it "succeeds" and will proceed with the installation, which is not correct and is a bug in the script...

Versions:
Any Antrea version using the ovs-3.0.5-antrea.0-win64.zip package. That includes v1.14.1.

Additional context

When using https://downloads.antrea.io/ovs/ovs-2.15.2-antrea.0-win64.zip instead, driver installation succeeds.
I tested this using an older version of Antrea (v1.11.4):

PS C:\Windows\system32> Invoke-WebRequest -UseBasicParsing https://raw.githubusercontent.com/antrea-io/antrea/v1.11.4/hack/windows/Install-OVS.ps1 -o Install-OVS.ps1
PS C:\Windows\system32> .\Install-OVS.ps1
12/15/2023 7:33 PM Installation log location: C:\Windows\system32\install_ovs.log

12/15/2023 7:33 PM Downloading OVS package from https://downloads.antrea.io/ovs/ovs-2.15.2-antrea.0-win64.zip to C:\Windows\system32\ovs-win64.zip

12/15/2023 7:33 PM Download OVS package success.

12/15/2023 7:33 PM Extract C:\Windows\system32\ovs-win64.zip to C:\Windows\system32

12/15/2023 7:33 PM Copying OVS package from C:\Windows\system32\openvswitch to C:\openvswitch

12/15/2023 7:33 PM Installing OVS driver certificate.



   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\TrustedPublisher

Thumbprint                                Subject
----------                                -------
9E27F7B475263235DF5A75A4936D0B21460F4630  CN="WDKTestCert appveyor,132785990465004813"


   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\Root

Thumbprint                                Subject
----------                                -------
9E27F7B475263235DF5A75A4936D0B21460F4630  CN="WDKTestCert appveyor,132785990465004813"
12/15/2023 7:33 PM Installing OVS kernel driver

12/15/2023 7:33 PM Hyper-V Virtual Machine Management service status: Running


C:\openvswitch\driver>netcfg -l .\ovsext.inf -c s -i OVSExt
Trying to install OVSExt ...

... .\ovsext.inf was copied to C:\Windows\INF\oem84.inf.

... done.

[antoninbas]

@wenyingd @rajnkamr @NamanAg30 FYI

[antoninbas]

Note that it would also be good to improve the Install-OVS.ps1 script so that it fails early if OVS driver installation fails.

[wenyingd]

The opensource OVS driver is never signed. We need to enable test-signing mode on Windows Server if using the opensource OVS.

Update: I guess you are running Install-OVS.ps1 via a powershell over ssh connection? If yes, the error is not related with test-signing mode, it is because you are running the script without a GUI console. With a unsigned driver, Windows gives out a pop-up Window when the driver is installed ( for Windows Server 2019, the pop-up window is given at the first installation, for Windows Server 2022, the pop-up is given every time installing the driver). The pop-up window is to confirm the admin if the publisher of the driver is trusted or not. We didn't find a valid solution to avoid the pop-up window from silent installation.

[rajnkamr]

@wenyingd , looks like the driver installation is working fine in v1.11.4 even if connected via console

@antoninbas , In v1.11.4 and v1.14.1 , ovs driver installation /error is handled in similar way with error on exit https://github.com/antrea-io/antrea/blob/77cfa63ce53ef68f190c3205bd8ce8f331840df7/hack/windows/Install-OVS.ps1#L203C9-L203C53,

We might have to check wrt ovs version 3.0.5 which is used in v.14.1 is behaving differently !

[wenyingd]

Note: the difference is https://downloads.antrea.io/ovs/ovs-2.15.2-antrea.0-win64.zip has a certificate in the driver, I am confused why we provide driver cert in the opensource deliverables. Having sync with @XinShuYang , the driver cert is not updated for later OVS versions, so I doubt the root cause is the cert dismatches with the newer version of OVS driver.

[wenyingd]

Having made some tests, it is confirmed that the existing cert in OVS 3.0.5 deliverable is not matching the driver bits, Windows failed to verify the provided cert in the package, and gives out a pop-up window to require confirmation on if the publisher of OVS driver is trusted or not. If I replace the cert with a self-signed cert which was generated on the new driver bits (OVS 3.0.5), the pop-up window is not shown. Besides, the script can work with silent installation if running on GUI console, otherwise something wrong may happen when starting ovsdb-server service via a powershell console over ssh connections. This may be a different issue.

Back to the current issue, we can have two options to fix:

1. Provide a new self-signed cert in the deliverable when OVS driver is updated. The cert is generated by Antrea
2. Remove the package.cer file in the deliverables, and require user to must run the script on GUI console when installing OVS driver. Then the user can handle the popup window by trusting the opernsource OVS publisher.

Personally, I would prefer option 2, as I don't think it is a good idea to provide any "trusted opensource OVS" cert from Antrea perspective. @antoninbas @XinShuYang @rajnkamr

[XinShuYang]

Thanks @wenyingd for the investigation. I also prefer the second option because I don’t think Antrea community should maintain the ovs driver certificate.

[antoninbas]

If we require a GUI because there is a mandatory pop-up dialog box, this is a major usability issue IMO.
This prevents all kinds of automation, and it means I cannot use (for example) the AWS Core Windows images (which don't have a GUI and only provide Powershell access). These images are much more convenient to use, and I don't want to be forced to use an image with a GUI and have to create an RDP session (for each Windows Node) just to install the OVS driver.

So I definitely prefer option 1. This is what test-signing is (https://learn.microsoft.com/en-us/windows-hardware/drivers/install/how-to-test-sign-a-driver-package). My understanding is that test-signed is different from unsigned, and that seems to have created some confusion when I first opened this issue. We should have the appropriate warnings in our documentation of course, but that will be a major improvement in usability.

There may be another option, and I didn't look into it but if we can provide a script or a convenient way for users to self-sign the driver, I think that would achieve a similar purpose.

@rajnkamr the line you referenced didn't work in my experience (https://github.com/antrea-io/antrea/blob/77cfa63ce53ef68f190c3205bd8ce8f331840df7/hack/windows/Install-OVS.ps1#L203C9-L203C53), it didn't interrupt the installation.

[wenyingd]

If we require a GUI because there is a mandatory pop-up dialog box, this is a major usability issue IMO. This prevents all kinds of automation, and it means I cannot use (for example) the AWS Core Windows images (which don't have a GUI and only provide Powershell access). These images are much more convenient to use, and I don't want to be forced to use an image with a GUI and have to create an RDP session (for each Windows Node) just to install the OVS driver.

Hi @antoninbas , I don't prefer to provide an injected cert along with the OVS deliverables, because we can't guarantee the same provided cert is suitable for all Windows host (e.g., Server 2019, Server 2022,etc) and we can't make multiple certs for each kind of Windows OS.

If you think GUI installation is not as good and we must provide support for silent installation, we may need to search for the solutions to dynamically generate a self-signed cert on the target Server for the provided OVS driver.

BTW, this sentence is incorrect "I cannot use (for example) the AWS Core Windows images (which don't have a GUI and only provide Powershell access". For Windows core on AWS, as long as it provides a web console or supports RDP protocal, we can use it to install OVS, which doesn't block the pop-up window. The GUI is different from Windows desktop. I use this word just to present that the Windows popup window can't be delivered over SSH. We didn't require desktop, but we require a console to access Windows host.

[antoninbas]

If you think GUI installation is not as good and we must provide support for silent installation, we may need to search for the solutions to dynamically generate a self-signed cert on the target Server for the provided OVS driver.

Yes, I think that requiring a GUI is not as good, and prevents automation

BTW, this sentence is incorrect "I cannot use (for example) the AWS Core Windows images (which don't have a GUI and only provide Powershell access". For Windows core on AWS, as long as it provides a web console or supports RDP protocal, we can use it to install OVS, which doesn't block the pop-up window. The GUI is different from Windows desktop. I use this word just to present that the Windows popup window can't be delivered over SSH. We didn't require desktop, but we require a console to access Windows host.

You are right, I was still able to start RDP with Windows Core (which doesn't have a desktop) and when I ran Install-OVS.ps1, I got the pop up window and I was able to install OVS successfully.

[rajnkamr]

In previous versions, we have provided certificates alongside ovs driver and that works, however since certificate is windows platform os agnostic, we could update document and mention that driver can only be used on specific windows host with which it was packaged.

@antoninbas Error handling is provided, however it is not interrupting the installation, we could check if in this case, instead of silent exit it can throw some exception to let user know of driver installation failure.

[wenyingd]

Having double checked Install-OVS.ps1, I found that the existing script has provided the logic to auto-generate a cert for OVS driver (https://github.com/antrea-io/antrea/blob/main/hack/windows/Install-OVS.ps1#L158), the issue we hit is because an "existing" cert file was provided in the package, so the logic is not executed.

I tried on Windows Server 2019, if we remove the cert from the OVS deliverable, the logic will be called, and no popup window is given in this case. So I think the next step we can do is to remove the cert in the OVS deliverable, and update the files and footprint in antrea. @antoninbas @XinShuYang

[antoninbas]

So I think the next step we can do is to remove the cert in the OVS deliverable, and update the files and footprint in antrea

That sounds good to me. I am fine with either of these solutions:

1. Publish ovs-3.0.5-antrea.1-win64.zip without the cert, and update the download URL in Antrea scripts (with appropriate backporting)
2. Mutate ovs-3.0.5-antrea.0-win64.zip to remove the cert

@XinShuYang do you think you can take care of this?
cc @luolanzone

[antoninbas]

BTW, it may be good to have an extra log line in

antrea/hack/windows/Install-OVS.ps1

Lines 156 to 160 in 7702924

	if (!(Test-Path $CertificateFile)) {
	$ExportType = [System.Security.Cryptography.X509Certificates.X509ContentType]::Cert;
	$Cert = (Get-AuthenticodeSignature $DriverFile).SignerCertificate;
	[System.IO.File]::WriteAllBytes($CertificateFile, $Cert.Export($ExportType));
	}

, when we generate the certificate

[XinShuYang]

@XinShuYang do you think you can take care of this?

Yes, I will create a PR for this fix.