Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add curlpp::Options::SslVerifyHost(0L) for ignore_cert_check flag #95

Merged
merged 2 commits into from
Oct 6, 2023

Conversation

TorrentialFire
Copy link
Contributor

@TorrentialFire TorrentialFire commented Oct 5, 2023

In addition to CURLOPT_SSL_VERIFYPEER, libcurl also has CURLOPT_SSL_VERIFYHOST which when set to 2L strictly checks the hostname provided against hostnames in the site's certificate. When running in insecure mode, this value should be set to 0L to allow a connection to a host not listed in the certificate (which is the case in some test MinIO deployment configurations).

I have a MinIO tenant deployed via the MinIO operator on a RKE2-based k8s deployment that I use for development and testing purposes. This configuration uses the default CA generated for the cluster, which is not a trusted CA. The executable generated by the build for src/tests/tests.cc fails on the first request because curlpp (libcurl) cannot verify the hostname of MinIO service (it is just an IP address).

This fix explicitly sets the value of CURLOPT_SSL_VERIFYHOST in both cases (secure/insecure). In the secure case, it is simply set to what is already the default value. Feel free to remove that line if you do not prefer explicit configuration.

References:
https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html
https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html

Copy link
Member

@balamurugana balamurugana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you run src/tests/tests.cc and what error did you get?

@TorrentialFire
Copy link
Contributor Author

How did you run src/tests/tests.cc and what error did you get?

I meant tests/tests.cc (it's not in src, my bad).

I built from source using the instructions in the readme. Running on a stock Ubuntu 20.04 distro with CMake 3.22.1.

wget --quiet -O vcpkg-master.zip https://github.com/microsoft/vcpkg/archive/refs/heads/master.zip
unzip -qq vcpkg-master.zip
./vcpkg-master/bootstrap-vcpkg.sh
./vcpkg-master/vcpkg integrate install
cmake -B ./build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=./vcpkg-master/scripts/buildsystems/vcpkg.cmake
cmake --build ./build --config Debug

Set environment vars as such:

export SERVER_ENDPOINT=https://<ip-address>:<port>/
export ACCESS_KEY=<minio-tenant-access-key>
export SECRET_KEY=<minio-tenant-secret-key>
export ENABLE_HTTPS=1
export IGNORE_CERT_CHECK=1
# pwd: /path/to/minio-cpp
./build/tests/tests

I don't have the exact output in front of me at the moment, but it did throw a runtime error from tests/tests.cc#L119 (bad response). The status code was 0 with no additional description.

I tested my configuration with MinIO's CLI tool mc (I verified with ~/.mc/config.json the the server address, port, and keys were all correct). I could connect and perform operations with the --insecure flag passed to mc.

@TorrentialFire
Copy link
Contributor Author

TorrentialFire commented Oct 6, 2023

It appears by default the MinIO operator deploys tenants with only HTTPS/SSL (443) ports exposed as services in the cluster. It is a unique situation of untrusted CA and host but needing to force SSL despite that. This is all on a private cluster with no services forwarded out of a closed network (and it's all for running some internal benchmarks and comparisons).

@balamurugana
Copy link
Member

@TorrentialFire I ran minio server with self-signed certificate like below

$ ./minio server ~/tmp/d{1..4}
Formatting 1st pool, 1 set(s), 4 drives per set.
WARNING: Host local has more than 2 drives of set. A host failure will result in data becoming unavailable.
MinIO Object Storage Server
Copyright: 2015-2023 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: DEVELOPMENT.2023-10-06T02-54-49Z (go1.21.1 linux/amd64)

Status:         4 Online, 0 Offline. 
S3-API: https://192.168.86.35:9000  https://192.168.124.1:9000  https://192.168.130.1:9000  https://127.0.0.1:9000     
RootUser: minio 
RootPass: minio123 

Console: https://192.168.86.35:46405 https://192.168.124.1:46405 https://192.168.130.1:46405 https://127.0.0.1:46405   
RootUser: minio 
RootPass: minio123 

Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
   $ mc alias set 'myminio' 'https://192.168.86.35:9000' 'minio' 'minio123'

Documentation: https://min.io/docs/minio/linux/index.html

and ran tests binary like below

$ SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minio SECRET_KEY=minio123 ENABLE_HTTPS=1 IGNORE_CERT_CHECK=1 ./build/tests/tests
MakeBucket()
RemoveBucket()
BucketExists()
ListBuckets()
StatObject()
RemoveObject()
DownloadObject()
GetObject()
ListObjects()
ListObjects() 1010 objects
PutObject()
CopyObject()
UploadObject()
RemoveObjects()
SelectObjectContent()
ListenBucketNotification()

Wonder why it is failing for you.

@TorrentialFire
Copy link
Contributor Author

TorrentialFire commented Oct 6, 2023

Can you dump the certificate information for your minio instance? I am interested in CN, alternative names, etc.

I will compare it with my test setup tomorrow and see what is different. I'd guess the issue is almost certainly a lack of domain name or IP address in my certificate (therefore libcurl can't verify the hostname, requiring this change for things to work).

@TorrentialFire
Copy link
Contributor Author

I did not modify the MinIO operator, tenant, or create custom CA/certs for my cluster. So, this seems like an issue out-of-the-box.

@balamurugana
Copy link
Member

Can you dump the certificate information for your minio instance? I am interested in CN, alternative names, etc.

I will compare it with my test setup tomorrow and see what is different. I'd guess the issue is almost certainly a lack of domain name or IP address in my certificate (therefore libcurl can't verify the hostname, requiring this change for things to work).

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:20:3e:ae:46:01:97:9c:3e:2c:52:b0:8c:f9:d1:0d
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = mkcert development CA, OU = bala@localhost.localdomain, CN = mkcert bala@localhost.localdomain
        Validity
            Not Before: Oct  8 12:40:53 2021 GMT
            Not After : Jan  8 12:40:53 2024 GMT
        Subject: O = mkcert development certificate, OU = bala@localhost.localdomain
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:bc:78:dc:8d:48:08:53:fd:b5:f7:34:e0:4b:95:
                    3a:3a:da:c8:04:c2:d2:0c:49:ee:98:02:71:c6:5e:
                    da:df:a4:1a:a7:7d:2b:0b:ed:9b:80:d2:1e:96:cb:
                    3e:0f:ff:37:49:fa:81:d7:84:61:88:9f:3f:ae:9e:
                    c1:06:a6:56:2f:6d:ae:3a:d7:74:b3:f1:7f:eb:06:
                    a9:ad:2c:35:20:1f:78:5b:f0:1d:1e:d7:be:61:14:
                    3b:49:9a:d4:97:a6:5f:0e:1b:3d:67:55:46:a5:58:
                    77:10:37:15:07:95:c0:7f:aa:a7:67:1b:1c:98:6a:
                    36:d5:f1:10:08:87:ee:24:c2:d4:f9:b3:02:4b:c0:
                    a0:a3:4d:f3:61:88:9a:34:1c:72:12:6d:2d:80:26:
                    56:82:99:29:76:bb:82:7e:eb:7b:21:37:ed:ab:0c:
                    26:4e:9f:51:ad:b2:1f:12:c6:81:45:60:4c:40:7c:
                    9d:8d:32:93:01:97:d0:26:b7:54:c5:eb:c5:91:b6:
                    aa:e6:f7:1b:d2:a8:c4:81:4c:ec:db:a0:e2:f6:fe:
                    e7:e3:09:d8:68:07:a6:ee:9a:ea:a9:55:35:7d:ba:
                    5a:f3:0d:25:fe:f8:18:5d:6a:8e:aa:55:33:92:18:
                    3d:1b:f1:85:95:b0:a2:49:93:40:e1:aa:86:ac:f7:
                    94:31
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Authority Key Identifier: 
                6C:C9:A8:9C:22:00:91:64:C3:47:F4:0F:7C:7A:B5:B7:11:2B:9E:1C
            X509v3 Subject Alternative Name: 
                DNS:localhost
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        44:e0:b9:b7:84:22:21:37:f3:ca:9e:f5:e7:29:65:b6:dc:46:
        9a:2c:61:dd:55:e8:44:7b:bb:cc:b7:84:fd:ee:cd:9e:95:ae:
        4f:cb:37:1a:45:fb:c4:7a:10:e7:42:2e:c2:15:f7:b4:83:71:
        73:49:61:9f:f0:0b:d7:2a:61:8e:f4:8d:ee:ad:17:9a:b4:a8:
        21:93:23:b8:fa:77:cc:be:89:d3:d1:51:19:46:ec:1d:60:49:
        4d:23:92:c0:55:9b:db:8b:7b:01:7d:97:fb:d2:c9:ea:cd:1f:
        09:8c:f4:5a:2d:35:47:fa:2a:48:9a:d5:df:ec:eb:94:db:86:
        fa:e2:9a:4a:86:23:0b:6a:1b:84:85:a3:6e:dc:d5:d6:af:0a:
        ce:16:08:e8:fb:b2:99:f1:bb:72:c0:80:fe:c1:c7:ac:20:f9:
        8e:49:65:8c:03:82:ab:f6:99:54:4d:9d:7e:60:0d:9f:31:88:
        73:b8:f9:80:83:ff:05:ea:8a:af:4a:36:ce:59:07:fb:ab:2e:
        fc:9e:b5:eb:45:2d:ae:57:a9:36:8f:cf:f0:50:17:35:a5:f2:
        b4:19:8a:8d:73:ec:cc:b5:bc:51:5d:5e:69:50:f3:74:98:f2:
        cb:4a:30:8c:ae:0e:38:1e:85:21:af:c9:b7:4b:ec:dd:9c:37:
        11:ac:c0:67:05:f4:e4:0e:06:e5:3a:5a:83:c0:c8:3e:61:5d:
        cc:0f:4e:01:eb:b7:69:27:f3:43:28:2c:77:82:8a:83:62:ca:
        bc:f4:77:29:5d:08:1b:7c:7b:ef:1e:82:dd:5c:49:80:90:08:
        bd:2d:6b:52:2c:a9:0c:9c:06:47:d6:0c:61:f6:6d:cb:1d:e5:
        fd:06:c3:ad:8f:33:02:76:79:67:47:5b:7f:8e:36:00:9a:70:
        f4:d3:b7:76:c1:4f:44:79:bb:20:b5:e8:47:b8:e4:0b:e4:36:
        6f:2e:46:72:56:a1:50:a5:05:23:a3:8c:f7:36:98:f5:f2:7f:
        b5:30:bc:7a:90:6b

If required information is missing, it is not minio-cpp issue.

@dvaldivia
Copy link

In Kubernetes we sign the certificates for multiple SANs, such as:

  • *.my-tenant-hl.ns-1.svc.cluster.local
  • my-tenant-pool-0-{0...3}.my-tenant-hl.ns-1.svc.cluster.local
  • minio.ns-1.svc.cluster.local
  • minio.ns-1
  • minio.ns-1.svc
  • *.my-tenant-hl.ns-1.svc.cluster.local
  • *.ns-1.svc.cluster.local

so even if the name is being checked, as long as you came through any of these you should be fine, but I think what you want is to access the tenant via a different name (ie: localhost) and not have it verify the name right @TorrentialFire ?

Here's a sample cert for a tenant called my-tenant on ns-1

-----BEGIN CERTIFICATE-----
MIIDSDCCAjCgAwIBAgIQLsnO9f8nvct7XetM1xE1ajANBgkqhkiG9w0BAQsFADAV
MRMwEQYDVQQDEwprdWJlcm5ldGVzMB4XDTIzMTAwNjA0MzkwNloXDTI0MTAwNTA0
MzkwNlowUzEVMBMGA1UEChMMc3lzdGVtOm5vZGVzMTowOAYDVQQDDDFzeXN0ZW06
bm9kZToqLm15LXRlbmFudC1obC5ucy0xLnN2Yy5jbHVzdGVyLmxvY2FsMFkwEwYH
KoZIzj0CAQYIKoZIzj0DAQcDQgAEa/g7d+HqiSphD8uKvKzZ+cRpePRdZJ6Vkr7X
3X5Y4mTObGATMjldJJNbYcR2QRq6CUuWc5j+EfIwHEnM1Wkc8KOCAR8wggEbMA4G
A1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAA
MB8GA1UdIwQYMBaAFF3i5Z0eKMEDPI/c+vK1y67zQqjrMIHEBgNVHREEgbwwgbmC
PG15LXRlbmFudC1wb29sLTAtezAuLi4zfS5teS10ZW5hbnQtaGwubnMtMS5zdmMu
Y2x1c3Rlci5sb2NhbIIcbWluaW8ubnMtMS5zdmMuY2x1c3Rlci5sb2NhbIIKbWlu
aW8ubnMtMYIObWluaW8ubnMtMS5zdmOCJSoubXktdGVuYW50LWhsLm5zLTEuc3Zj
LmNsdXN0ZXIubG9jYWyCGCoubnMtMS5zdmMuY2x1c3Rlci5sb2NhbDANBgkqhkiG
9w0BAQsFAAOCAQEAgKhO1ro20bg/zrgHfVfXqQlOoU2ZFbRuC1rm5+V8S86T+BSx
x2DKeQGpyElmm9FYCIP6Q8TENtPQn8zmK4DLEcIIuyyADvikpAu/J/sfae4/S0xh
gvnAxc2HC/LaW8sdGI7sc/WLkmeUQPRRoxGCuvaju/vgeVh42J7wJUQZTMwvuiCk
0RR2AtXXI+I/4Yp+/GbuI0Jt2F2+OjL1GOtfjR7p+yuRM6N6EaHlIwFFb4A1S2qR
QZTUF9YR238z02uKQ+5JiJFnU3gEcp6UM9pBhG5hecg4aOMk59yNmC6ujm37a0U1
A3HPK4RcINaOlAFXcSH1XxvwToQClkA35dpiHA==
-----END CERTIFICATE-----

@TorrentialFire
Copy link
Contributor Author

TorrentialFire commented Oct 6, 2023

@dvaldivia correct, I am accessing the service from just outside of the cluster, the host is running a set of VMs on a libvirt virtual switch. I reach the service at the IP address of one of the cluster nodes.

I disagree with @balamurugana. The lack of proper ingress and ssl termination is an issue with a cert with matching subject or SAN but in the insecure case I think this is a minio-cpp issue because the user doesn't care of the hostname matches. The flag is IGNORE_CERT_CHECK, but because CURLOPT_SSL_VERIFYHOST defaults to 2L, minio-cpp is actually still checking the cert for a subject or SAN that matches.

As a user, if I'm asking to use SSL without verifying the cert, I accept the consequences. If I decide I want the full protection of SSL with trusted CA and proper hostname/SAN verification, I'll cross that bridge. The client in minio-cpp should allow the insecure case and enforce the secure case based on IGNORE_CERT_CHECK. It currently does not.

@TorrentialFire
Copy link
Contributor Author

For reference, here is the certificate info for my setup.

Kubernetes, exposed services:

$ kubectl get svc

Output:

NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes              ClusterIP      10.43.0.1       <none>        443/TCP          3d16h
minio                   LoadBalancer   10.43.143.108   <pending>     443:30942/TCP    2d18h
test-tenant-console     LoadBalancer   10.43.93.184    <pending>     9443:30318/TCP   2d18h
test-tenant-hl          ClusterIP      None            <none>        9000/TCP         2d18h

As previously mentioned, I make all my requests from the machine which is hosting a set of guest VMs on a virtual switch. One of the nodes on the cluster has an IP of 192.168.122.82 on this network.

Certificate info:

$ echo | openssl s_client -showcerts -servername 192.168.122.82 -connect 192.168.122.82:30942 2>/dev/null | openssl x509 -inform pem -noout -text

Output:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            51:75:ab:ea:10:4a:8d:36:18:7f:79:32:c3:79:37:bd
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = rke2-server-ca@1696279571
        Validity
            Not Before: Oct  3 19:31:53 2023 GMT
            Not After : Oct  2 19:31:53 2024 GMT
        Subject: O = system:nodes, CN = system:node:*.test-tenant-hl.default.svc.cluster.local
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:34:ff:95:6a:84:69:0e:b9:9b:5a:7a:d6:38:b9:
                    4f:c8:bd:74:f2:8d:3d:bb:0c:8f:59:22:eb:6d:89:
                    a2:c9:96:f6:65:48:ab:35:1d:cb:95:5a:f8:45:00:
                    a5:12:af:58:ff:2b:21:3b:28:75:44:ae:3e:5d:76:
                    45:ec:b4:a2:ef
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                EA:8C:68:21:90:8A:33:F4:31:BD:FB:B3:62:1E:4D:7C:85:B0:29:66
            X509v3 Subject Alternative Name:
                DNS:test-tenant-pool-0-{0...3}.test-tenant-hl.default.svc.cluster.local, DNS:minio.default.svc.cluster.local, DNS:minio.default, DNS:minio.default.svc, DNS:*.test-tenant-hl.default.svc.cluster.local, DNS:*.default.svc.cluster.local
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:44:02:20:06:2d:ab:00:bc:f2:22:2f:95:db:10:0a:0c:6a:
        7b:99:db:b5:48:12:2a:93:30:a8:10:7e:85:1a:fa:9f:27:b8:
        02:20:3d:15:c0:f1:72:54:d8:82:d9:3e:67:10:f1:1a:77:9a:
        ae:f9:b1:2e:77:34:18:4f:1a:bd:6c:bc:3a:c9:d0:1b

Neither the CN or the SAN's would be available outside of the cluster. This is the default configuration created by the MinIO operator. I could deploy my minio-cpp code into the cluster, but for initial testing and development that is more cumbersome than just compiling and running.

@TorrentialFire
Copy link
Contributor Author

I briefly mentioned in one of my earlier responses that the MinIO CLI mc when using the --insecure flag allows this connection using the IP address of the cluster node. In this regard, minio-cpp does not behave identically (since the connection instead fails with status code 0).

We could look into potential ways of determining if curlpp/libcurl is failing the request due to hostname verification and report that to the user. This would be better than status code 0 with no message (which is the current behavior for this error).

src/http.cc Outdated Show resolved Hide resolved
@balamurugana balamurugana changed the title fix SslVerifyHost (0L) flag required for insecure SSL connections Add curlpp::Options::SslVerifyHost(0L) for ignore_cert_check flag Oct 6, 2023
@balamurugana balamurugana merged commit 0e9183f into minio:main Oct 6, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants