SSL Certificate: Invalid header type 60

[LunaticMuch]

I am seeing a weird message in my crawler logs

crawler_1  | 2022-02-13 09:14:31 Unknown: Network error: 4/7 github.mycomp.com:443/ failed: InternalException 
(HandshakeFailed (Error_Packet_Parsing "Failed reading: invalid header type: 60\nFrom:\theader\n\n"))

I am using GitHub enterprise, and I am interesting in a single repo which is at URL https://github.mycomp.com/tech-eng/myrepo
The config I am using is

workspaces:
  - name: myworkspace
    crawlers:
      - name: mycompany-crawler
        update_since: '2019-01-01'
        provider:
          github_organization: tech-eng
          github_url: https://github.mycomp.com
          github_token: GITHUB_TOKEN
          github_repositories:
            - myepo

And the token in the secrets is my valid token. I am wondering if it might be a certificate issue as the ssl certificate is not public, so the container might not be able to validate it. If so, is there any option I could use to accept invalid certificate? or to upload another cert authority?

[TristanCacqueray]

That seems related to haskell-tls/hs-tls#179 , and perhaps using openssl would help. I'll propose a change shortly.

[TristanCacqueray]

To use another cert authority, you need to create a ca-bundle.crt file and bind mount it to the container /etc/pki/tls/certs/. You can also try to define this environment variable: TLS_NO_VERIFY=1 .

[LunaticMuch]

I was mistaken. The certificate we are using is a valid certificate which does not require any additional non-public Root CA or Intermediate CA.

Is it possible the GraphQL endpoint used it wrong? Trying to figure out if it might be an issue not related to the library.

[TristanCacqueray]

The crawler is currently using a library named http-client-tls to handle TLS, and it's possible that this implementation is not able to handle your endpoint. Does setting TLS_NO_VERIFY=1 help?

Then #823 replaces the implementation with an openssl binding, which may also fix your issue.

[LunaticMuch]

Does setting TLS_NO_VERIFY=1 help?

It did not, if I used it correctly. I did this in my docker-compose.yml (actually I tried both with and without the double quotes)

  crawler:
    command: monocle crawler
    depends_on:
      - api
    env_file: ".secrets"
     environment:
       CONFIG: /etc/monocle/config.yaml
+      TLS_NO_VERIFY: "1"

[LunaticMuch]

Surely something has changed, but it has not been solved

error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type

The root cause might be different than what we thought initially. I am wondering if, for some reason, the container is trying to use proxy? Not sure. Running curl against my github, from inside the crawler, produces this:

* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (OUT), TLS alert, handshake failure (552):
* error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
* Closing connection 0
curl: (35) error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type

After some tests and researches, it seems to be this bug which relates to openssl and possible the load balancer.
There's a workaround, ie to lower the seclevel, but I need some time to figure out how it should be used in this context

[morucci]

Hi,

Perhaps, in the crawler container you could decrease the seclevel via: /etc/crypto-policies/back-ends/openssl.config

[LunaticMuch]

Hi,

Perhaps, in the crawler container you could decrease the seclevel via: /etc/crypto-policies/back-ends/openssl.config

Yes, I did it mounting the file from my local with just

@SECLEVEL=1

but I did not like it. I also tried the full string @SECLEVEL=1:kEECDH:kRSA:kEDH:kPSK:kDHEPSK:kECDHEPSK:kRSAPSK:-aDSS:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:-ARI still without luck

[TristanCacqueray]

Could you please confirm the issue is related to the curl and openssl provided by fedora 35, by running:

$ podman run -it --rm registry.fedoraproject.org/fedora:35
# dnf update -y
# curl https://github.mycomp.com

If that is the case, could you try with an older version, e.g. fedora:30.

Looking at strace -ff curl https://... 2>&1 | grep \/etc\/ , it seems like the library is also looking for setting in /etc/pki/tls/openssl.cnf, so it might be good to try mounting this config too.

[LunaticMuch]

Up to fedora v32 works. Above, it does not.
I just used fedora containers on my mac.

[TristanCacqueray]

So it seems like that may be relevant: https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
I'll propose a change to enable LEGACY mode shortly.

[TristanCacqueray]

Actually, it seems like LEGACY is a crypto-policy settings, so you might have to try: TLS_CIPHERS=@SECLEVEL=1:kEECDH:kRSA:kEDH:kPSK:kDHEPSK:kECDHEPSK:-aDSS:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:-ARIA:-AESCCM8, which is the content of podman run -it --rm registry.fedoraproject.org/fedora:32 cat /etc/crypto-policies/back-ends/openssl.config

[LunaticMuch]

Changing the cypher in the file seems not effective. I am wondering whether that's the correct way. I tried to find how the setting should be written on openssl.org but I did not find anything Google has too many options instead.

Crawler container does not have the command update-crypto-policies onboard.

I am trying to see whether I can confirm on a Fedora 33 which changes I need to make in order to get it working.

[morucci]

Hi,
To install the update-crypto-policies tool you could jump in the running container and install the needed package via:

docker exec -it cdd1726d9c4f /bin/bash
dnf install -y crypto-policies-scripts

[morucci]

Hi @LunaticMuch, our proposed fixes have been merged. You can now specify the TLS ciphers using TLS_CIPHERS env var. Please make sure to use last published containers.

I close the issue. Please re-open if needed.

[LunaticMuch]

Confirmed working. It seems that error is gone. There's a different issue now. I will debug it separately.