GZIP encoding broken in 2.8.0+ with Prometheus

[der-eismann]

What happened: We upgraded kube-state-metrics from 2.7.0 to 2.8.0. Afterwards Prometheus couldn't scrape it with the error being gzip: invalid checksum. Sounds similar to #1371.

What you expected to happen: kube-state-metrics should continue to be scraped as usual.

How to reproduce it (as minimally and precisely as possible):

• Start kube-state-metrics v2.8.0 with --enable-gzip-encoding
• Let it be scraped by Prometheus

Anything else we need to know?: Weirdly it works fine if I request the endpoint via curl & gunzip like

$ curl http://kube-state-metrics.kube-system.svc:8080/metrics -v  -H 'Accept-encoding: gzip' | gunzip -

Maybe the changed headers in #1974 are causing some complications?

Environment:

• kube-state-metrics version: 2.8.0 (2.8.2 affected as well)
• Kubernetes version (use kubectl version): 1.25.8
• Cloud provider or hardware configuration: AWS EC2

[mrueg]

/triage accepted

[mrueg]

Can you test #2037 and see if this fixes the issue?

[der-eismann]

Works fine 🙂 After building & running the container, the Prometheus scraping works again & curl also still works. Thanks for the quick fix!

[mrueg]

Thanks for testing and the quick turnaround!