[kube-prometheus-stack] remoteWrite does no effect

[silencePR]

Describe the bug a clear and concise description of what the bug is.

Using remoteWrite feature does not cause prometheus write data to the remote location.

What's your helm version?

version.BuildInfo{Version:"v3.14.2", GitCommit:"c309b6f0ff63856811846ce18f3bdc93d2b4d54b", GitTreeState:"clean", GoVersion:"go1.21.7"}

What's your kubectl version?

Client Version: v1.29.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.3

Which chart?

kube-prometheus-stack

What's the chart version?

58.0.0

What happened?

I declared "remote write" in values.yaml, upgraded kube-prometheus-stack (with restarting pods).
Metrics are available from local source, just like before, no data in remote database.
I'm sure deployment has accepted configuration, because I've check it like this:

user@101-kuball:~/servers/kube-prometheus-stack$ helm get values kube-prometheus-stack -n monitoring
(...)

    remoteWrite:
    - basicAuth:
        password:
          key: pass
          name: influx-anarti
        username:
          key: user
          name: influx-anarti
      url: https://xxx.yyy.net:8086/api/v1/prom/write?db=importprometheus
prometheusOperator:
  enabled: true

What you expected to happen?

I would like data flow into remote Influx database.

How to reproduce it?

place into values.yaml
remoteWrite:
- url: https://xxx.yyy:8086/api/v1/prom/write?db=database

Enter the changed values of values.yaml?

alertmanager:
  enabled: false
grafana:
  enabled: false
kubeStateMetrics:
  enabled: true
prometheusOperator:
  enabled: true

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: https://xxx.yyy.net:8086/api/v1/prom/write?db=importprometheus
        basicAuth:
           username:
             name: influx-anarti
             key: user
           password:
             name: influx-anarti
             key: pass
        #tls_config:
        #  insecure_skip_verify: true
    nodeSelector:
      prometheus: "true"
    additionalScrapeConfigs:
      -  job_name: pod-k8s
         honor_timestamps: true
         scrape_interval: 5s
         scrape_timeout: 5s
         metrics_path: /actuator/prometheus
         scheme: http
         relabel_configs:
         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
           separator: ;
           regex: "true"
           replacement: $1
           action: keep
         - source_labels: [__meta_kubernetes_service_name]
           separator: ;
           regex: (.*)
           target_label: service
           replacement: $1
           action: replace
         - source_labels: [__meta_kubernetes_pod_container_name]
           separator: ;
           regex: (.*)
           target_label: container
           replacement: $1
           action: replace
         - source_labels: [__address__]
           separator: ;
           regex: ([^:;]+):(\d+)
           target_label: __address__
           replacement: ${1}:8081
           action: replace
         kubernetes_sd_configs:
         - role: pod
      -  job_name: gpu-k8s
         honor_timestamps: true
         scrape_interval: 5s
         scrape_timeout: 5s
         metrics_path: /metrics
         scheme: http
         relabel_configs:
         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
           separator: ;
           regex: "true"
           replacement: $1
           action: keep
         - source_labels: [__meta_kubernetes_service_name]
           separator: ;
           regex: (.*)
           target_label: service
           replacement: $1
           action: replace
         - source_labels: [__meta_kubernetes_pod_container_name]
           separator: ;
           regex: (.*)
           target_label: container
           replacement: $1
           action: replace
         - source_labels: [__address__]
           separator: ;
           regex: ([^:;]+):(\d+)
           target_label: __address__
           replacement: ${1}:9400
           action: replace
         kubernetes_sd_configs:
         - role: pod

Enter the command that you execute and failing/misfunctioning.

helm upgrade --recreate-pods kube-prometheus-stack prometheus-community/kube-prometheus-stack -f values.yaml -n monitoring --version 58.0.0

Anything else we need to know?

The connection is via nginx proxy. Analyzing logs I can tell, there's no signs of connection, even ended with some "bad" http code.
I've tried with curl from all k8s nodes, it occurred in logs, so there is no firewall issue.

[darshandeshmukh11]

I could reproduce this issue with below config -

version: 66.3.1
appVersion: v0.78.2

Helm Version -

WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /opt/kube_config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /opt/kube_config
version.BuildInfo{Version:"v3.13.2", GitCommit:"2a2fb3b98829f1e0be6fb18af2f6599e0f4e8243", GitTreeState:"clean", GoVersion:"go1.20.10"}

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: "https://monitoring-central.abc.com/api/v1/push"
        headers:
          X-Scope-OrgID: devops
        basic_auth:
           username: admin
           password: 3$dsdsd-dsd

Logs from prometheus pod -

$ kubectl logs prometheus-kube-prometheus-kube-prome-prometheus-0 -n monitoring-remote
pod service account via in-cluster config"
ts=2024-12-09T08:48:29.199Z caller=kubernetes.go:331 level=info component="discovery manager notify" discovery=kubernetes config=config-0 msg="Using pod service account via in-cluster config"
ts=2024-12-09T08:48:29.202Z caller=manager.go:164 level=info component="rule manager" msg="Starting rule manager..."
ts=2024-12-09T08:48:29.298Z caller=main.go:1452 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=111.246207ms db_storage=1.291µs remote_storage=193.901µs web_handler=469ns query_engine=750ns scrape=90.61µs scrape_sd=1.924761ms notify=21.56µs notify_sd=476.215µs rules=98.746136ms tracing=8.43µs
ts=2024-12-09T08:48:35.160Z caller=dedupe.go:112 component=remote level=info remote_name=43a5e0 url=https://monitoring-central.abc.com/api/v1/push msg="Done replaying WAL" duration=6.044983732s

But I dont see any other logs after this and data is missing in my remote mimir/minio setup.