metrics endpoint will not be started because metrics-address was not specified #1120

jisnardo · 2020-05-29T10:44:53Z

Describe the bug

Unable to get metrics from nodeplugin / provisioner.

Environment details

k8s v1.16
docker v18.6.3
csi-rbd v2.1.1 (latest available)
Ubuntu 18.04.4 LTS 4.15.0-101-generic

Steps to reproduce

Deploy the app.

NAME                                                   READY   STATUS    RESTARTS   AGE
pod/csi-rbd-ceph-csi-rbd-nodeplugin-bg4rn              3/3     Running   0          27h
pod/csi-rbd-ceph-csi-rbd-nodeplugin-rc4xf              3/3     Running   0          27h
pod/csi-rbd-ceph-csi-rbd-nodeplugin-sdz2b              3/3     Running   0          27h
pod/csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8-8thws   6/6     Running   0          27h
pod/csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8-pmhzm   6/6     Running   1          27h
pod/csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8-qctwh   6/6     Running   0          7h43m

NAME                                                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/csi-rbd-ceph-csi-rbd-nodeplugin-http-metrics    ClusterIP   10.90.8.59     <none>        8080/TCP   78d
service/csi-rbd-ceph-csi-rbd-provisioner-http-metrics   ClusterIP   10.90.182.75   <none>        8080/TCP   78d

NAME                                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/csi-rbd-ceph-csi-rbd-nodeplugin   3         3         3       3            3           <none>          78d

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/csi-rbd-ceph-csi-rbd-provisioner   3/3     3            3           78d

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/csi-rbd-ceph-csi-rbd-provisioner-549bf5fbf    0         0         0       2d20h
replicaset.apps/csi-rbd-ceph-csi-rbd-provisioner-66cb99d9b8   0         0         0       27h
replicaset.apps/csi-rbd-ceph-csi-rbd-provisioner-7f97889cb4   0         0         0       2d20h
replicaset.apps/csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8    3         3         3       4d1h

Port forward to the pod / service.

kworkspace -n csi-rbd port-forward pod/csi-rbd-ceph-csi-rbd-nodeplugin-bg4rn 9000:8080
Forwarding from 127.0.0.1:9000 -> 8080
Forwarding from [::1]:9000 -> 8080
Handling connection for 9000

Get metrics.

curl localhost:9000/metrics
curl: (52) Empty reply from server

Fail.

E0529 12:31:41.632577   32278 portforward.go:400] an error occurred forwarding 9000 -> 8080: error forwarding port 8080 to pod 955b0dab9f51e92102b50ae0f419025bad0af7673b56a0a2a13e80738c4211c7, uid : exit status 1: 2020/05/29 12:31:41 socat[6002] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused

Actual results

Unable to get metrics from nodeplugin / provisioner.

Expected behavior

curl localhost:9000/metrics and get the metrics.

Logs

kworkspace -n csi-rbd logs csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8-z7k6n csi-attacher

I0525 08:11:07.949454       1 main.go:91] Version: v2.1.1-0-g57193d79
I0525 08:11:07.952374       1 connection.go:153] Connecting to unix:///csi/csi-provisioner.sock
I0525 08:11:08.954524       1 common.go:111] Probing CSI driver for readiness
I0525 08:11:08.954587       1 connection.go:182] GRPC call: /csi.v1.Identity/Probe
I0525 08:11:08.954612       1 connection.go:183] GRPC request: {}
I0525 08:11:08.965213       1 connection.go:185] GRPC response: {}
I0525 08:11:08.966581       1 connection.go:186] GRPC error: <nil>
I0525 08:11:08.966641       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0525 08:11:08.966676       1 connection.go:183] GRPC request: {}
I0525 08:11:08.970928       1 connection.go:185] GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v2.1.1"}
I0525 08:11:08.973737       1 connection.go:186] GRPC error: <nil>
I0525 08:11:08.973758       1 main.go:136] CSI driver name: "rbd.csi.ceph.com"

This is the main clue i have found.

W0525 08:11:08.973783       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.

helm chart values

All related to metrics as default.

nodeplugin:
  name: nodeplugin
  # if you are using rbd-nbd client set this value to OnDelete
  updateStrategy: RollingUpdate

  httpMetrics:
    # Metrics only available for cephcsi/cephcsi => 1.2.0
    # Specifies whether http metrics should be exposed
    enabled: true
    # The port of the container to expose the metrics
    containerPort: 8080

    service:
      # Specifies whether a service should be created for the metrics
      enabled: true
      # The port to use for the service
      servicePort: 8080
      type: ClusterIP

      # Annotations for the service
      # Example:
      # annotations:
      #   prometheus.io/scrape: "true"
      #   prometheus.io/port: "8080"
      annotations: {}

      clusterIP: ""

      ## List of IP addresses at which the stats-exporter service is available
      ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
      ##
      externalIPs: []

      loadBalancerIP: ""
      loadBalancerSourceRanges: []

...

provisioner:
  name: provisioner
  replicaCount: 3
  # Timeout for waiting for creation or deletion of a volume
  timeout: 60s

  httpMetrics:
    # Metrics only available for cephcsi/cephcsi => 1.2.0
    # Specifies whether http metrics should be exposed
    enabled: true
    # The port of the container to expose the metrics
    containerPort: 8080

    service:
      # Specifies whether a service should be created for the metrics
      enabled: true
      # The port to use for the service
      servicePort: 8080
      type: ClusterIP

      # Annotations for the service
      # Example:
      # annotations:
      #   prometheus.io/scrape: "true"
      #   prometheus.io/port: "8080"
      annotations: {}

      clusterIP: ""

      ## List of IP addresses at which the stats-exporter service is available
      ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
      ##
      externalIPs: []

      loadBalancerIP: ""
      loadBalancerSourceRanges: []

Additional context

n/a

Thanks in advance

The text was updated successfully, but these errors were encountered:

Madhu-1 · 2020-05-29T11:42:27Z

nodeplugin uses the hostnetwork can you please verify it once using hostIP and port instead of localhost

jisnardo · 2020-05-29T12:46:00Z

The test using host ip and port instead of localhost outputs the same result.

Madhu-1 · 2020-07-01T07:29:51Z

E0529 12:31:41.632577 32278 portforward.go:400] an error occurred forwarding 9000 -> 8080: error forwarding port 8080 to pod 955b0dab9f51e92102b50ae0f419025bad0af7673b56a0a2a13e80738c4211c7, uid : exit status 1: 2020/05/29 12:31:41 socat[6002] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused

looks to be an issue in kubernetes itself not in CSI, the kubernetes need to handle the port forwarding

This is the main clue i have found.
W0525 08:11:08.973783 1 metrics.go:142] metrics endpoint will not be started because metrics-address was not specified.

we are not starting any mertrics server in sidecar containers

HaveFun83 · 2020-07-15T08:24:05Z

This is the main clue i have found.
W0525 08:11:08.973783 1 metrics.go:142] metrics endpoint will not be started because metrics-address was not specified.

we are not starting any mertrics server in sidecar containers

@Madhu-1 What about this one here?
https://github.com/ceph/ceph-csi/blob/master/charts/ceph-csi-rbd/templates/provisioner-deployment.yaml#L159
https://github.com/ceph/ceph-csi/blob/master/charts/ceph-csi-rbd/templates/nodeplugin-daemonset.yaml#L130

Madhu-1 · 2020-09-07T07:23:21Z

This is the main clue i have found.
W0525 08:11:08.973783 1 metrics.go:142] metrics endpoint will not be started because metrics-address was not specified.

we are not starting any mertrics server in sidecar containers

@Madhu-1 What about this one here?
https://github.com/ceph/ceph-csi/blob/master/charts/ceph-csi-rbd/templates/provisioner-deployment.yaml#L159
https://github.com/ceph/ceph-csi/blob/master/charts/ceph-csi-rbd/templates/nodeplugin-daemonset.yaml#L130

This is the liveness sidecar which belongs to csi, not for the kubernetes sidecar containers, @jisnardo can you please retest this with v3.1.0?

humblec · 2020-09-29T09:12:08Z

@jisnardo here is the feature request to consume CSI liveness sidecar #1096 .. we are not using it atm.

jisnardo · 2020-09-30T07:11:01Z

Thanks @Madhu-1 @humblec

stale · 2021-07-21T04:50:12Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions · 2021-09-11T21:05:08Z

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

nixpanic added the component/deployment Helm chart, kubernetes templates and configuration Issues/PRs label Jun 2, 2020

humblec assigned Yuggupta27 Sep 29, 2020

Yuggupta27 removed their assignment Feb 19, 2021

stale bot added the wontfix This will not be worked on label Jul 21, 2021

github-actions bot closed this as completed Sep 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metrics endpoint will not be started because metrics-address was not specified #1120

metrics endpoint will not be started because metrics-address was not specified #1120

jisnardo commented May 29, 2020

Madhu-1 commented May 29, 2020

jisnardo commented May 29, 2020

Madhu-1 commented Jul 1, 2020

HaveFun83 commented Jul 15, 2020

Madhu-1 commented Sep 7, 2020

humblec commented Sep 29, 2020

jisnardo commented Sep 30, 2020

stale bot commented Jul 21, 2021

github-actions bot commented Sep 11, 2021

metrics endpoint will not be started because metrics-address was not specified #1120

metrics endpoint will not be started because metrics-address was not specified #1120

Comments

jisnardo commented May 29, 2020

Describe the bug

Environment details

Steps to reproduce

Get metrics.

Fail.

Actual results

Expected behavior

Logs

helm chart values

Additional context

Madhu-1 commented May 29, 2020

jisnardo commented May 29, 2020

Madhu-1 commented Jul 1, 2020

HaveFun83 commented Jul 15, 2020

Madhu-1 commented Sep 7, 2020

humblec commented Sep 29, 2020

jisnardo commented Sep 30, 2020

stale bot commented Jul 21, 2021

github-actions bot commented Sep 11, 2021