GCS external image upload error response status code 404

[puzzloholic]

What happened:
I am using GCS for external image storage solution. I already mount required key_file json. No file is uploaded when alert is triggered, and log states that GCS response is 404

t=2020-07-09T07:30:45+0000 lvl=dbug msg="Rendered alert panel image" logger=alerting.notifier ruleId=6 path=/var/lib/grafana/png/PaICSdwfoqdYUnOa6DkM.png took=4.289621693s
t=2020-07-09T07:30:45+0000 lvl=dbug msg="Uploading alert panel image to external image store" logger=alerting.notifier ruleId=6 path=/var/lib/grafana/png/PaICSdwfoqdYUnOa6DkM.png
t=2020-07-09T07:30:45+0000 lvl=dbug msg="Opening key file " logger=gcsuploader /etc/secrets/<REDACTED>.json=nil LOG15_ERROR="Normalized odd number of arguments by adding nil"
t=2020-07-09T07:30:45+0000 lvl=dbug msg="Creating JWT conf" logger=gcsuploader
t=2020-07-09T07:30:45+0000 lvl=dbug msg="Creating HTTP client" logger=gcsuploader
t=2020-07-09T07:30:45+0000 lvl=dbug msg="Opening image file " logger=gcsuploader /var/lib/grafana/png/PaICSdwfoqdYUnOa6DkM.png=nil LOG15_ERROR="Normalized odd number of arguments by adding nil"
t=2020-07-09T07:30:45+0000 lvl=dbug msg="Request URL: " logger=gcsuploader https://www.googleapis.com/upload/storage/v1/b/<REDACTED>/o?uploadType=media&name=uJ0fW2e0UqOZ5Q51hp3C.png&predefinedAcl=publicRead=nil LOG15_ERROR="Normalized odd number of arguments by adding nil"
t=2020-07-09T07:30:45+0000 lvl=dbug msg="Sending POST request to GCS" logger=gcsuploader
t=2020-07-09T07:30:45+0000 lvl=eror msg="Failed to render and upload alert panel image." logger=alerting.notifier ruleId=6 error="GCS response status code 404"

What you expected to happen:
Image is uploaded to GCS every time alert triggers.

How to reproduce it (as minimally and precisely as possible):

• Create k8s Secret for the GCP SA key

apiVersion: v1
kind: Secret
metadata:
  name: secret-name
stringData:
  your-key-file.json: |-
    {
        "type": "service_account",
        ...
    }

• Deploy Grafana Helm Chart with this values

image:
  repository: grafana/grafana
  tag: 7.0.3
extraSecretMounts:
- name: secret-files
  mountPath: /etc/secrets
  secretName: secret-name
  readOnly: true
  subPath: ""
grafana.ini:
  external_image_storage:
    provider: gcs
  external_image_storage.gcs:
    key_file: /etc/secrets/your-key-file.json
    bucket: your-gcs-bucket

• Try to config and trigger an alarm that will upload image file to GCS

Anything else we need to know?:
Official GCP Documentation states this is the API for uploading file to GCS.

curl -X POST --data-binary @[OBJECT_LOCATION] \
-H "Authorization: Bearer [OAUTH2_TOKEN]" \
-H "Content-Type: [OBJECT_CONTENT_TYPE]" \
"https://storage.googleapis.com/upload/storage/v1/b/[BUCKET_NAME]/o?uploadType=media&name=[OBJECT_NAME]"

Apparently there is changes in the endpoint at some time

This is the line that defines the URL

grafana/pkg/components/imguploader/gcsuploader.go

Line 18 in 45cfb34

uploadUrl string = "https://www.googleapis.com/upload/storage/v1/b/%s/o?uploadType=media&name=%s&predefinedAcl=publicRead"

Environment:

• Grafana version: Grafana v7.0.3 (00ee734)
• Others: Helm Stable Grafana Deployed on GKE Cluster

[gastonqiu]

I propose to do fileUploader with gcp go sdk instead of http request. https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-code-sample

[gastonqiu]

@puzzloholic I patch a new version by using GKE SDK but I'm not so sure about how to use this. Is it possible to help me on testing it?