Access denied from table_manager.go when trying to sync tables to S3

[ckacbot]

Describe the bug
Access denied from table_manager.go when trying to sync tables even though we have granted full permissions to the specified S3 bucket
We have tried to configure loki to ship logs via boltdb-shipper to S3. We run Loki on Fargate and tried to piece together a working config for this purpose. We have not found a complete example and the documention does not seem to be updated, no mention of 'storage_config->boltdb_shipper' exist on: https://grafana.com/docs/loki/latest/configuration/. (The boltdb_shipper config block is mentioned in other places, but not with an S3 example)
We have successfully used Loki together with DynamoDB so the task role has worked previously.

To Reproduce
Steps to reproduce the behavior:

1. Start Loki (2.0.0) with boltdb-shipper config, monitor logs

Expected behavior
Ship logs to s3
Environment:

• Infrastructure: ECS Fargate
• Deployment tool: Terraform

Screenshots, Promtail config, or terminal output

Log:

1604413944819,"level=info ts=2020-11-03T14:32:24.819211191Z caller=loki.go:227 msg=""Loki started"""
1604413944819,"level=info ts=2020-11-03T14:32:24.819943846Z caller=table_manager.go:324 msg=""synching tables"" expected_tables=126"
1604413944904,"level=error ts=2020-11-03T14:32:24.904558751Z caller=table_manager.go:218 msg=""error syncing tables"" err=""AccessDenied: Access Denied
\tstatus code: 403, request id: 13650B3B4BBE6387, host id: 82wYupHP/XAKe66FDSAlEwlfTju5D8mRobyDx2G0BccSkK22q5pXbyWF2Qwtn8OAgj+i5OH7br8="""
1604414004815,"level=info ts=2020-11-03T14:33:24.815548044Z caller=table_manager.go:171 msg=""uploading tables"""
1604414064815,"level=info ts=2020-11-03T14:34:24.815474955Z caller=table_manager.go:171 msg=""uploading tables"""
1604414064819,"level=info ts=2020-11-03T14:34:24.819799077Z caller=table_manager.go:324 msg=""synching tables"" expected_tables=126"
1604414064843,"level=error ts=2020-11-03T14:34:24.84365353Z caller=table_manager.go:234 msg=""error syncing tables"" err=""AccessDenied: Access Denied
\tstatus code: 403, request id: 1A75A0A8998C4C6D, host id: 1bl8ZyfIP16fVPDF63k0LMlpHwP1MquWigNuKM6d2HaF10L42X3p1iAV6T24q2R+mD6KVwq9jNk="""

Config:

auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0
schema_config:
  configs:
    - from: 2020-07-01
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
storage_config:
  aws:
    s3: s3://eu-west-1/loki-logs-core-prod
  boltdb_shipper:
    active_index_directory: /loki/active_index
    cache_location: /loki/index_cache
    shared_store: s3
compactor:
  working_directory: /loki/data/boltdb_shipper_compactor
  shared_store: filesystem
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Fargate Task Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "logs:PutLogEvents",
                "logs:DescribeLogStreams",
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetAuthorizationToken",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability"
            ],
            "Resource": "*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::loki-logs-core-prod/*",
                "arn:aws:s3:::loki-logs-core-prod/",
                "arn:aws:s3:::loki-config-core-prod/*",
                "arn:aws:s3:::loki-config-core-prod/"
            ]
        }
    ]
}

[fabianmet]

We are running into the same issue.

[fabianmet]

Was able to fix it by specifying iam user credentials in the config file. It seems somewhere the role credentials are not properly used.

[emilmark-wowgroup]

@fabianmet can you share your config? (we already tried specifying the s3 credentials but it did not work for us)

[fabianmet]

Sure! @emilmark-wowgroup. We are running inside an EKS cluster with a role that has complete access to that specific bucket.
I tried with the role assume via KIAM which goes fine for all other projects we use so i have trust in the KIAM solution. But the role failed. The only difference was adding the key and secret directly which suddenly made it start working.

The iam user i created had the exact same profile as the role.

apiVersion: v1
kind: ConfigMap
metadata:
  name: loki
  namespace: monitoring
data:
  loki.yaml: |
    auth_enabled: false
    server:
      http_listen_port: 3100
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: inmemory
          replication_factor: 1
        final_sleep: 0s
      chunk_idle_period: 5m
      chunk_retain_period: 30s
    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h

    compactor:
      working_directory: /data/loki/boltdb-shipper-compactor
      shared_store: s3

    schema_config:
      configs:
        - from: 2018-04-15
          store: boltdb-shipper
          object_store: s3
          schema: v11
          index:
            prefix: loki_index_
            period: 24h

    storage_config:

      boltdb_shipper:
        active_index_directory: /data/loki/index
        shared_store: s3
        cache_location: /data/loki/boltdb-cache

      aws:
        bucketnames: loki-with-some-random-hash
        endpoint: s3.eu-central-1.amazonaws.com
        region: eu-central-1
        access_key_id: KEYHERE
        secret_access_key: SECRETHERE
        sse_encryption: true

[emilmark-wowgroup]

Thanks, same story here, but we use IAM Task Roles. We will try it out!

[ns0092]

Encountered the same problem. Resolved it by granting S3:ListObjects & S3:PutObject access to the ec2 node running Loki. (We do not provide access keys in the values.yaml and we allow Loki to assume EC2 role)

[MatteoJoliveau]

We're having the same problem on EKS (using IAM Roles for Service Accounts and the permissions listed here).

level=error ts=2020-12-28T14:58:18.888242396Z caller=table_manager.go:218 msg="error syncing tables" err="AccessDenied: Access Denied\n\tstatus code: 403 [omitted rest for brevity]

I would really prefer not having to use static access key credentials nor instance-wide roles for security reasons. Anyone has any lead?

[MatteoJoliveau]

Found out: I was using bucketnames and region in the storage config, switching to s3: s3://region/bucket-name made it work

[frittentheke]

I just ran into the issue but am using Ceph RADOSGW as S3 backend, also with full permissions to access the bucket granted.

I followed the example at https://github.com/grafana/loki/blob/f6fd6ae7da0d929788250a1273efc531ae25ec65/docs/sources/configuration/examples.md#s3-compatible-apis, but only after changing the S3-URL as suggested in
#1051 (comment) it works.

So instead of s3://endpoint/bucket it should rather be https://endpoint:443/bucket.

[emilmark-wowgroup]

@slim-bean we still have this problem and would really appreciate some clarification regarding the configuration or more complete example for S3 + boltdb-shipper.

We have tried all the above changes that have been suggested + tried all the different configurations that we have found in your documentation.

I "verified" that the config works in a docker-compose setup with MinIO (modified version of Grafana TNS demo):
https://github.com/emilmark/grafana-tns-plus-minio

Whats more confusing is that our s3 bucket receives files in both fake and index dirs even though we get the following errors:

level=error ts=2021-02-19T06:47:15.341157457Z caller=table_manager.go:234 msg="error syncing tables" err="AccessDenied: Access Denied\n\tstatus code: 403 level=error ts=2021-02-19T06:45:15.450667827Z caller=compactor.go:89 msg="failed to run compaction" err="AccessDenied: Access Denied\n\tstatus code: 403 level=error ts=2021-02-19T06:45:15.44721119Z caller=table_manager.go:218 msg="error syncing tables" err="AccessDenied: Access Denied\n\tstatus code: 403,

[owen-d]

Hey, apologies for closing this, but the issue has gotten off topic. This looks like a provider related authorization process that's outside of Loki itself. Here's a link to which permissions you should need: https://grafana.com/docs/loki/latest/operations/storage/#cloud-storage-permissions

[MuhammadNaeemAkhtar]

Encountered the same problem. Resolved it by granting S3:ListObjects & S3:PutObject access to the ec2 node running Loki. (We do not provide access keys in the values.yaml and we allow Loki to assume EC2 role)

Hi there,
I'm trying in the same way but having issues. Can you specify the steps how to create s3 with without providing secrets in values.yaml.
Thanks!

[ognjenVlad]

Still having the same issue, using s3://region/bucket_name.

[MuhammadNaeemAkhtar]

Still having the same issue, using s3://region/bucket_name.

No, created a policy with the following actions and added the specific bucket as resource in the policy.

"Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ]

Then attached that policy to the cluster roles and it is working fine.
Thank You!

[ognjenVlad]

@MuhammadNaeemAkhtar Can you please share your config file? I still cant make it work. I have created service-account with the right permissions and every other pod in my cluster can access S3 bucket except for Loki pod. I am creating service account through Helm chart.

[MuhammadNaeemAkhtar]

@ognjenVlad Are you using EKS cluster?
If yes then it creates two roles related to your cluster. You need to create a policy with the following configuration and attach it to your cluster roles

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Then in the configuration of your loki, I'm using like this.

compactor:
  shared_store: s3
schema_config:
  configs:
    - from: 2020-09-07
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
    
storage_config:
  boltdb_shipper:
    shared_store: s3
    active_index_directory: /var/loki/index
    cache_location: /var/loki/cache
    cache_ttl: 168h
  aws:
    s3: s3://your-region/your-bucket-name
    sse_encryption: true

Hope it'll help you.