Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3: Support specifying S3 storage class #25

Merged
merged 1 commit into from
Oct 6, 2022

Conversation

juanrh
Copy link
Contributor

@juanrh juanrh commented Sep 1, 2022

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Changes

For thanos-io/thanos#5663, enable specifying S3 storage class by specifying it in config.put_user_metadata. For example launching the sidecar with a --objstore.config-file as follows stores all S3 files with storage class of STANDARD_IA

type: S3
prefix: thanos-test-standard-ia
config:
  endpoint: s3.us-east-1.amazonaws.com
  region: us-east-1
  bucket: MY_BUCKET
  put_user_metadata:
    X-Amz-Storage-Class: STANDARD_IA
  trace:
    enable: true

while using the following config file uses the default storage class of STANDARD

type: S3
prefix: thanos-test-standard-ia
config:
  endpoint: s3.us-east-1.amazonaws.com
  region: us-east-1
  bucket: MY_BUCKET
  trace:
    enable: true

Verification

Added unit tests.

Manual tests launching the sidecar with the two configurations listed above, doing rm ${PROM_ROOT}/data/thanos.shipper.json before that to force uploading files, and with a prometheus instance with small block duration to force upload.

${PROM_ROOT}/prometheus --config.file=${PROM_ROOT}/prometheus.yml \
  --storage.tsdb.min-block-duration=1m \
  --storage.tsdb.max-block-duration=1m

@juanrh juanrh marked this pull request as ready for review September 7, 2022 16:47
@juanrh juanrh changed the title Enable specifying S3 storage class Support specifying S3 storage class Sep 8, 2022
@juanrh juanrh changed the title Support specifying S3 storage class S3: Support specifying S3 storage class Sep 8, 2022
@juanrh juanrh force-pushed the thanos-5663 branch 2 times, most recently from 8a06c92 to 55c8089 Compare September 8, 2022 09:36
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Proposed some changes, otherwise LGTM (:

CHANGELOG.md Outdated
@@ -14,6 +14,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re

### Added
- [#15](https://github.com/thanos-io/objstore/pull/15) Add Oracle Cloud Infrastructure Object Storage Bucket support.
- [#25](https://github.com/thanos-io/objstore/pull/25) Thanos Objstore S3: Support specifying S3 storage class.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [#25](https://github.com/thanos-io/objstore/pull/25) Thanos Objstore S3: Support specifying S3 storage class.
- [#25](https://github.com/thanos-io/objstore/pull/25) S3: Support specifying S3 storage class.

providers/s3/s3.go Outdated Show resolved Hide resolved
@juanrh juanrh force-pushed the thanos-5663 branch 2 times, most recently from 3b1f06f to a5ee588 Compare September 9, 2022 16:41
Allow specifying a storage class for
S3 object storage in
config.put_user_metadata:X-Amz-Storage-Class.
If not specified then STANDARD is used by default.

Solves: thanos-io/thanos#5663

Signed-off-by: Juan Rodriguez Hortala <juanrh@redhat.com>
@matej-g matej-g merged commit 79dcec7 into thanos-io:main Oct 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants