Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: GCS folder support #4652

Merged
merged 3 commits into from
Oct 31, 2024
Merged

Conversation

IvanPsurtcev
Copy link
Contributor

Description

This PR fixes the problem of this issue:
https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4449
The change was only in the gcs_storage.rs file. Slightly tweaked methods so that you can write files to a specified folder in the terminal.

When trying to run a validator specifying checkpointSyncer settings for GCS:
--checkpointSyncer.type gcs --checkpointSyncer.bucket an-s3-signatures-repository --checkpointSyncer.folder atletaolympia

Building provider without signer
thread 'main' panicked at 'Failed to report agent metadata: error validating bucket name an-s3-signatures-repository/atletaolympia

Caused by:
    Character '/' @ 27 is not allowed

Location:
    hyperlane-base/src/types/gcs_storage.rs:181:9', agents/validator/src/validator.rs:178:14

To solve this problem I added the folder: Option<String> field to the GcsStorageClient structure, I also added bucket and folder processing to the impl CheckpointSyncer, GcsStorageClientBuilder methods

Copy link

changeset-bot bot commented Oct 9, 2024

⚠️ No Changeset found

Latest commit: e543e26

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@sapph1re
Copy link
Contributor

@tkporter @daniel-savu 👀

Copy link

codecov bot commented Oct 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.27%. Comparing base (f26453e) to head (e543e26).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4652   +/-   ##
=======================================
  Coverage   74.27%   74.27%           
=======================================
  Files         101      101           
  Lines        1481     1481           
  Branches      192      192           
=======================================
  Hits         1100     1100           
  Misses        360      360           
  Partials       21       21           
Components Coverage Δ
core 84.61% <ø> (ø)
hooks 77.77% <ø> (ø)
isms 77.77% <ø> (ø)
token 89.01% <ø> (ø)
middlewares 77.58% <ø> (ø)

Copy link
Contributor

@daniel-savu daniel-savu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, so the TL;DR is that GCS has no notion of folders and you just prepend it to the object key?

rust/main/hyperlane-base/src/types/gcs_storage.rs Outdated Show resolved Hide resolved
rust/main/hyperlane-base/src/types/gcs_storage.rs Outdated Show resolved Hide resolved
rust/main/hyperlane-base/src/types/gcs_storage.rs Outdated Show resolved Hide resolved
@daniel-savu daniel-savu linked an issue Oct 29, 2024 that may be closed by this pull request
@sapph1re
Copy link
Contributor

nice, so the TL;DR is that GCS has no notion of folders and you just prepend it to the object key?

GCS does have folders, but the existing code for GCS doesn't work with them correctly. That's why @IvanPsurtcev has made this addition to support folders like they are for AWS.

This is actually quite essential, because running multiple validators on one bucket requires them using different folders.

Copy link
Contributor

@daniel-savu daniel-savu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!

@daniel-savu
Copy link
Contributor

@IvanPsurtcev I think merging main will fix CI

auto-merge was automatically disabled October 31, 2024 05:21

Head branch was pushed to by a user without write access

@IvanPsurtcev
Copy link
Contributor Author

IvanPsurtcev commented Oct 31, 2024

@IvanPsurtcev I think merging main will fix CI

@daniel-savu Done

@daniel-savu daniel-savu added this pull request to the merge queue Oct 31, 2024
Merged via the queue into hyperlane-xyz:main with commit 21a0cf9 Oct 31, 2024
31 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

GCS: validator fails to understand folders
3 participants