Skip to content

Commit

Permalink
GH-36770: [C++] Use custom endpoint for s3 using environment variable…
Browse files Browse the repository at this point in the history
… AWS_ENDPOINT_URL (#36791)

### Rationale for this change
we need a way to read custom object storage (such as minio host or other s3-like storage).
use environment variable `AWS_ENDPOINT_URL `

### What changes are included in this PR?
set variable endpoint_override according the environment variable

### Are these changes tested?
unittest and tested on pyarrow

### Are there any user-facing changes?
No

* Closes: #36770

Authored-by: yiwei.wang <yiwei.wang@weride.ai>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
  • Loading branch information
adbmal authored Jul 21, 2023
1 parent d98b3a1 commit f43bfd6
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 0 deletions.
5 changes: 5 additions & 0 deletions cpp/src/arrow/filesystem/s3fs.cc
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ using internal::ToAwsString;
using internal::ToURLEncodedAwsString;

static const char kSep = '/';
constexpr char kAwsEndpointUrlEnvVar[] = "AWS_ENDPOINT_URL";

// -----------------------------------------------------------------------
// S3ProxyOptions implementation
Expand Down Expand Up @@ -337,6 +338,10 @@ Result<S3Options> S3Options::FromUri(const Uri& uri, std::string* out_path) {
} else {
options.ConfigureDefaultCredentials();
}
auto endpoint_env = arrow::internal::GetEnvVar(kAwsEndpointUrlEnvVar);
if (endpoint_env.ok()) {
options.endpoint_override = *endpoint_env;
}

bool region_set = false;
for (const auto& kv : options_map) {
Expand Down
7 changes: 7 additions & 0 deletions cpp/src/arrow/filesystem/s3fs_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,13 @@ TEST_F(S3OptionsTest, FromUri) {

// Invalid option
ASSERT_RAISES(Invalid, S3Options::FromUri("s3://mybucket/?xxx=zzz", &path));

// Endpoint from environment variable
{
EnvVarGuard endpoint_guard("AWS_ENDPOINT_URL", "http://127.0.0.1:9000");
ASSERT_OK_AND_ASSIGN(options, S3Options::FromUri("s3://mybucket/", &path));
ASSERT_EQ(options.endpoint_override, "http://127.0.0.1:9000");
}
}

TEST_F(S3OptionsTest, FromAccessKey) {
Expand Down

0 comments on commit f43bfd6

Please sign in to comment.