Correct an issue for README for tools and also correct s3 solution in Args.scala #2618
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Correct an issue for README for tools and also correct s3 solution in Args.scala
gerashegalov
requested changes
Jun 7, 2021
| export AWS_ACCESS_KEY_ID=xxx | ||
| export AWS_SECRET_ACCESS_KEY=xxx | ||
| # Need to download hadoop-aws-<version>.jar and aws-java-sdk-<version>.jar firstly, | ||
| # and then put fs.s3a.access.key and fs.s3a.secret.key in $SPARK_HOME/conf/hdfs-site.xml. |
There was a problem hiding this comment.
S3a is a separate filesystem from HDFS. It's typically stored in core-site.xml. But you can also just configure in spark via spark.hadoop.fs.s3a... settings. Environment variables should work too if EnvironmentVariableCredentialsProvider is configured.
I suggest that we don't go into details in our repo, and simply defer/link Hadoop/Spark docs for this.
There was a problem hiding this comment.
@gerashegalov I just added a doc link in README and removed the instructions in 2 Args.scala files.
How is now?thx
gerashegalov
approved these changes
Jun 7, 2021
tgravescs
approved these changes
Jun 7, 2021
|
build |
nartal1
approved these changes
Jun 7, 2021
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
… Args.scala (NVIDIA#2618) * Signed-off-by: Hao Zhu <hazhu@nvidia.com> Correct an issue for README for tools and also correct s3 solution in Args.scala * Add doc link for Hadoop-AWS module * Remove S3 instructions. * Remove S3 instructions from QualificationArgs
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
… Args.scala (NVIDIA#2618) * Signed-off-by: Hao Zhu <hazhu@nvidia.com> Correct an issue for README for tools and also correct s3 solution in Args.scala * Add doc link for Hadoop-AWS module * Remove S3 instructions. * Remove S3 instructions from QualificationArgs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Correct an issue for README for tools and also correct s3 solution in Args.scala