Correct an issue for README for tools and also correct s3 solution in…

… Args.scala (#2618) * Signed-off-by: Hao Zhu <hazhu@nvidia.com> Correct an issue for README for tools and also correct s3 solution in Args.scala * Add doc link for Hadoop-AWS module * Remove S3 instructions. * Remove S3 instructions from QualificationArgs
NVIDIA · Jun 7, 2021 · a7a3d63 · a7a3d63
1 parent 99dec80
commit a7a3d63
Show file tree

Hide file tree

Showing 3 changed files with 5 additions and 5 deletions.
diff --git a/tools/README.md b/tools/README.md
@@ -34,7 +34,7 @@ mvn -pl .,tools clean verify -DskipTests
 ```
 The jar is generated in below directory :
 
-`./rapids-4-spark-tools/target/rapids-4-spark-tools_2.12-<version>.jar`
+`./tools/target/rapids-4-spark-tools_2.12-<version>.jar`
 
 ## Accessing files from S3
 
@@ -61,6 +61,8 @@ Take Hadoop 2.7.4 for example, we can download and include below jars in the '--
 </property>
 </configuration>
 ```
+Please refer to this [doc](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html) on 
+more options about integrating hadoop-aws module with S3.
 
 ## Qualification Tool
 

diff --git a/tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileArgs.scala b/tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileArgs.scala
@@ -33,8 +33,7 @@ rapids-4-spark-tools_2.12-<version>.jar /path/to/eventlog1 /path/to/eventlog2
  rapids-4-spark-tools_2.12-<version>.jar /path/to/DirOfManyEventLogs
 
 # If any event log is from S3:
-export AWS_ACCESS_KEY_ID=xxx
-export AWS_SECRET_ACCESS_KEY=xxx
+# Need to download hadoop-aws-<version>.jar and aws-java-sdk-<version>.jar firstly.
 ./bin/spark-submit --class com.nvidia.spark.rapids.tool.profiling.ProfileMain
 rapids-4-spark-tools_2.12-<version>.jar s3a://<BUCKET>/eventlog1 /path/to/eventlog2
 

diff --git a/tools/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala b/tools/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala
@@ -33,8 +33,7 @@ rapids-4-spark-tools_2.12-<version>.jar /path/to/eventlog1 /path/to/eventlog2
 rapids-4-spark-tools_2.12-<version>.jar /path/to/DirOfManyEventLogs
 
 # If any event log is from S3:
-export AWS_ACCESS_KEY_ID=xxx
-export AWS_SECRET_ACCESS_KEY=xxx
+# Need to download hadoop-aws-<version>.jar and aws-java-sdk-<version>.jar firstly.
 ./bin/spark-submit --class com.nvidia.spark.rapids.tool.qualification.QualificationMain
 rapids-4-spark-tools_2.12-<version>.jar s3a://<BUCKET>/eventlog1 /path/to/eventlog2