From 384b4caf0a1181cb0564637ae35c66b131420d5e Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Wed, 21 Jul 2021 09:24:46 -0700 Subject: [PATCH 01/21] Signed-off-by: Hao Zhu Move tools README under docs folder. --- .../qualification-profiling-tools.md | 201 ++++++++++-------- 1 file changed, 109 insertions(+), 92 deletions(-) rename tools/README.md => docs/additional-functionality/qualification-profiling-tools.md (87%) diff --git a/tools/README.md b/docs/additional-functionality/qualification-profiling-tools.md similarity index 87% rename from tools/README.md rename to docs/additional-functionality/qualification-profiling-tools.md index 8613bbb4781..2292bc321a7 100644 --- a/tools/README.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -1,3 +1,9 @@ +--- +layout: page +title: Qualification and Profiling tools +parent: Additional Functionality +nav_order: 7 +--- # Spark Qualification and Profiling tools The qualification tool is used to look at a set of applications to determine if the RAPIDS Accelerator for Apache Spark @@ -9,6 +15,23 @@ GPU generated event logs. (The code is based on Apache Spark 3.1.1 source code, and tested using Spark 3.0.x and 3.1.1 event logs) +This document covers below topics: + +* [Prerequisites](#Prerequisites) +* [Download the tools jar or compile it](#Download the tools jar or compile it) +* [Qualification Tool](#Qualification Tool) + * [Qualification tool functions](#Qualification tool functions) + * [Download the Spark 3.x distribution](#Download the Spark 3.x distribution) + * [How to use qualification tool](#How to use qualification tool) + * [Qualification tool options](#Qualification tool options) + * [Qualification tool output](#Qualification tool output) +* [Profiling Tool](#Profiling Tool) + * [Profiling tool functions](#Profiling tool functions) + * [Profiling tool metrics definitions](#Profiling tool metrics definitions) + * [How to use profiling tool](#How to use profiling tool) + * [Profiling tool options](#Profiling tool options) + * [Profiling tool output](#Profiling tool output) + ## Prerequisites - Spark 3.0.1 or newer, the Qualification tool just needs the Spark jars and the Profiling tool runs a Spark application so needs the Spark runtime. @@ -72,31 +95,9 @@ Take Hadoop 2.7.4 for example, we can download and include below jars in the '-- Please refer to this [doc](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html) on more options about integrating hadoop-aws module with S3. -## Filter input event logs -Both of the qualification tool and profiling tool have this function which is to filter event logs. -Here are 2 filter options: -1. N newest or oldest event logs. (-f option) - -This is based on timestamp of the event log files. - -2. Event log file names match the string. (-m option) - -This is based on the event log file name. - -Below is an example of profiling tool: - -Filter event logs to be processed. 10 newest file with filenames containing "local": -```bash -$SPARK_HOME/bin/spark-submit --class com.nvidia.spark.rapids.tool.profiling.ProfileMain \ -rapids-4-spark-tools_2.12-.jar \ --m "local" -f "10-newest-filesystem" \ -/directory/with/eventlogs/ -``` - - ## Qualification Tool -### Functions +### Qualification tool functions The qualification tool is used to look at a set of applications to determine if the RAPIDS Accelerator for Apache Spark might be a good fit for those applications. The tool works by processing the CPU generated event logs from Spark. @@ -121,7 +122,7 @@ The other file is a CSV file that contains more information and can be used for Note, potential problems are reported in the CSV file in a separate column, which is not included in the score. This currently includes some UDFs and some decimal operations. The tool won't catch all UDFs, and some of the UDFs can be -handled with additional steps. Please refer to [supported_ops.md](../docs/supported_ops.md) for more details on UDF. +handled with additional steps. Please refer to [supported_ops.md](../supported_ops.md) for more details on UDF. For decimals, it tries to recognize decimal operations but it may not catch them all. The CSV output also contains a `Executor CPU Time Percent` column that is not included in the score. This is an estimate @@ -151,7 +152,7 @@ Sample output in text: |app-20210507174503-1704| 83738| 6760| 0| ``` -## Download the Spark 3.x distribution +### Download the Spark 3.x distribution The Qualification tool requires the Spark 3.x jars to be able to run. If you do not already have Spark 3.x installed, you can download the Spark distribution to any machine and include the jars in the classpath. @@ -160,9 +161,9 @@ in the classpath. 2. Extract the Spark distribution into a local directory. 3. Either set `SPARK_HOME` to point to that directory or just put the path inside of the classpath `java -cp toolsJar:pathToSparkJars/*:...` when you run the qualification tool. See the - [How to use this tool](#how-to-use-this-tool) section below. + [How to use qualification tool](#How to use qualification tool) section below. -### How to use this tool +### How to use qualification tool This tool parses the Spark CPU event log(s) and creates an output report. Acceptable input event log paths are files or directories containing spark events logs in the local filesystem, HDFS, S3 or mixed. @@ -179,65 +180,74 @@ java -cp ~/rapids-4-spark-tools_2.12-21..jar:$SPARK_HOME/jars/*:$HADOOP com.nvidia.spark.rapids.tool.qualification.QualificationMain /eventlogDir ``` -### Options (`--help` output) +### Qualification tool options Note: `--help` should be before the trailing event logs. ```bash +java -cp ~/rapids-4-spark-tools_2.12-21..jar:$SPARK_HOME/jars/*:$HADOOP_CONF_DIR/ \ + com.nvidia.spark.rapids.tool.qualification.QualificationMain --help + RAPIDS Accelerator for Apache Spark qualification tool Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* com.nvidia.spark.rapids.tool.qualification.QualificationMain [options] - -a, --application-name Filter event logs whose application name - matches exactly or is a substring of input - string. Regular expressions not supported. - --application-name ~ Filter event logs based on the complement - of a selection criterion. i.e Select all - event logs except the ones which have - application name as the input string. - -f, --filter-criteria Filter newest or oldest N eventlogs based on application start - timestamp, unique application name or filesystem - timestamp. Filesystem based filtering happens before any - application based filtering. - For application based filtering, the order in which filters are - applied is: application-name, start-app-time, filter-criteria. - Application based filter-criteria are: - 100-newest (for processing newest 100 event logs based on - timestamp of the application inside the eventlog i.e application - start time) - 100-oldest (for processing oldest 100 event logs based on - timestamp of the application inside the eventlog i.e application - start time) - 100-newest-per-app-name (select at most 100 newest log files for - each unique application name) - 100-oldest-per-app-name (select at most 100 oldest log files for - each unique application name) - Filesystem based filter criteria are: - 100-newest-filesystem (for processing newest 100 event - logs based on filesystem timestamp). - 100-oldest-filesystem (for processing oldest 100 event logs - based on filesystem timestamp). - -m, --match-event-logs Filter event logs whose filenames contain the - input string. Filesystem based filtering happens before - any application based filtering. + -a, --application-name Filter event logs whose application name + matches exactly or is a substring of input + string. Regular expressions not + supported.For filtering based on complement + of application name, use ~APPLICATION_NAME. + i.e Select all event logs except the ones + which have application name as the input + string. + -f, --filter-criteria Filter newest or oldest N eventlogs based on + application start timestamp, unique + application name or filesystem timestamp. + Filesystem based filtering happens before + any application based filtering.For + application based filtering, the order in + which filters areapplied is: + application-name, start-app-time, + filter-criteria.Application based + filter-criteria are:100-newest (for + processing newest 100 event logs based on + timestamp insidethe eventlog) i.e + application start time) 100-oldest (for + processing oldest 100 event logs based on + timestamp insidethe eventlog) i.e + application start time) + 100-newest-per-app-name (select at most 100 + newest log files for each unique application + name) 100-oldest-per-app-name (select at + most 100 oldest log files for each unique + application name)Filesystem based filter + criteria are:100-newest-filesystem (for + processing newest 100 event logs based on + filesystem timestamp). 100-oldest-filesystem + (for processing oldest 100 event logsbased + on filesystem timestamp). + -m, --match-event-logs Filter event logs whose filenames contain + the input string. Filesystem based filtering + happens before any application based + filtering. -n, --num-output-rows Number of output rows in the summary report. Default is 1000. --num-threads Number of thread to use for parallel - processing. The default is the number of cores - on host divided by 4. - --order Specify the sort order of the report. desc or - asc, desc is the default. desc (descending) - would report applications most likely to be - accelerated at the top and asc (ascending) - would show the least likely to be accelerated - at the top. + processing. The default is the number of + cores on host divided by 4. + --order Specify the sort order of the report. desc + or asc, desc is the default. desc + (descending) would report applications most + likely to be accelerated at the top and asc + (ascending) would show the least likely to + be accelerated at the top. -o, --output-directory Base output directory. Default is current directory for the default filesystem. The final output will go into a subdirectory - called rapids_4_spark_qualification_output. It - will overwrite any existing directory with the - same name. + called rapids_4_spark_qualification_output. + It will overwrite any existing directory + with the same name. -r, --read-score-percent The percent the read format and datatypes apply to the score. Default is 20 percent. --report-read-schema Whether to output the read formats and @@ -249,11 +259,12 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* min(minute),h(hours),d(days),w(weeks),m(months). If a period is not specified it defaults to days. - -t, --timeout Maximum time in seconds to wait for the event - logs to be processed. Default is 24 hours - (86400 seconds) and must be greater than 3 - seconds. If it times out, it will report what - it was able to process up until the timeout. + -t, --timeout Maximum time in seconds to wait for the + event logs to be processed. Default is 24 + hours (86400 seconds) and must be greater + than 3 seconds. If it times out, it will + report what it was able to process up until + the timeout. -h, --help Show help message trailing arguments: @@ -262,7 +273,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* /path/to/eventlog2 ``` -### Output +### Qualification tool output The summary report goes to STDOUT and by default it outputs 2 files under sub-directory `./rapids_4_spark_qualification_output/` that contain the processed applications. The output will go into your default filesystem, it supports local filesystem or HDFS. @@ -279,7 +290,8 @@ It will run a Spark application so requires Spark to be installed and setup. If you can run it on that, or you can simply run it in local mode as well. See the Apache Spark documentation for [Downloading Apache Spark 3.x](http://spark.apache.org/downloads.html) -### Functions +### Profiling tool functions + #### A. Collect Information or Compare Information(if more than 1 event logs are as input and option -c is specified) - Application information - Executors information @@ -573,15 +585,15 @@ Failed jobs: +--------+-----+------+--------+---------------------------------------------------------------------------------------------------+ ``` -### Metrics Definitions +### Profiling tool metrics definitions All the metrics definitions can be found in the [executor task metrics doc](https://spark.apache.org/docs/latest/monitoring.html#executor-task-metrics) / [executor metrics doc](https://spark.apache.org/docs/latest/monitoring.html#executor-metrics) or the [SPARK webUI doc](https://spark.apache.org/docs/latest/web-ui.html#content). -### How to use this tool +### How to use profiling tool This tool parses the Spark CPU or GPU event log(s) and creates an output report. Acceptable input event log paths are files or directories containing spark events logs in the local filesystem, HDFS, S3 or mixed. -### Use from spark-shell +#### Use from spark-shell 1. Include `rapids-4-spark-tools_2.12-.jar` in the '--jars' option to spark-shell or spark-submit 2. After starting spark-shell: @@ -595,14 +607,14 @@ For multiple event logs comparison and analysis: com.nvidia.spark.rapids.tool.profiling.ProfileMain.main(Array("/path/to/eventlog1", "/path/to/eventlog2")) ``` -### Use from spark-submit +#### Use from spark-submit ```bash $SPARK_HOME/bin/spark-submit --class com.nvidia.spark.rapids.tool.profiling.ProfileMain \ rapids-4-spark-tools_2.12-.jar \ /path/to/eventlog1 /path/to/eventlog2 /directory/with/eventlogs ``` -### Options (`--help` output) +### Profiling tool options Note: `--help` should be before the trailing event logs. ```bash @@ -613,23 +625,28 @@ rapids-4-spark-tools_2.12-.jar \ For usage see below: + -c, --compare Compare Applications (Recommended to compare less than 10 applications). Default is false - -f, --filter-criteria Filter newest or oldest N event logs for processing. - Supported formats are: - To process 10 recent event logs: --filter-criteria "10-newest-timestamp" - To process 10 oldest event logs: --filter-criteria "10-oldest-timestamp" + -f, --filter-criteria Filter newest or oldest N eventlogs for + processing.eg: 100-newest-filesystem (for + processing newest 100 event logs). eg: + 100-oldest-filesystem (for processing oldest + 100 event logs) -g, --generate-dot Generate query visualizations in DOT format. Default is false - -m, --match-event-logs Filter event logs filenames which contains the input string. + --generate-timeline Write an SVG graph out for the full + application timeline. + -m, --match-event-logs Filter event logs whose filenames contain the + input string -n, --num-output-rows Number of output rows for each Application. Default is 1000 -o, --output-directory Base output directory. Default is current directory for the default filesystem. The final output will go into a subdirectory - called rapids_4_spark_profile. - It will overwrite any existing files with - the same name. + called rapids_4_spark_profile. It will + overwrite any existing files with the same + name. -p, --print-plans Print the SQL plans to a file starting with 'planDescriptions-'. Default is false -h, --help Show help message @@ -640,7 +657,7 @@ For usage see below: /path/to/eventlog2 ``` -### Output +### Profiling tool output By default this outputs a log file under sub-directory `./rapids_4_spark_profile` named `rapids_4_spark_tools_output.log` that contains the processed applications. The output will go into your default filesystem, it supports local filesystem or HDFS. There are separate files that are generated From a7b8d8f5dcf5b8e94b0c083e023bd23971fb4511 Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Wed, 21 Jul 2021 09:28:23 -0700 Subject: [PATCH 02/21] Signed-off-by: Hao Zhu Correct some typos of the link --- .../qualification-profiling-tools.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 2292bc321a7..d3bd2c6b8af 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -18,19 +18,19 @@ GPU generated event logs. This document covers below topics: * [Prerequisites](#Prerequisites) -* [Download the tools jar or compile it](#Download the tools jar or compile it) -* [Qualification Tool](#Qualification Tool) - * [Qualification tool functions](#Qualification tool functions) - * [Download the Spark 3.x distribution](#Download the Spark 3.x distribution) - * [How to use qualification tool](#How to use qualification tool) - * [Qualification tool options](#Qualification tool options) - * [Qualification tool output](#Qualification tool output) -* [Profiling Tool](#Profiling Tool) - * [Profiling tool functions](#Profiling tool functions) - * [Profiling tool metrics definitions](#Profiling tool metrics definitions) - * [How to use profiling tool](#How to use profiling tool) - * [Profiling tool options](#Profiling tool options) - * [Profiling tool output](#Profiling tool output) +* [Download the tools jar or compile it](#Download-the-tools-jar-or-compile-it) +* [Qualification Tool](#Qualification-Tool) + * [Qualification tool functions](#Qualification-tool-functions) + * [Download the Spark 3.x distribution](#Download-the-Spark-3.x-distribution) + * [How to use qualification tool](#How-to-use-qualification-tool) + * [Qualification tool options](#Qualification-tool-options) + * [Qualification tool output](#Qualification-tool-output) +* [Profiling Tool](#Profiling-Tool) + * [Profiling tool functions](#Profiling-tool-functions) + * [Profiling tool metrics definitions](#Profiling-tool-metrics-definitions) + * [How to use profiling tool](#How-to-use-profiling-tool) + * [Profiling tool options](#Profiling-tool-options) + * [Profiling tool output](#Profiling-tool-output) ## Prerequisites - Spark 3.0.1 or newer, the Qualification tool just needs the Spark jars and the Profiling tool @@ -161,7 +161,7 @@ in the classpath. 2. Extract the Spark distribution into a local directory. 3. Either set `SPARK_HOME` to point to that directory or just put the path inside of the classpath `java -cp toolsJar:pathToSparkJars/*:...` when you run the qualification tool. See the - [How to use qualification tool](#How to use qualification tool) section below. + [How to use qualification tool](#How-to-use-qualification-tool) section below. ### How to use qualification tool This tool parses the Spark CPU event log(s) and creates an output report. From 80228a05e93d89d525a2e06535fdbd6f0e13f3d3 Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Wed, 21 Jul 2021 09:42:57 -0700 Subject: [PATCH 03/21] Signed-off-by: Hao Zhu Fix some typo --- .../additional-functionality/qualification-profiling-tools.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index d3bd2c6b8af..67773159dbe 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -21,7 +21,7 @@ This document covers below topics: * [Download the tools jar or compile it](#Download-the-tools-jar-or-compile-it) * [Qualification Tool](#Qualification-Tool) * [Qualification tool functions](#Qualification-tool-functions) - * [Download the Spark 3.x distribution](#Download-the-Spark-3.x-distribution) + * [Download the Spark 3 distribution](#Download-the-Spark-3-distribution) * [How to use qualification tool](#How-to-use-qualification-tool) * [Qualification tool options](#Qualification-tool-options) * [Qualification tool output](#Qualification-tool-output) @@ -152,7 +152,7 @@ Sample output in text: |app-20210507174503-1704| 83738| 6760| 0| ``` -### Download the Spark 3.x distribution +### Download the Spark 3 distribution The Qualification tool requires the Spark 3.x jars to be able to run. If you do not already have Spark 3.x installed, you can download the Spark distribution to any machine and include the jars in the classpath. From 346dff1b1ca2693b71b7b3ad7a3cbce1e41f676b Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Wed, 21 Jul 2021 10:27:35 -0700 Subject: [PATCH 04/21] Signed-off-by: Hao Zhu Add more example options for tools doc --- .../qualification-profiling-tools.md | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 67773159dbe..5628936a140 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -273,6 +273,23 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* /path/to/eventlog2 ``` +Example commands: +- Process the 10 newest logs, and only output the top 3 in the output: +```bash +java -cp ~/rapids-4-spark-tools_2.12-21..jar:$SPARK_HOME/jars/*:$HADOOP_CONF_DIR/ \ + com.nvidia.spark.rapids.tool.qualification.QualificationMain -f 10-newest -n 3 /eventlogDir +``` +- Process last 100 days' logs: +```bash +java -cp ~/rapids-4-spark-tools_2.12-21..jar:$SPARK_HOME/jars/*:$HADOOP_CONF_DIR/ \ + com.nvidia.spark.rapids.tool.qualification.QualificationMain -s 100d /eventlogDir +``` +- Process only the newest log with the same application name: +```bash +java -cp ~/rapids-4-spark-tools_2.12-21..jar:$SPARK_HOME/jars/*:$HADOOP_CONF_DIR/ \ + com.nvidia.spark.rapids.tool.qualification.QualificationMain -f 1-newest-per-app-name /eventlogDir +``` + ### Qualification tool output The summary report goes to STDOUT and by default it outputs 2 files under sub-directory `./rapids_4_spark_qualification_output/` that contain the processed applications. The output will @@ -657,6 +674,22 @@ For usage see below: /path/to/eventlog2 ``` +Example commands: +- Process 10 newest logs with filenames containing "local": +```bash +$SPARK_HOME/bin/spark-submit --class com.nvidia.spark.rapids.tool.profiling.ProfileMain \ +rapids-4-spark-tools_2.12-.jar \ +-m "local" -f "10-newest-filesystem" \ +/directory/with/eventlogs/ +``` +- Print SQL plans, generate dot files and also generate timeline(SVG graph): +```bash +$SPARK_HOME/bin/spark-submit --class com.nvidia.spark.rapids.tool.profiling.ProfileMain \ +rapids-4-spark-tools_2.12-.jar \ +-p -g --generate-timeline \ +/directory/with/eventlogs/ +``` + ### Profiling tool output By default this outputs a log file under sub-directory `./rapids_4_spark_profile` named `rapids_4_spark_tools_output.log` that contains the processed applications. The output will go into your From 52b11817447ab5b90fa197a7dfbea29d404d7438 Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Wed, 21 Jul 2021 11:37:54 -0700 Subject: [PATCH 05/21] Signed-off-by: Hao Zhu Add a new README for tools. --- tools/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 tools/README.md diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 00000000000..1100383eba8 --- /dev/null +++ b/tools/README.md @@ -0,0 +1,11 @@ +# Spark Qualification and Profiling tools + +The qualification tool is used to look at a set of applications to determine if the RAPIDS Accelerator for Apache Spark +might be a good fit for those applications. + +The profiling tool generates information which can be used for debugging and profiling applications. +Information such as Spark version, executor information, properties and so on. This runs on either CPU or +GPU generated event logs. + +Please refer to [this documentation](../docs/additional-functionality/qualification-profiling-tools.md) +for more details on how to use the 2 tools. \ No newline at end of file From 5305c24c9f858f0b603b5843803632fd981b1978 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:34:25 -0700 Subject: [PATCH 06/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 5628936a140..add73b9478a 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -6,7 +6,7 @@ nav_order: 7 --- # Spark Qualification and Profiling tools -The qualification tool is used to look at a set of applications to determine if the RAPIDS Accelerator for Apache Spark +The qualification tool analyzes applications to determine if the RAPIDS Accelerator for Apache Spark might be a good fit for those applications. The profiling tool generates information which can be used for debugging and profiling applications. From 770586275052dbf6b80e0f45ae513e5eb1134593 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:34:38 -0700 Subject: [PATCH 07/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index add73b9478a..396a3cb6bdd 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -10,7 +10,7 @@ The qualification tool analyzes applications to determine if the RAPIDS Accelera might be a good fit for those applications. The profiling tool generates information which can be used for debugging and profiling applications. -Information such as Spark version, executor information, properties and so on. This runs on either CPU or +The information contains the Spark version, executor details, properties, etc. This runs on either CPU or GPU generated event logs. (The code is based on Apache Spark 3.1.1 source code, and tested using Spark 3.0.x and 3.1.1 event logs) From e8e03a7bce3247294c79080e4384e01f28007577 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:34:55 -0700 Subject: [PATCH 08/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- .../additional-functionality/qualification-profiling-tools.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 396a3cb6bdd..362f83b43e1 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -37,8 +37,8 @@ This document covers below topics: runs a Spark application so needs the Spark runtime. - Java 8 or above - Spark event log(s) from Spark 2.0 or above version. - Support both rolled and compressed event logs with `.lz4`, `.lzf`, `.snappy` and `.zstd` suffixes. - Also support Databricks specific rolled and compressed(.gz) eventlogs. + Supports both rolled and compressed event logs with `.lz4`, `.lzf`, `.snappy` and `.zstd` suffixes as + well as Databricks-specific rolled and compressed(.gz) eventlogs. The tool does not support nested directories, event log files or event log directories should be at the top level when specifying a directory. From f688b3d39314ba29993fedb546476f3f2f0948bd Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:35:04 -0700 Subject: [PATCH 09/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 362f83b43e1..93f861011cb 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -39,7 +39,7 @@ This document covers below topics: - Spark event log(s) from Spark 2.0 or above version. Supports both rolled and compressed event logs with `.lz4`, `.lzf`, `.snappy` and `.zstd` suffixes as well as Databricks-specific rolled and compressed(.gz) eventlogs. - The tool does not support nested directories, event log files or event log directories should be + The tool does not support nested directories. Event log files or event log directories should be at the top level when specifying a directory. Note: Spark event logs can be downloaded from Spark UI using a "Download" button on the right side, From 7e6892f85b1baafe7877f455d1b04e4c1bf83244 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:35:10 -0700 Subject: [PATCH 10/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 93f861011cb..398b08391e6 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -48,7 +48,7 @@ or can be found in the location specified by `spark.eventLog.dir`. See the more information. Optional: -- maven installed +- Maven installed (only if you want to compile the jar yourself) - hadoop-aws-.jar and aws-java-sdk-.jar (only if any input event log is from S3) From ffb43b0ab6977d415d78b4ff4581088c9dfff0b8 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:35:22 -0700 Subject: [PATCH 11/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 398b08391e6..ed0e1529ded 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -54,7 +54,7 @@ Optional: (only if any input event log is from S3) ## Download the tools jar or compile it -You do not need to compile the jar yourself because you can download it from maven repository directly. +You do not need to compile the jar yourself because you can download it from the Maven repository directly. Here are 2 options: 1. Download the jar file from [maven repository](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/21.06.0/) From d2455a12f068a6643d0e808e883f04d9ae2c3c7f Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:35:29 -0700 Subject: [PATCH 12/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index ed0e1529ded..7ec59a7b64e 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -209,7 +209,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* application based filtering, the order in which filters areapplied is: application-name, start-app-time, - filter-criteria.Application based + filter-criteria. Application based filter-criteria are:100-newest (for processing newest 100 event logs based on timestamp insidethe eventlog) i.e From fa7be3d887d952eb514eebd190f2bc1e2508cba8 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:35:44 -0700 Subject: [PATCH 13/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 7ec59a7b64e..4c81bb245fe 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -210,7 +210,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* which filters areapplied is: application-name, start-app-time, filter-criteria. Application based - filter-criteria are:100-newest (for + filter-criteria are: 100-newest (for processing newest 100 event logs based on timestamp insidethe eventlog) i.e application start time) 100-oldest (for From 98d11918855dc9f091ea612b24e85eab43fa8ad1 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:35:51 -0700 Subject: [PATCH 14/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 4c81bb245fe..59c1359b8be 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -221,7 +221,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* newest log files for each unique application name) 100-oldest-per-app-name (select at most 100 oldest log files for each unique - application name)Filesystem based filter + application name). Filesystem based filter criteria are:100-newest-filesystem (for processing newest 100 event logs based on filesystem timestamp). 100-oldest-filesystem From c9a6ae9e63ee9c688939812fc53e72a142e9a5b0 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:35:57 -0700 Subject: [PATCH 15/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 59c1359b8be..38362f6a269 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -222,7 +222,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* name) 100-oldest-per-app-name (select at most 100 oldest log files for each unique application name). Filesystem based filter - criteria are:100-newest-filesystem (for + criteria are: 100-newest-filesystem (for processing newest 100 event logs based on filesystem timestamp). 100-oldest-filesystem (for processing oldest 100 event logsbased From ed581d8c405c0e4d98e7d9b683d69e560bfcd203 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:36:04 -0700 Subject: [PATCH 16/21] Update tools/README.md Co-authored-by: Jason Lowe --- tools/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/README.md b/tools/README.md index 1100383eba8..51f0d40c350 100644 --- a/tools/README.md +++ b/tools/README.md @@ -8,4 +8,4 @@ Information such as Spark version, executor information, properties and so on. T GPU generated event logs. Please refer to [this documentation](../docs/additional-functionality/qualification-profiling-tools.md) -for more details on how to use the 2 tools. \ No newline at end of file +for more details on how to use the tools. From 7021a48005de8f638a3233dc755f92baba2442bf Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:36:19 -0700 Subject: [PATCH 17/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 38362f6a269..6af2de18695 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -57,7 +57,7 @@ Optional: You do not need to compile the jar yourself because you can download it from the Maven repository directly. Here are 2 options: -1. Download the jar file from [maven repository](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/21.06.0/) +1. Download the jar file from [Maven repository](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/21.06.0/) 2. Compile the jar from github repo ```bash From 4df732a88210d732950d23d392e7b17f38c59209 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:36:27 -0700 Subject: [PATCH 18/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 6af2de18695..b4c1c24de02 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -196,7 +196,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* -a, --application-name Filter event logs whose application name matches exactly or is a substring of input string. Regular expressions not - supported.For filtering based on complement + supported. For filtering based on complement of application name, use ~APPLICATION_NAME. i.e Select all event logs except the ones which have application name as the input From a1f319bdd6e29e83165d5104f3221dd44da65227 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:36:40 -0700 Subject: [PATCH 19/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index b4c1c24de02..1d521bd403e 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -205,7 +205,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* application start timestamp, unique application name or filesystem timestamp. Filesystem based filtering happens before - any application based filtering.For + any application based filtering. For application based filtering, the order in which filters areapplied is: application-name, start-app-time, From 934f610ded32325a70342597f72bf231749a75bb Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:36:48 -0700 Subject: [PATCH 20/21] Update docs/additional-functionality/qualification-profiling-tools.md Co-authored-by: Jason Lowe --- docs/additional-functionality/qualification-profiling-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 1d521bd403e..78a4f02a571 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -207,7 +207,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* Filesystem based filtering happens before any application based filtering. For application based filtering, the order in - which filters areapplied is: + which filters are applied is: application-name, start-app-time, filter-criteria. Application based filter-criteria are: 100-newest (for From 4276496f6854d6fe624d1ebf37840b7cad49842b Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:50:56 -0700 Subject: [PATCH 21/21] Remove some lines. --- docs/additional-functionality/qualification-profiling-tools.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/additional-functionality/qualification-profiling-tools.md b/docs/additional-functionality/qualification-profiling-tools.md index 78a4f02a571..367a32ec0ab 100644 --- a/docs/additional-functionality/qualification-profiling-tools.md +++ b/docs/additional-functionality/qualification-profiling-tools.md @@ -13,8 +13,6 @@ The profiling tool generates information which can be used for debugging and pro The information contains the Spark version, executor details, properties, etc. This runs on either CPU or GPU generated event logs. -(The code is based on Apache Spark 3.1.1 source code, and tested using Spark 3.0.x and 3.1.1 event logs) - This document covers below topics: * [Prerequisites](#Prerequisites)