Update docs for 21.08 release #3080

sameerz · 2021-07-29T08:06:20Z

Signed-off-by: Sameer Raheja sraheja@nvidia.com

Documentation updates for 21.08, prior to release.

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

jlowe

The changes seem OK to me, but I'm wondering if we should be making these changes at the same time we change the pom.xml to reflect what these docs are saying. We've already been bitten in the past with users coming to the dev branch page in github before the release is ready and getting confused. Plus if one actually builds on branch-21.08 in the future, they will produce a 21.08.0-SNAPSHOT artifact, not a 21.08.0 version.

So I'm wondering if these fixups should appear as a separate commit in the merge to the main branch when we're ready to release, just as the pom changes to remove SNAPSHOT also occur.

Note that it would be OK to remove the cudf SNAPSHOT text as soon as branch-21.08 moves to the official cudf-21.08.0 artifact. All I'm advocating for here is to tie the doc changes to the pom.xml changes that are corresponding. Disconnecting them can lead to confusion, as we've encountered in the past.

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

sameerz · 2021-07-29T14:12:03Z

The changes seem OK to me, but I'm wondering if we should be making these changes at the same time we change the pom.xml to reflect what these docs are saying. We've already been bitten in the past with users coming to the dev branch page in github before the release is ready and getting confused. Plus if one actually builds on branch-21.08 in the future, they will produce a 21.08.0-SNAPSHOT artifact, not a 21.08.0 version.

So I'm wondering if these fixups should appear as a separate commit in the merge to the main branch when we're ready to release, just as the pom changes to remove SNAPSHOT also occur.

The question in the past, as I understood it, was whether to make documentation changes after we cut a drop to QA, or giving QA the changes as close as possible to the final version to test. This PR errs on the side of giving QA the doc changes, while recognizing there is a gap period where the links do not match what might be available. It also gives us a chance to "test" the documentation during the virtual review next week. I am open to changing our approach.

jlowe

We can proceed as usual for now, although I'd like to see this be in draft for now and be one of the last commits. The pom hasn't even changed to pull in cudf 21.08.0, for example, so it's a bit early to commit this, IMO.

sameerz · 2021-07-29T15:12:18Z

build

docs/download.md

Salonijain27

lgtm

docs/download.md

Co-authored-by: Hao Zhu <9665750+viadea@users.noreply.github.com>

sameerz · 2021-07-29T20:51:53Z

build

pxLi · 2021-07-30T07:19:28Z

cudf and cudfjni 21.08.0 have not be released yet (plan on Aug 5), we will update pom dependency after cudfjni 21.08.0 available in maven central. I agree that we can keep the development branch to stick w/ -SNAPSHOT, and only update doc to release branch (main)

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

pxLi · 2021-08-05T01:40:53Z

hi looks like this PR did not include version update to https://github.com/NVIDIA/spark-rapids/blob/branch-21.08/docs/get-started/getting-started-on-prem.md#download-the-rapids-jars

is this intentional to keep the sample version as snapshot here?

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

sameerz · 2021-08-05T05:28:11Z

hi looks like this PR did not include version update to https://github.com/NVIDIA/spark-rapids/blob/branch-21.08/docs/get-started/getting-started-on-prem.md#download-the-rapids-jars
is this intentional to keep the sample version as snapshot here?

That was a miss. I also updated the README.md for the integration_tests.

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

sameerz · 2021-08-05T18:02:47Z

build

sameerz · 2021-08-05T21:38:49Z

@pxLi @GaryShen2008 @NvTimLiu this is ready to merge right before you create the CHANGELOG for 21.08.

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

sameerz · 2021-08-06T16:52:55Z

build

pxLi · 2021-08-07T02:09:40Z

As cudf tagged release v21.08.02, I think we could go directly w/ dep cudfjni 21.08.2,
cudf-21.08.0-cuda11 -> cudf-21.08.2-cuda11

pxLi · 2021-08-09T00:50:58Z

docs/configs.md

@@ -10,7 +10,7 @@ The following is the list of options that `rapids-plugin-4-spark` supports.
 On startup use: `--conf [conf key]=[conf value]`. For example:

 ```
-${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar,cudf-21.08.0-SNAPSHOT-cuda11.jar' \
+${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar' \


Suggested change

${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar' \

${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.2-cuda11.jar' \

pxLi · 2021-08-09T00:52:05Z

docs/demo/Databricks/generate-init-script-cuda11.ipynb

@@ -1 +1 @@
-{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.06.1.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.06.1/rapids-4-spark_2.12-21.06.1.jar\nsudo wget -O /databricks/jars/cudf-21.06.1-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.06.1/cudf-21.06.1-cuda11.jar\n\nsudo wget -O /etc/apt/preferences.d/cuda-repository-pin-600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin\nsudo wget -O ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb https://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo dpkg -i ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo apt-key add /var/cuda-repo-ubuntu1804-11-0-local/7fa2af80.pub\nsudo apt-get update\nsudo apt -y install cuda-toolkit-11-0\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}
+{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.08.0.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar\nsudo wget -O /databricks/jars/cudf-21.08.0-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.0/cudf-21.08.0-cuda11.jar\n\nsudo wget -O /etc/apt/preferences.d/cuda-repository-pin-600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin\nsudo wget -O ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb https://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo dpkg -i ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo apt-key add /var/cuda-repo-ubuntu1804-11-0-local/7fa2af80.pub\nsudo apt-get update\nsudo apt -y install cuda-toolkit-11-0\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}


Suggested change

{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.08.0.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar\nsudo wget -O /databricks/jars/cudf-21.08.0-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.0/cudf-21.08.0-cuda11.jar\n\nsudo wget -O /etc/apt/preferences.d/cuda-repository-pin-600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin\nsudo wget -O ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb https://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo dpkg -i ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo apt-key add /var/cuda-repo-ubuntu1804-11-0-local/7fa2af80.pub\nsudo apt-get update\nsudo apt -y install cuda-toolkit-11-0\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}

{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.08.0.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar\nsudo wget -O /databricks/jars/cudf-21.08.2-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.2/cudf-21.08.2-cuda11.jar\n\nsudo wget -O /etc/apt/preferences.d/cuda-repository-pin-600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin\nsudo wget -O ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb https://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo dpkg -i ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo apt-key add /var/cuda-repo-ubuntu1804-11-0-local/7fa2af80.pub\nsudo apt-get update\nsudo apt -y install cuda-toolkit-11-0\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}

pxLi · 2021-08-09T00:52:43Z

docs/demo/Databricks/generate-init-script.ipynb

@@ -1 +1 @@
-{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.06.1.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.06.1/rapids-4-spark_2.12-21.06.1.jar\nsudo wget -O /databricks/jars/cudf-21.06.1-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.06.1/cudf-21.06.1-cuda11.jar\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}
+{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.08.0.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar\nsudo wget -O /databricks/jars/cudf-21.08.0-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.0/cudf-21.08.0-cuda11.jar\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}


Suggested change

{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.08.0.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar\nsudo wget -O /databricks/jars/cudf-21.08.0-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.0/cudf-21.08.0-cuda11.jar\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}

{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.08.0.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar\nsudo wget -O /databricks/jars/cudf-21.08.2-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.2/cudf-21.08.2-cuda11.jar\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}

pxLi · 2021-08-09T00:53:07Z

docs/download.md

+### Download v21.08.0
+* Download the [RAPIDS
+  Accelerator for Apache Spark 21.08.0 jar](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar)
+* Download the [RAPIDS cuDF 21.08.0 jar](https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.0/cudf-21.08.0-cuda11.jar)


Suggested change

* Download the [RAPIDS cuDF 21.08.0 jar](https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.0/cudf-21.08.0-cuda11.jar)

* Download the [RAPIDS cuDF 21.08.2 jar](https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.2/cudf-21.08.2-cuda11.jar)

pxLi · 2021-08-09T00:53:31Z

docs/get-started/Dockerfile.cuda

@@ -50,8 +50,8 @@ COPY spark-3.0.2-bin-hadoop3.2/examples /opt/spark/examples
 COPY spark-3.0.2-bin-hadoop3.2/kubernetes/tests /opt/spark/tests
 COPY spark-3.0.2-bin-hadoop3.2/data /opt/spark/data

-COPY cudf-21.08.0-SNAPSHOT-cuda11.jar /opt/sparkRapidsPlugin
-COPY rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar /opt/sparkRapidsPlugin
+COPY cudf-21.08.0-cuda11.jar /opt/sparkRapidsPlugin


Suggested change

COPY cudf-21.08.0-cuda11.jar /opt/sparkRapidsPlugin

COPY cudf-21.08.2-cuda11.jar /opt/sparkRapidsPlugin

pxLi · 2021-08-09T00:53:47Z

docs/get-started/getting-started-on-prem.md

@@ -55,17 +55,17 @@ CUDA and will not run on other versions. The jars use a maven classifier to keep
 - CUDA 11.0/11.1/11.2 => classifier cuda11

 For example, here is a sample version of the jars and cudf with CUDA 11.0 support:
- cudf-21.08.0-SNAPSHOT-cuda11.jar
- rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar
+- cudf-21.08.0-cuda11.jar


Suggested change

- cudf-21.08.0-cuda11.jar

- cudf-21.08.2-cuda11.jar

pxLi · 2021-08-09T00:54:29Z

docs/get-started/getting-started-on-prem.md

 jar that your version of the accelerator depends on.


 For simplicity export the location to these jars. This example assumes the sample jars above have
 been placed in the `/opt/sparkRapidsPlugin` directory:
 ```shell 
 export SPARK_RAPIDS_DIR=/opt/sparkRapidsPlugin
-export SPARK_CUDF_JAR=${SPARK_RAPIDS_DIR}/cudf-21.08.0-SNAPSHOT-cuda11.jar
-export SPARK_RAPIDS_PLUGIN_JAR=${SPARK_RAPIDS_DIR}/rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar
+export SPARK_CUDF_JAR=${SPARK_RAPIDS_DIR}/cudf-21.08.0-cuda11.jar


Suggested change

export SPARK_CUDF_JAR=${SPARK_RAPIDS_DIR}/cudf-21.08.0-cuda11.jar

export SPARK_CUDF_JAR=${SPARK_RAPIDS_DIR}/cudf-21.08.2-cuda11.jar

pxLi · 2021-08-09T00:54:53Z

integration_tests/README.md

@@ -157,7 +157,7 @@ If you just want to verify the SQL replacement is working you will need to add t
 example assumes CUDA 11.0 is being used.

 ```
-$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar,rapids-4-spark-udf-examples_2.12-21.08.0-SNAPSHOT.jar,cudf-21.08.0-SNAPSHOT-cuda11.jar" ./runtests.py
+$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0.jar,rapids-4-spark-udf-examples_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar" ./runtests.py


Suggested change

$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0.jar,rapids-4-spark-udf-examples_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar" ./runtests.py

$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0.jar,rapids-4-spark-udf-examples_2.12-21.08.0.jar,cudf-21.08.2-cuda11.jar" ./runtests.py

pxLi · 2021-08-09T00:55:08Z

integration_tests/README.md

@@ -256,7 +256,7 @@ To run cudf_udf tests, need following configuration changes:
 As an example, here is the `spark-submit` command with the cudf_udf parameter on CUDA 11.0:

 ```
-$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar,rapids-4-spark-udf-examples_2.12-21.08.0-SNAPSHOT.jar,cudf-21.08.0-SNAPSHOT-cuda11.jar,rapids-4-spark-tests_2.12-21.08.0-SNAPSHOT.jar" --conf spark.rapids.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.concurrentPythonWorkers=2 --py-files "rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar" --conf spark.executorEnv.PYTHONPATH="rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar" ./runtests.py --cudf_udf
+$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0.jar,rapids-4-spark-udf-examples_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar,rapids-4-spark-tests_2.12-21.08.0.jar" --conf spark.rapids.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.concurrentPythonWorkers=2 --py-files "rapids-4-spark_2.12-21.08.0.jar" --conf spark.executorEnv.PYTHONPATH="rapids-4-spark_2.12-21.08.0.jar" ./runtests.py --cudf_udf


Suggested change

$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0.jar,rapids-4-spark-udf-examples_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar,rapids-4-spark-tests_2.12-21.08.0.jar" --conf spark.rapids.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.concurrentPythonWorkers=2 --py-files "rapids-4-spark_2.12-21.08.0.jar" --conf spark.executorEnv.PYTHONPATH="rapids-4-spark_2.12-21.08.0.jar" ./runtests.py --cudf_udf

$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0.jar,rapids-4-spark-udf-examples_2.12-21.08.0.jar,cudf-21.08.2-cuda11.jar,rapids-4-spark-tests_2.12-21.08.0.jar" --conf spark.rapids.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.concurrentPythonWorkers=2 --py-files "rapids-4-spark_2.12-21.08.0.jar" --conf spark.executorEnv.PYTHONPATH="rapids-4-spark_2.12-21.08.0.jar" ./runtests.py --cudf_udf

pxLi · 2021-08-09T00:55:19Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala

@@ -1280,7 +1280,7 @@ object RapidsConf {
        |On startup use: `--conf [conf key]=[conf value]`. For example:
        |
        |```
-        |${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0-SNAPSHOT.jar,cudf-21.08.0-SNAPSHOT-cuda11.jar' \
+        |${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar' \


Suggested change

|${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar' \

|${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.2-cuda11.jar' \

pxLi · 2021-08-09T00:59:26Z

We are going to deploy cudf jni 21.08.2 to maven central, I will merge this doc update first and will cover the cudf 21.08.0 -> 21.08.2 doc update in #3146

* Update docs for 21.08 release (#3080) * Update docs for 21.08 release Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Modify "or higher" -> "or later" when referring to versions Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Update docs/download.md Co-authored-by: Hao Zhu <9665750+viadea@users.noreply.github.com> * Move AQE for developers doc into development section of web site Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Add 21.06.1 release information to the download page Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Add information about kryoserialization buffer overflow Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Remove SNAPSHOT from getting-started-on-prem.md Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Update README.md for integration tests to remove SNAPSHOT Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Fix table in supported ops doc issue #2779 Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Fix typo (Abillity -> Ability) Signed-off-by: Sameer Raheja <sraheja@nvidia.com> Co-authored-by: Hao Zhu <9665750+viadea@users.noreply.github.com> Signed-off-by: yuali <yuali@nvidia.com> * Update docs for 21.08 release (#3080) * Update docs for 21.08 release Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Modify "or higher" -> "or later" when referring to versions Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Update docs/download.md Co-authored-by: Hao Zhu <9665750+viadea@users.noreply.github.com> * Move AQE for developers doc into development section of web site Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Add 21.06.1 release information to the download page Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Add information about kryoserialization buffer overflow Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Remove SNAPSHOT from getting-started-on-prem.md Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Update README.md for integration tests to remove SNAPSHOT Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Fix table in supported ops doc issue #2779 Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Fix typo (Abillity -> Ability) Signed-off-by: Sameer Raheja <sraheja@nvidia.com> Co-authored-by: Hao Zhu <9665750+viadea@users.noreply.github.com> Signed-off-by: yuali <yuali@nvidia.com> * update config.md and supported_ops.md Signed-off-by: yuali <yuali@nvidia.com> * update rapids-shuffle.md and download.md Signed-off-by: yuali <yuali@nvidia.com> * update dev/README.md and formated Signed-off-by: yuali <yuali@nvidia.com> * fix typos Signed-off-by: yuali <yuali@nvidia.com> Co-authored-by: Hao Zhu <9665750+viadea@users.noreply.github.com>

Update docs for 21.08 release

0386f7d

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

sameerz added the documentation Improvements or additions to documentation label Jul 29, 2021

sameerz added this to the July 19 - July 30 milestone Jul 29, 2021

sameerz requested review from jlowe, pxLi and viadea July 29, 2021 08:06

sameerz self-assigned this Jul 29, 2021

jlowe reviewed Jul 29, 2021

View reviewed changes

Modify "or higher" -> "or later" when referring to versions

758bd01

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

jlowe previously approved these changes Jul 29, 2021

View reviewed changes

sameerz marked this pull request as draft July 29, 2021 15:12

Salonijain27 reviewed Jul 29, 2021

View reviewed changes

docs/download.md Show resolved Hide resolved

Salonijain27 previously approved these changes Jul 29, 2021

View reviewed changes

viadea reviewed Jul 29, 2021

View reviewed changes

docs/download.md Outdated Show resolved Hide resolved

Update docs/download.md

2921966

Co-authored-by: Hao Zhu <9665750+viadea@users.noreply.github.com>

sameerz dismissed stale reviews from Salonijain27 and jlowe via 2921966 July 29, 2021 18:26

sameerz modified the milestones: July 19 - July 30, Aug 2 - Aug 13 Jul 30, 2021

sameerz and others added 4 commits July 31, 2021 11:41

Move AQE for developers doc into development section of web site

9d085fe

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

Update a change made in the remote repository based on PR feedback

eebda32

Add 21.06.1 release information to the download page

92e4914

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

Merge branch 'branch-21.08' into branch-21.08-doc-updates

dc7d303

pxLi mentioned this pull request Aug 5, 2021

update cudf Java binding version to 21.08.2 #3146

Merged

sameerz added 2 commits August 4, 2021 22:21

Remove SNAPSHOT from getting-started-on-prem.md

c3290d3

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

Update README.md for integration tests to remove SNAPSHOT

4a54003

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

jlowe previously approved these changes Aug 5, 2021

View reviewed changes

Fix table in supported ops doc issue NVIDIA#2779

d4b3ef5

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

sameerz dismissed jlowe’s stale review via d4b3ef5 August 5, 2021 17:52

sameerz linked an issue Aug 5, 2021 that may be closed by this pull request

[DOC] Supported Ops Input/Output section combined with Partitioning section #2779

Closed

jlowe previously approved these changes Aug 5, 2021

View reviewed changes

sameerz marked this pull request as ready for review August 5, 2021 21:38

Fix typo (Abillity -> Ability)

1264395

Signed-off-by: Sameer Raheja <sraheja@nvidia.com>

sameerz dismissed jlowe’s stale review via 1264395 August 6, 2021 16:52

jlowe approved these changes Aug 6, 2021

View reviewed changes

pxLi reviewed Aug 9, 2021

View reviewed changes

pxLi merged commit 1bbbc4f into NVIDIA:branch-21.08 Aug 9, 2021

nvliyuan mentioned this pull request Aug 10, 2021

Update gh-pages for 21.08 release #3035

Closed

sameerz deleted the branch-21.08-doc-updates branch April 21, 2022 00:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update docs for 21.08 release #3080

Update docs for 21.08 release #3080

sameerz commented Jul 29, 2021

jlowe left a comment

sameerz commented Jul 29, 2021

jlowe left a comment

sameerz commented Jul 29, 2021

Salonijain27 left a comment

sameerz commented Jul 29, 2021

pxLi commented Jul 30, 2021 •

edited

Loading

pxLi commented Aug 5, 2021

sameerz commented Aug 5, 2021

sameerz commented Aug 5, 2021

sameerz commented Aug 5, 2021

sameerz commented Aug 6, 2021

pxLi commented Aug 7, 2021 •

edited

Loading

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi Aug 9, 2021

pxLi commented Aug 9, 2021

	${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar' \
	${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.2-cuda11.jar' \

		@@ -1 +1 @@
		{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.06.1.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.06.1/rapids-4-spark_2.12-21.06.1.jar\nsudo wget -O /databricks/jars/cudf-21.06.1-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.06.1/cudf-21.06.1-cuda11.jar\n\nsudo wget -O /etc/apt/preferences.d/cuda-repository-pin-600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin\nsudo wget -O ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb https://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo dpkg -i ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo apt-key add /var/cuda-repo-ubuntu1804-11-0-local/7fa2af80.pub\nsudo apt-get update\nsudo apt -y install cuda-toolkit-11-0\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}
		{"cells":[{"cell_type":"code","source":["dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")\n \ndbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n#!/bin/bash\nsudo wget -O /databricks/jars/rapids-4-spark_2.12-21.08.0.jar https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar\nsudo wget -O /databricks/jars/cudf-21.08.0-cuda11.jar https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.0/cudf-21.08.0-cuda11.jar\n\nsudo wget -O /etc/apt/preferences.d/cuda-repository-pin-600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin\nsudo wget -O ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb https://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo dpkg -i ~/cuda-repo-ubuntu1804-11-0-local_11.0.3-450.51.06-1_amd64.deb\nsudo apt-key add /var/cuda-repo-ubuntu1804-11-0-local/7fa2af80.pub\nsudo apt-get update\nsudo apt -y install cuda-toolkit-11-0\"\"\", True)"],"metadata":{},"outputs":[],"execution_count":1},{"cell_type":"code","source":["%sh\ncd ../../dbfs/databricks/init_scripts\npwd\nls -ltr\ncat init.sh"],"metadata":{},"outputs":[],"execution_count":2},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":3}],"metadata":{"name":"generate-init-script","notebookId":2645746662301564},"nbformat":4,"nbformat_minor":0}

	* Download the [RAPIDS cuDF 21.08.0 jar](https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.0/cudf-21.08.0-cuda11.jar)
	* Download the [RAPIDS cuDF 21.08.2 jar](https://repo1.maven.org/maven2/ai/rapids/cudf/21.08.2/cudf-21.08.2-cuda11.jar)

	COPY cudf-21.08.0-cuda11.jar /opt/sparkRapidsPlugin
	COPY cudf-21.08.2-cuda11.jar /opt/sparkRapidsPlugin

	export SPARK_CUDF_JAR=${SPARK_RAPIDS_DIR}/cudf-21.08.0-cuda11.jar
	export SPARK_CUDF_JAR=${SPARK_RAPIDS_DIR}/cudf-21.08.2-cuda11.jar

	$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0.jar,rapids-4-spark-udf-examples_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar" ./runtests.py
	$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-21.08.0.jar,rapids-4-spark-udf-examples_2.12-21.08.0.jar,cudf-21.08.2-cuda11.jar" ./runtests.py

	\|${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.0-cuda11.jar' \
	\|${SPARK_HOME}/bin/spark --jars 'rapids-4-spark_2.12-21.08.0.jar,cudf-21.08.2-cuda11.jar' \

Update docs for 21.08 release #3080

Update docs for 21.08 release #3080

Conversation

sameerz commented Jul 29, 2021

jlowe left a comment

Choose a reason for hiding this comment

sameerz commented Jul 29, 2021

jlowe left a comment

Choose a reason for hiding this comment

sameerz commented Jul 29, 2021

Salonijain27 left a comment

Choose a reason for hiding this comment

sameerz commented Jul 29, 2021

pxLi commented Jul 30, 2021 • edited Loading

pxLi commented Aug 5, 2021

sameerz commented Aug 5, 2021

sameerz commented Aug 5, 2021

sameerz commented Aug 5, 2021

sameerz commented Aug 6, 2021

pxLi commented Aug 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pxLi commented Aug 9, 2021

pxLi commented Jul 30, 2021 •

edited

Loading

pxLi commented Aug 7, 2021 •

edited

Loading