Getting TiSpark
TiSpark is a third-party jar package for Spark that provides the ability to read/write TiKV
We will not provide the mysql-connector-java dependency because of the limit of the GPL license.
The following versions of TiSpark's jar will no longer include mysql-connector-java.
- TiSpark > 3.0.1
- TiSpark > 2.5.1 for TiSpark 2.5.x
- TiSpark > 2.4.3 for TiSpark 2.4.x
Now, TiSpark needs mysql-connector-java for writing and auth. Please import mysql-connector-java manually when you need to write or auth.
-
you can import it by putting the jar into spark jars file
-
you can also import it when you submit spark job like
spark-submit --jars tispark-assembly-3.0_2.12-3.1.0-SNAPSHOT.jar,mysql-connector-java-8.0.29.jar
| TiSpark version | TiDB、TiKV、PD version | Spark version | Scala version |
|---|---|---|---|
| 2.4.x-scala_2.11 | 5.x, 4.x | 2.3.x, 2.4.x | 2.11 |
| 2.4.x-scala_2.12 | 5.x, 4.x | 2.4.x | 2.12 |
| 2.5.x | 5.x, 4.x | 3.0.x, 3.1.x | 2.12 |
| 3.0.x | 5.x, 4.x | 3.0.x, 3.1.x, 3.2.x | 2.12 |
- TiSpark 2.4.3, 2.5.1, 3.0.1 is the latest stable version, which is highly recommended.
- Spark < 2.3 is not supported anymore. if your spark version is below 2.3, follow this document to use another version of TiSpark
- get from maven central and search with GroupId
- get from TiSpark releases
- build from source with the steps below
Currently, java8 is the only choice to build TiSpark, run mvn -version to check.
git clone https://github.com/pingcap/tispark.git
Run the following command under the TiSpark root directory:
// add -Dmaven.test.skip=true to skip the tests
mvn clean install -Dmaven.test.skip=true
// or you can add properties to specify spark version
mvn clean install -Dmaven.test.skip=true -Pspark3.2.1
The Artifact ID of TiSpark is a bit different in different TiSpark version
| TiSpark version | Artifact ID |
|---|---|
| 2.4.x-${scala_version}, 2.5.0 | tispark-assembly |
| 2.5.1 | tispark-assembly-${spark_version} |
| 3.0.x | tispark-assembly-${spark_version}_${scala_version} |