Getting TiSpark
TiSpark is a third-party jar package for Spark that provides the ability to read/write TiKV
| TiSpark version | TiDB、TiKV、PD version | Spark version | Scala version |
|---|---|---|---|
| 2.4.x-scala_2.11 | 5.x, 4.x | 2.3.x, 2.4.x | 2.11 |
| 2.4.x-scala_2.12 | 5.x, 4.x | 2.4.x | 2.12 |
| 2.5.x | 5.x, 4.x | 3.0.x, 3.1.x | 2.12 |
| 3.0.x | 5.x, 4.x | 3.0.x, 3.1.x, 3.2.x | 2.12 |
- TiSpark 2.4.3, 2.5.1, 3.0.0 is the latest stable version, which is highly recommended.
- Spark < 2.3 is not supported anymore. if your spark version is below 2.3, follow this document to use another version of TiSpark
- get from maven central and search with GroupId
- get from TiSpark releases
- build from source with the steps below
Currently, java8 is the only choice to build TiSpark, run mvn -version to check.
git clone https://github.com/pingcap/tispark.git
Run the following command under the TiSpark root directory:
// add -Dmaven.test.skip=true to skip the tests
mvn clean install -Dmaven.test.skip=true
The Artifact ID of TiSpark is a bit different in different TiSpark version
| TiSpark version | Artifact ID |
|---|---|
| 2.4.x-${scala_version}, 2.5.0 | tispark-assembly |
| 2.5.1 | tispark-assembly-${spark_version} |
| 3.0.x | tispark-assembly-${spark_version}_${scala_version} |