From a3497d784c8618ae518188659632efe0c85b2955 Mon Sep 17 00:00:00 2001 From: Adam Cervenka Date: Tue, 13 Feb 2024 15:29:33 +0100 Subject: [PATCH 1/2] abris add installation documentation --- README.md | 41 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index aae1946..233fb19 100644 --- a/README.md +++ b/README.md @@ -23,12 +23,12 @@ ## Supported versions -| Abris | Spark | Scala | -|:-----: |:-----: |:-----: | -| 6.2.0 - 6.x.x | 3.2.1 - 3.2.x | 2.12 / 2.13 | -| 6.0.0 - 6.1.1 | 3.2.0 | 2.12 / 2.13 | +| Abris | Spark | Scala | +|:-----: |:-------------:|:-----: | +| 6.2.0 - 6.x.x | 3.2.1 - 3.5.x | 2.12 / 2.13 | +| 6.0.0 - 6.1.1 | 3.2.0 | 2.12 / 2.13 | | 5.0.0 - 5.x.x | 3.0.x / 3.1.x | 2.12 | -| 5.0.0 - 5.x.x | 2.4.x | 2.11 / 2.12 | +| 5.0.0 - 5.x.x | 2.4.x | 2.11 / 2.12 | From version 6.0.0, ABRiS only supports Spark 3.2.x. @@ -43,6 +43,37 @@ This is documentation for Abris **version 6**. Documentation for older versions ## Confluent Schema Registry Version Abris by default uses Confluent client version 6.2.0. +## Installation +Abris needs `spark-avro` to run, make sure you include the `spark-avro` dependency when using Abris. +The version of `spark-avro` and `Spark` should be identical. + +Example: submitting a Spark job: +``` +./bin/spark-submit \ + --packages org.apache.spark:spark-avro_2.12:3.5.0,za.co.absa:abris_2.12:6.2.0 \ + ...rest of submit params... +``` + +Example: using Abris in maven project: +```xml + + org.apache.spark + spark-core_2.12 + 3.5.0 + provided + + + org.apache.spark + spark-avro_2.12 + 3.5.0 + + + za.co.absa + abris_2.12 + 6.4.0 + +``` + ## Usage ABRiS API is in it's most basic form almost identical to Spark built-in support for Avro, but it provides additional functionality. From 10a297215a14882f22e7fef9a4429449c33cbc1d Mon Sep 17 00:00:00 2001 From: Adam Cervenka Date: Thu, 22 Feb 2024 10:07:45 +0100 Subject: [PATCH 2/2] code review changes --- README.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 233fb19..4b7860a 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ The version of `spark-avro` and `Spark` should be identical. Example: submitting a Spark job: ``` ./bin/spark-submit \ - --packages org.apache.spark:spark-avro_2.12:3.5.0,za.co.absa:abris_2.12:6.2.0 \ + --packages org.apache.spark:spark-avro_2.12:3.5.0,za.co.absa:abris_2.12:6.4.0 \ ...rest of submit params... ``` @@ -74,6 +74,16 @@ Example: using Abris in maven project: ``` +Example: using Abris in SBT project: +```Scala +libraryDependencies ++= Seq( + "org.apache.spark" %% "spark-core" % "3.5.0" % Provided, + "org.apache.spark" %% "spark-avro" % "3.5.0", + "za.co.absa" %% "abris" % "6.4.0" +) +``` + + ## Usage ABRiS API is in it's most basic form almost identical to Spark built-in support for Avro, but it provides additional functionality.