Code

Feature/logisland 532#553

Code

Merged

oalam merged 43 commits intodevelopHurence/logisland:developfrom feature/LOGISLAND-532Hurence/logisland:feature/LOGISLAND-532Copy head branch name to clipboardApr 9, 2020

+7,545-445

Commits

Commits on Feb 4, 2020

• Fix url for jcenter.bintray.com repository: from http to https

  mathieu-rossignol

  committed
  1bc36df

Commits on Feb 5, 2020

• Added databricks option to support databricks run: load config from DBFS, using the DB spark context

  mathieu-rossignol

  committed
  919c337

• Introducing SparkConfigReader class to be packaged and loaded only in databricks environment (no more useless spark dependency when outside of databricks)

  mathieu-rossignol

  committed
  587814e

Commits on Feb 10, 2020

• Introducing logisland-engine-spark_2_4plus_kafka_2_4plus

  mathieu-rossignol

  committed
  25fddb0

• Attempt to add -checkpointLocation option

  mathieu-rossignol

  committed
  4a36bdb

Commits on Feb 11, 2020

• Remove building of spark2.4_kafka2.4 engine for the moment

  mathieu-rossignol

  committed
  f44321a

• Added spark 1.4 support in 2.x engine. Also added azure eventhubs spark streaming dependencies

  mathieu-rossignol

  committed
  bfdf3ec

Commits on Feb 13, 2020

• Saving intermediate step towards new service to support azure event hubs

  mathieu-rossignol

  committed
  bee69e0

• Eventhbubs service: all config properties read

  mathieu-rossignol

  committed
  da0ce2e

• First attempt to have azure event hubs service running, will have to test now...

  mathieu-rossignol

  committed
  1075c46

• Fixed spark 2.4 module pom mqtt dependency

  mathieu-rossignol

  committed
  ab6a254

Commits on Feb 14, 2020

• Using checkpoint location in azure structured stream service. Also fixed a bug where read position configiguration property cannot be anything else than string

  mathieu-rossignol

  committed
  3491194

• Support spark 2.4 engine in logisland.sh script

  mathieu-rossignol

  committed
  65d2f46

Commits on Feb 17, 2020

• Update version of spark-sql-streaming-mqtt_2.11 dependency

  mathieu-rossignol

  committed
  dc360dd

• Removed useless parenthesis

  mathieu-rossignol

  committed
  086aeef

• Added spark 2 common jar to logisland doc dependency

  mathieu-rossignol

  committed
  7ffb978

Commits on Feb 18, 2020

• Use right scala version

  mathieu-rossignol

  committed
  280e34d

• Explicitly adding bahir-common dependency in spark2.4 engine, as a workaround to class not found for org.apache.bahir.utils.Logging when using AzureEventHubsStructuredStreamProviderService

  mathieu-rossignol

  committed
  8d0ea78

Commits on Feb 19, 2020

• Set checkpointlocation in structured stream

  mathieu-rossignol

  committed
  f59dabc

Commits on Feb 21, 2020

• Added spark deploy mode property to support sparl standlone cluster deploy mode

  mathieu-rossignol

  committed
  bbf270e

• Introudcing support of spark standalone cluster in logisland.sh

  mathieu-rossignol

  committed
  2a719c5

Commits on Feb 26, 2020

• Fix for ClassNotFoundException in databricks environment on executor: the plugin registry is not filled as the current thread context classloader is a repl.ExecutorClassLoader, so class of plugins …

  Show description for 6bd742b
  mathieu-rossignol

  committed
  6bd742b

• Need spark 2.4.0 in spark 2.4 engine to align with latest supported version of azure event hub spark structured stream source/sink

  mathieu-rossignol

  committed
  337f895

Commits on Feb 27, 2020

• Use offset as record key when reading from azure event hub. Also removing some debug traces and adding some info logs

  mathieu-rossignol

  committed
  7f5c0f7

• Remove not finished and useless engine for spark 2.4+ and kafka 2.4+

  mathieu-rossignol

  committed
  2e1de7d

• Update user doc after full build

  mathieu-rossignol

  committed
  94732be

Commits on Feb 28, 2020

• Added experimental logisland.sh feature that allows minimal launch on spark standalone cluster in client mode, not tested cluster mode. Also to be improved

  mathieu-rossignol

  committed
  b4d1f77

• Added OpenDistro Elasticsearch integration test. For the moment, only user password without https is supported and validated

  mathieu-rossignol

  committed
  af2e8b7

• Centralized opendistro user name and password in integration tests

  mathieu-rossignol

  committed
  bd0636a

• update test containers version in ES7 integration test

  mathieu-rossignol

  committed
  85b1c6e

• No more need of mock service in ES 6 and 7 integration test: use real client connection created by the real service himself

  mathieu-rossignol

  committed
  e5cec1c

• Support for HTTPS opendistro elasticsearch

  mathieu-rossignol

  committed
  2607fac

Commits on Mar 10, 2020

• Added comment in ES7 service regarding SSL

  mathieu-rossignol

  committed
  e66b83a

Commits on Mar 11, 2020

• Added support for avro serializer/deserializer in structured streams by supporting input/output avro schema configuration properties

  mathieu-rossignol

  committed
  0b36c05

• From avro 1.8.2 to avro 1.9.2 to have avro schema serializable and thus now usable in structured streams

  mathieu-rossignol

  committed
  557cbec

Commits on Mar 12, 2020

• Forgotten close when loading maxmind db from URI. Alos upgraded versions of iptogeo service and processor dependencies (jackson/hadoop)

  mathieu-rossignol

  committed
  84cb63a

• Changed log messages when loading maxmind DB from URI

  mathieu-rossignol

  committed
  dd876d0

Commits on Mar 13, 2020

• Rollback park-sql-streaming-mqtt_2.11 version to not have conflict with bahir underlying dependency that uses scala 2.12 when using 2.4.0 version (don't know why)

  mathieu-rossignol

  committed
  4343104

Commits on Mar 16, 2020

• Fixed Record JSON deserialization problem after jackson version bump. Also attempting to align bumped jackson version everywhere where possible and also use the jackson.version property

  mathieu-rossignol

  committed
  0e5a5fa

• Changed http://jcenter.bintray.com into https://jcenter.bintray.com in ivy.xml

  mathieu-rossignol

  committed
  c531acd

• Added https://repo1.maven.org/maven2 repo to ivy.xml

  mathieu-rossignol

  committed
  2159598

Commits on Mar 17, 2020

• Fix logisland.sh script using wrong spark engine: when spark 2.3.3 it was using 2.4 engine, this made fail the structured stream travis integration test

  mathieu-rossignol

  committed
  15ffd64

Commits on Mar 18, 2020

• Try to fix 20 minutes timeout when running travis integration tests

  mathieu-rossignol

  committed
  aaae6be