Add Spark 3.x support for sarplus #1566

simonzhaoms · 2021-11-22T07:58:25Z

Description

Add Spark 3.x support for sarplus

Related Issues

Checklist:

I have followed the contribution guidelines and code style for this project.
I have added tests covering my contributions.
I have updated the documentation accordingly.
This PR is being made to staging branch and not to main branch.

miguelgfierro

Great work!

contrib/sarplus/python/tests/test_pyspark_sar.py

contrib/sarplus/scala/python/setup.py

gramhagen · 2021-11-22T14:29:37Z

contrib/sarplus/DEVELOPMENT.md

@@ -39,3 +39,33 @@ sbt test
 ```

 (use ~test and it will automatically check for changes in source files, but not build.sbt)
+


are the previous instructions still functional (e.g. sbt spPublish?)

if not I suggest removing the old instructions to avoid confusion

OK, let me try to update the instructions as well

Updated. Please take a look and leave comments if any improvements

contrib/sarplus/scala/project/plugins.sbt

gramhagen · 2021-11-22T14:31:54Z

does this work with the existing azureml pipeline? if not, maybe we can make an issue to create a github pipeline to build release artifacts for sarplus

gramhagen · 2021-11-22T14:32:13Z

pipeline I'm talking about is this one: https://github.com/simonzhaoms/recommenders/blob/simonz/sarplus/spark3/contrib/sarplus/azure-pipelines.yml

contrib/sarplus/python/setup.py

contrib/sarplus/scala/python/setup.py

simonzhaoms · 2021-11-23T07:24:41Z

does this work with the existing azureml pipeline? if not, maybe we can make an issue to create a github pipeline to build release artifacts for sarplus

I don't think the pipeline work with this upgrade, will update the pipeline later.

codecov-commenter · 2021-11-23T08:16:38Z

Codecov Report

Merging #1566 (594674c) into staging (9117189) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           staging    #1566      +/-   ##
===========================================
- Coverage    61.89%   61.88%   -0.02%     
===========================================
  Files           84       84              
  Lines         8458     8458              
===========================================
- Hits          5235     5234       -1     
- Misses        3223     3224       +1

Flag	Coverage Δ
pr-gate	`61.88% <ø> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
recommenders/evaluation/spark_evaluation.py	`86.60% <0.00%> (-0.45%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b2b9a4...594674c. Read the comment docs.

miguelgfierro · 2021-11-24T09:24:26Z

hey @simonzhaoms in regards to:

does this work with the existing azureml pipeline? if not, maybe we can make an issue to create a github pipeline to build release artifacts for sarplus. pipeline I'm talking about is this one: https://github.com/simonzhaoms/recommenders/blob/simonz/sarplus/spark3/contrib/sarplus/azure-pipelines.yml

@laserprec is our master pipeliner, he might have some code you can reuse

miguelgfierro · 2021-11-24T09:29:36Z

btw, @simonzhaoms I noticed that you are doing the PR from a fork. I made you admin of the repo. I did the same with @angusrtaylor. Now you can create branches and do PRs from there, there is no need to fork.

miguelgfierro · 2021-11-29T09:56:03Z

@laserprec is there a way to avoid triggering the tests if there is a change in contrib or the markdown files?

In ADO, I did it like this: https://github.com/microsoft/recommenders/blob/main/tests/ci/azure_pipeline_test/dsvm_notebook_linux_cpu.yml#L10
I guess there should be a similar way in Github acctions

laserprec · 2021-11-30T16:16:27Z

@laserprec is there a way to avoid triggering the tests if there is a change in contrib or the markdown files?

In ADO, I did it like this: https://github.com/microsoft/recommenders/blob/main/tests/ci/azure_pipeline_test/dsvm_notebook_linux_cpu.yml#L10 I guess there should be a similar way in Github acctions

Yes, there are similar mechanics in GitHub Action too: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#example-ignoring-paths.

anargyri · 2021-12-07T15:20:18Z

@simonzhaoms I would suggest that you merge staging into this PR in order to get the latest test pipelines which should fix the failing tests.

.github/workflows/sarplus.yml

contrib/sarplus/DEVELOPMENT.md

contrib/sarplus/python/pysarplus/SARPlus.py

contrib/sarplus/scala/python/setup.py

contrib/sarplus/VERSION

…nz/sarplus/spark3

miguelgfierro

@simonzhaoms this is great, please, see my comments, the only thing I'm a little bit concerned is the token

contrib/sarplus/python/pysarplus/__init__.py

contrib/sarplus/python/setup.py

contrib/sarplus/python/tests/test_pyspark_sar.py

contrib/sarplus/scala/build.sbt

…/recommenders into simonz/sarplus/spark3

miguelgfierro

this is amazing work Simon!

simonzhaoms added 2 commits November 22, 2021 14:32

Upgrade sarplus to support Spark 3.x

0d5fa3d

Add corresponding docs and simplify version specification

66b76d9

simonzhaoms requested review from anargyri, gramhagen, loomlike, miguelgfierro and wutaomsft as code owners November 22, 2021 07:58

miguelgfierro approved these changes Nov 22, 2021

View reviewed changes

contrib/sarplus/python/tests/test_pyspark_sar.py Outdated Show resolved Hide resolved

gramhagen reviewed Nov 22, 2021

View reviewed changes

contrib/sarplus/scala/python/setup.py Outdated Show resolved Hide resolved

gramhagen reviewed Nov 22, 2021

View reviewed changes

contrib/sarplus/scala/project/plugins.sbt Show resolved Hide resolved

gramhagen reviewed Nov 22, 2021

View reviewed changes

contrib/sarplus/python/setup.py Outdated Show resolved Hide resolved

gramhagen requested changes Nov 22, 2021

View reviewed changes

contrib/sarplus/scala/python/setup.py Outdated Show resolved Hide resolved

Update python package url

b5d3b55

Add macros for Spark 3.2.x

dd1ceeb

Add sarplus testing and packaging workflow

24b4d48

anargyri reviewed Dec 7, 2021

View reviewed changes

.github/workflows/sarplus.yml Outdated Show resolved Hide resolved

anargyri reviewed Dec 7, 2021

View reviewed changes

contrib/sarplus/DEVELOPMENT.md Outdated Show resolved Hide resolved

anargyri reviewed Dec 7, 2021

View reviewed changes

contrib/sarplus/python/pysarplus/SARPlus.py Outdated Show resolved Hide resolved

anargyri approved these changes Dec 7, 2021

View reviewed changes

anargyri reviewed Dec 7, 2021

View reviewed changes

contrib/sarplus/scala/python/setup.py Show resolved Hide resolved

simonzhaoms and others added 8 commits December 13, 2021 15:34

Update workflow

d2dbe11

Remove unused code

87d78d8

Remove azure-pipelines.yml

52376d1

Update DEVELOPMENT.md

5c786fb

Update README.md

f1f8cf0

Update sarplus.yml

da74735

Add link to publish scala package manually to central repository

8c696e4

Merge branch 'staging' into simonz/sarplus/spark3

2da6acd

simonzhaoms commented Dec 13, 2021

View reviewed changes

contrib/sarplus/VERSION Show resolved Hide resolved

simonzhaoms added 2 commits December 13, 2021 18:40

Add docstring for SARPlus init function

1094e2a

Merge remote-tracking branch 'origin/simonz/sarplus/spark3' into simo…

6845051

…nz/sarplus/spark3

miguelgfierro reviewed Dec 13, 2021

View reviewed changes

anargyri and others added 14 commits December 13, 2021 11:35

Merge branch 'staging' into simonz/sarplus/spark3

bf1f0ec

Use VERSION

fc7cab2

Merge branch 'simonz/sarplus/spark3' of simonzhaomsgithub:simonzhaoms…

932c698

…/recommenders into simonz/sarplus/spark3

Add simon in AUTHORS.md

34c72f7

Merge branch 'staging' into simonz/sarplus/spark3

ebff27b

Remove GPG key

fa9ece8

Update sarplus.yml

d016370

Resolve flake8 errors

b8f31f2

Merge branch 'simonz/sarplus/spark3' of simonzhaomsgithub:simonzhaoms…

decdb28

…/recommenders into simonz/sarplus/spark3

Update setup.py

7ef200d

Move VERSION as package data file of pysarplus

75028d5

Remove test data access token and move fixtures into conftest.py

8c44d14

Corrent VERSION path

4ec139a

Fix flake8 issues

594674c

miguelgfierro approved these changes Dec 14, 2021

View reviewed changes

anargyri approved these changes Dec 14, 2021

View reviewed changes

anargyri merged commit 6f2088d into recommenders-team:staging Dec 14, 2021

miguelgfierro mentioned this pull request Dec 17, 2021

[FEATURE] Update SARplus build pipeline #1489

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Spark 3.x support for sarplus #1566

Add Spark 3.x support for sarplus #1566

simonzhaoms commented Nov 22, 2021

miguelgfierro left a comment

gramhagen Nov 22, 2021

gramhagen Nov 22, 2021

simonzhaoms Nov 23, 2021

simonzhaoms Dec 13, 2021

gramhagen commented Nov 22, 2021

gramhagen commented Nov 22, 2021

simonzhaoms commented Nov 23, 2021

codecov-commenter commented Nov 23, 2021 •

edited

Loading

miguelgfierro commented Nov 24, 2021

miguelgfierro commented Nov 24, 2021

miguelgfierro commented Nov 29, 2021

laserprec commented Nov 30, 2021

anargyri commented Dec 7, 2021

miguelgfierro left a comment

miguelgfierro left a comment

		@@ -39,3 +39,33 @@ sbt test
		```

		(use ~test and it will automatically check for changes in source files, but not build.sbt)

Add Spark 3.x support for sarplus #1566

Add Spark 3.x support for sarplus #1566

Conversation

simonzhaoms commented Nov 22, 2021

Description

Related Issues

Checklist:

miguelgfierro left a comment

Choose a reason for hiding this comment

gramhagen Nov 22, 2021

Choose a reason for hiding this comment

gramhagen Nov 22, 2021

Choose a reason for hiding this comment

simonzhaoms Nov 23, 2021

Choose a reason for hiding this comment

simonzhaoms Dec 13, 2021

Choose a reason for hiding this comment

gramhagen commented Nov 22, 2021

gramhagen commented Nov 22, 2021

simonzhaoms commented Nov 23, 2021

codecov-commenter commented Nov 23, 2021 • edited Loading

Codecov Report

miguelgfierro commented Nov 24, 2021

miguelgfierro commented Nov 24, 2021

miguelgfierro commented Nov 29, 2021

laserprec commented Nov 30, 2021

anargyri commented Dec 7, 2021

miguelgfierro left a comment

Choose a reason for hiding this comment

miguelgfierro left a comment

Choose a reason for hiding this comment

codecov-commenter commented Nov 23, 2021 •

edited

Loading