Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Spark extension #2162

Merged
merged 3 commits into from
Nov 29, 2022
Merged

Add Spark extension #2162

merged 3 commits into from
Nov 29, 2022

Conversation

xyang16
Copy link
Contributor

@xyang16 xyang16 commented Nov 18, 2022

Description

Brief description of what this PR is about

Add Spark extension

@codecov-commenter
Copy link

codecov-commenter commented Nov 18, 2022

Codecov Report

Base: 72.08% // Head: 71.48% // Decreases project coverage by -0.60% ⚠️

Coverage data is based on head (506b2f5) compared to base (bb5073f).
Patch coverage: 71.75% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2162      +/-   ##
============================================
- Coverage     72.08%   71.48%   -0.61%     
- Complexity     5126     6345    +1219     
============================================
  Files           473      629     +156     
  Lines         21970    28044    +6074     
  Branches       2351     3019     +668     
============================================
+ Hits          15838    20046    +4208     
- Misses         4925     6522    +1597     
- Partials       1207     1476     +269     
Impacted Files Coverage Δ
api/src/main/java/ai/djl/modality/cv/Image.java 69.23% <ø> (-4.11%) ⬇️
...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java 76.00% <ø> (ø)
...rc/main/java/ai/djl/modality/cv/output/Joints.java 71.42% <ø> (ø)
.../main/java/ai/djl/modality/cv/output/Landmark.java 100.00% <ø> (ø)
...main/java/ai/djl/modality/cv/output/Rectangle.java 72.41% <0.00%> (ø)
...i/djl/modality/cv/translator/BigGANTranslator.java 21.42% <0.00%> (-5.24%) ⬇️
.../modality/cv/translator/ImageFeatureExtractor.java 0.00% <0.00%> (ø)
.../ai/djl/modality/cv/translator/YoloTranslator.java 27.77% <0.00%> (+18.95%) ⬆️
...modality/cv/translator/wrapper/FileTranslator.java 44.44% <ø> (ø)
...y/cv/translator/wrapper/InputStreamTranslator.java 44.44% <ø> (ø)
... and 568 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

import java.io.Serializable

/**
* `SparkModel` is the implementation of [[ai.djl.Model]] for Spark support.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does scala doc support markdown?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. This is the scala format of scaladoc

@@ -0,0 +1,16 @@
name := "spark"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason we add build.sbt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

import java.util

/** A [[ai.djl.translate.Translator]] for Spark Image Classification tasks. */
@SerialVersionUID(123456789L)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@SerialVersionUID(123456789L)
@SerialVersionUID(1L)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

Copy link
Contributor

@lanking520 lanking520 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please consolidate versioning in a doc. By the way, could you also create a simple Python wrapper to demo the capability in PySpark?


resolvers += Resolver.jcenterRepo

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason we are using 3.0.1? Do we also want to put this into gradle.properties?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, put into gradle.properties.

<dependency>
<groupId>ai.djl.spark</groupId>
<artifactId>spark</artifactId>
<version>0.19.0</version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0.20.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed

@xyang16 xyang16 merged commit 5b2e760 into deepjavalibrary:master Nov 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants