Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SEDONA-285] Sedona Spark Jupyterlab docker image #939

Merged
merged 52 commits into from
Aug 2, 2023
Merged

[SEDONA-285] Sedona Spark Jupyterlab docker image #939

merged 52 commits into from
Aug 2, 2023

Conversation

jiayuasu
Copy link
Member

@jiayuasu jiayuasu commented Aug 2, 2023 •
edited
Loading

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

  1. Add Sedona Spark Jupyter lab image
    • Cross platform compilation on amd64 and arm64. For this purpose, only 1 dockerfile is allowed.
    • Build from official Sedona releases and Sedona master branch
    • Sedona in this container runs in a real cluster mode rather than pseudo mode.
    • JDK is explicitly set to Java 19 to avoid situations that the latest Sedona jar is compiled by Java > 11.
  2. Add docs for this

TODO

  1. Currently, the images were pushed to https://hub.docker.com/r/drjiayu/sedona-jupyterlab . I am working on requesting a ASF docker seat and then the images will be pushed to an official Sedona DockerHub repo.
  2. Add sedona spark zeppelin docker in 2nd PR
  3. Add sedona flink zeppelin docker in 3rd PR

How was this patch tested?

  1. Locally tested
  2. Pushed to https://hub.docker.com/r/drjiayu/sedona-jupyterlab

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation update.

Hadiya Kartikey and others added 30 commits May 26, 2023 22:58
Added docker images for spark-sedona:1.3.1-zeppelin and spark-sedona1…
9aa8914 
….4.0-jupyterlab
@kartikeyhadiya
Merge branch 'apache:master' into main
0270034
Changes in jupyterlab dockerfile: 1) Removed duplicate data and noteb…
0843750 
…ooks, instead added binder folder as volume in docker-compose file.
1) Modified docker files to download jars on the fly. 2) Added Apache…
b5a6f61 
… License as file header to .dockerfiles and .sh script
@kartikeyhadiya
Merge branch 'apache:master' into main
5c595ce
@kartikeyhadiya
Merge branch 'apache:master' into main
c609f4f
@kartikeyhadiya
Restructed docker files for spark to include only single copy for zep…
531c49a 
…pelin and jupyterlab. By default spark 3.3.2 version used but can be changed with --build-args. Added workflow for pushing image to dockerhub
@kartikeyhadiya
Update dockerhub-deploy.yml
be63322 
Updated dokcerhub-deploy.yml workflow
@yyy1000
Merge branch 'master' of https://github.com/yyy1000/sedona into docke…
97ac895 
…r-img
@yyy1000
update sedona version
a055e81
@yyy1000
update sedona version
6107193
@yyy1000
update sedona version
d3094e2
@yyy1000
add bash script to accelarate
a76104d
@yyy1000
feat: add keplergl in Docker build
acf3e83
@yyy1000
feat: update pyspark dependency from Pipfile
41e74fa
@yyy1000
feat: remove unused command
fdb04b3
@yyy1000
remove install packages
b71ea78
@yyy1000
add bask-jdk in docker.sh
bdaae12
@yyy1000
feat: add curl install
70eceed
@yyy1000
feat: add update
d704e93
@yyy1000
feat: spark-py
3d3d167
@yyy1000
feat: using python as base
bdd14b6
@yyy1000
feat: using new base-jdk
ce96110
@yyy1000
feat: add curl
c0adfdd
@yyy1000
feat: change pyspark jars
3f285b5
@yyy1000
fix: nit
8b803e0
@yyy1000
feat: try using spark-base as base
e0e2710
@yyy1000
fix: degrade to java8
35c91b6
@yyy1000
feat: change sedona jars
42b766b
@yyy1000
fix: nit
f73cb33
@jiayuasu jiayuasu mentioned this pull request Aug 2, 2023
Closed
@jiayuasu jiayuasu added this to the sedona-1.5.0 milestone Aug 2, 2023
@jiayuasu jiayuasu merged commit bb87200 into apache:master Aug 2, 2023
41 checks passed
@jiayuasu jiayuasu deleted the build-sedona-jupyter-docker branch August 2, 2023 18:53
@jiayuasu jiayuasu mentioned this pull request Aug 11, 2023
Merged
Kontinuation pushed a commit to Kontinuation/sedona that referenced this pull request Oct 11, 2024
@jiayuasu @kartikeyhadiya
@yyy1000 @Kontinuation
[SEDONA-285] Sedona Spark Jupyterlab docker image (apache#939)
6334510 
Co-authored-by: Hadiya Kartikey <kartikey.hadiya@esri.in>
Co-authored-by: Kartikey <56991178+kartikeyhadiya@users.noreply.github.com>
Co-authored-by: Kartikey <hadiyakartikey123@gmail.com>
Co-authored-by: yyy1000 <992364620@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
affect public APIs improvement sedona-docker
Projects
None yet
Milestone
sedona-1.5.0
Development

Successfully merging this pull request may close these issues.
3 participants
@jiayuasu @kartikeyhadiya @yyy1000