[ENH] Updating resen-base and resen-core #61

asreimer · 2020-09-11T05:36:56Z

Accomplishes all checklist items in #59

Updated resen-base to ubuntu 20.04 and updated start scripts to track the latest jupyter-stacks versions.
upgrade to python 3.8
deprecate python 2.7
install seaborn, sunpy, pyephem, and skyfield,
replace custom npm install with apt-get install

Testing

Build the resen-base image:

$ cd resen-base
$ docker build -t resenbasetest

Bulid the resen-core image:

$ cd resen-core
$ docker build -t resencoretest

Create a container with resencoretest and test things (e.g.):

$ docker run -it --name test <sha256:imageid> /bin/bash

… latest jupyter-stacks.

asreimer · 2020-09-11T05:44:36Z

resen-core/Dockerfile

@@ -1,4 +1,5 @@
-FROM earthcubeingeo/resen-base:2020.1.0
+#FROM earthcubeingeo/resen-base:2020.1.0
+FROM resenbasetest:latest


This needs to be changed to resen-base:2020.2.0 once we are happy with resen-base!

asreimer

Checked all files. Only one comment.

asreimer · 2020-09-14T14:34:54Z

There's a small issue with the terminal in jupyterlab not starting up with bash. Should be able to be solved with a jupyterlab_config.py file.

ljlamarche · 2020-09-15T21:14:23Z

I'm having issues building the resen-core image - looks like jupyterlab is failing.

Step 33/37 : RUN /bin/bash -cl 'source /home/$NB_USER/envs/py38/bin/activate &&                    jupyter labextension install @jupyter-widgets/jupyterlab-manager &&                    jupyter labextension install jupyter-matplotlib'
 ---> Running in 3803aedb71cc
Building jupyterlab assets (build:prod:minimize)
An error occured.
RuntimeError: JupyterLab failed to build
See the log file for details:  /tmp/jupyterlab-debug-egv3ho2l.log
The command '/bin/sh -c /bin/bash -cl 'source /home/$NB_USER/envs/py38/bin/activate &&                    jupyter labextension install @jupyter-widgets/jupyterlab-manager &&                    jupyter labextension install jupyter-matplotlib'' returned a non-zero code: 1

This issue persists if I try running the commands from within the docker container.

Start a docker container from the last image constructed before the docker build failed:

$ docker run -it --name resen-test <image id> /bin/bash

Attempt to run the two jupyterlab commands:

$ jupyter labextension install @jupyter-widgets/jupyterlab-manager

$ jupyter labextension install jupyter-matplotlib

Both these commands produce the error

Building jupyterlab assets (build:prod:minimize)
An error occured.
RuntimeError: JupyterLab failed to build

End of log file:

Error: Command failed: /home/jovyan/envs/py38/share/jupyter/lab/staging/node_modules/.bin/webpack --config webpack.prod.minimize.config.js
    at checkExecSyncError (child_process.js:629:11)
    at Object.execFileSync (child_process.js:647:13)
    at Object.<anonymous> (/home/jovyan/envs/py38/share/jupyter/lab/staging/node_modules/@jupyterlab/buildutils/lib/ensure-max-old-space.js:38:17)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

JupyterLab failed to build
Traceback (most recent call last):

  File "/home/jovyan/envs/py38/lib/python3.8/site-packages/jupyterlab/debuglog.py", line 47, in debug_logging
    yield

  File "/home/jovyan/envs/py38/lib/python3.8/site-packages/jupyterlab/labextensions.py", line 104, in start
    build(clean_staging=self.should_clean,

  File "/home/jovyan/envs/py38/lib/python3.8/site-packages/jupyterlab/commands.py", line 459, in build
    return handler.build(name=name, version=version, static_url=static_url,

  File "/home/jovyan/envs/py38/lib/python3.8/site-packages/jupyterlab/commands.py", line 661, in build
    raise RuntimeError(msg)

RuntimeError: JupyterLab failed to build

Exiting application: jupyter

edit by @asreimer: here's the full log file:
jupyterlab-debug-bwy28z9x.log

asreimer · 2020-09-16T04:05:07Z

I cannot reproduce on my fedora VM (has 9GB of RAM available).

I think the issue might be related to memory usage during minification: facebook/create-react-app#8320 (comment) (edit: Yes, 100% this must be our problem, see here: webpack-contrib/terser-webpack-plugin#202 (comment))

If you increase the amount of memory that the container is allowed to use while building, does this problem go away?

pmreyes2 · 2020-09-16T06:30:28Z

I cannot reproduce the error either. My system is similar to @ljlamarche : MacBook Pro 16 GB RAM, and my docker is allowed to use up to 6 CPUs, 4 GB memory, 1GB swap, and disk image size is 60 GB (34 GB used)

ljlamarche · 2020-09-16T15:36:46Z

Increasing the CPUs, memory, and swap to @pmreyes2 configuration fixed the problem. Thanks!

valentic · 2020-09-16T17:38:44Z

It's scary to think about the amount of system resources that we can through at things nowadays...

…

On 9/15/20 11:30 PM, Pablo M. Reyes wrote: I cannot reproduce the error either. My system is similar to @ljlamarche <https://github.com/ljlamarche> : MacBook Pro 16 GB RAM, and my docker is allowed to use up to 6 CPUs, 4 GB memory, 1GB swap, and disk image size is 60 GB (34 GB used) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMC5ZVKD3UTCNODNDPTPUTSGBLRFANCNFSM4RGVDDVQ>.

resen-core/Dockerfile

ljlamarche · 2020-09-16T18:18:16Z

Tried this out, everything looks good to me! We'll have to update some of the tutorials, but I can do that on a separate branch.

resen-core/resources/helpers/setup_basemap.sh

ljlamarche · 2020-09-18T15:28:07Z

We're running into this issue: timduly4/pyglow#128
Basically, pyglow will crash the kernel if you give IRI 2016 a date after 2010, but IRI 2012 still works fine. It looks like it's a documented pyglow bug though so I'm not sure there's much we can/should do about it.

…sion.

asreimer · 2020-10-16T05:27:53Z

I bumped the version of basemap and have fixed the bug with the jupyterlab terminal not using bash.

I think we're good to push the new resen-base to dockerhub and merge this into develop. Agree?

…that fixes an iri2016 issue for dates after year 2010.

ljlamarche · 2020-10-19T18:56:41Z

I agree that we're ready to push the new resen-base to dockerhub. Do we want to include sciunit2 in this version of resen-core? (Can discuss this Thursday if anyone has strong opinions.)

updating pip to 20.2.4 (2020-10-16)

pmreyes2 · 2020-10-19T23:41:33Z

After two very small changes: adding pyglow latest commit and recently released sciunit2 for python3 I agree to merge this branch into develop. And yes, resen-base should be pushed to dockerhub.

asreimer · 2020-10-20T04:52:24Z

@pmreyes2 You accidentally added a bunch of python2.7 scripts back into this PR, which specifically removed them.

Please check your commit a81f824 and clean things up before merging.

ljlamarche · 2020-10-20T15:45:25Z

In order for bokeh to work, we need to install the following jupyterlab extensions:

jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install @bokeh/jupyter_bokeh

https://docs.bokeh.org/en/latest/docs/user_guide/jupyter.html

pmreyes2 · 2020-10-20T17:26:29Z

@ljlamarche I just added widgets interaction for bokeh. The jupyterlab-manager was already installed for interactions with matplotlib, so just the bokeh interactions was added. I think we can merge this to develop branch to keep fine tuning there.

ljlamarche · 2020-10-20T23:05:04Z

Just tested the bokeh widget. I have no further concerns about merging this.

Updated resen-base to ubuntu 20.04 and updated start scripts to track…

5d5c363

… latest jupyter-stacks.

asreimer added the enhancement New feature or request label Sep 11, 2020

asreimer added 3 commits September 10, 2020 22:38

Updated resen-core: removed py27, install new packages

49de0af

remove py27 scripts

069d595

update python env to py38

35417ef

asreimer commented Sep 11, 2020

View reviewed changes

This was referenced Sep 11, 2020

Release candidate for v2020.2.0 EarthCubeInGeo/resen#70

Merged

[ENH]: Next release TODOs #59

Closed

ljlamarche reviewed Sep 16, 2020

View reviewed changes

resen-core/Dockerfile Outdated Show resolved Hide resolved

ljlamarche reviewed Sep 18, 2020

View reviewed changes

resen-core/resources/helpers/setup_basemap.sh Outdated Show resolved Hide resolved

Fixed bug with jupyterlab terminado not using bash. Fixed basemap ver…

b70e14f

…sion.

ljlamarche mentioned this pull request Oct 19, 2020

Tutorials v2020.2.0 #62

Merged

Updating the latest commit(19 Oct 2020) of pyglow in setup_pyglow.sh …

a81f824

…that fixes an iri2016 issue for dates after year 2010.

Adding sciunit2 to setup_py38_env.sh

86eb1bf

updating pip to 20.2.4 (2020-10-16)

Removing files accidentally reintroduced in the two previous commits

72f5a2d

Adding jupyter-widget interaction with bokeh

77bf1a0

asreimer merged commit e18ab38 into develop Oct 29, 2020

asreimer deleted the enh_ubuntu20.04 branch October 29, 2020 21:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH] Updating resen-base and resen-core #61

[ENH] Updating resen-base and resen-core #61

asreimer commented Sep 11, 2020

asreimer Sep 11, 2020

asreimer left a comment

asreimer commented Sep 14, 2020

ljlamarche commented Sep 15, 2020 •

edited by asreimer

Loading

asreimer commented Sep 16, 2020 •

edited

Loading

pmreyes2 commented Sep 16, 2020

ljlamarche commented Sep 16, 2020

valentic commented Sep 16, 2020 via email

ljlamarche commented Sep 16, 2020

ljlamarche commented Sep 18, 2020

asreimer commented Oct 16, 2020 •

edited

Loading

ljlamarche commented Oct 19, 2020

pmreyes2 commented Oct 19, 2020 •

edited

Loading

asreimer commented Oct 20, 2020

ljlamarche commented Oct 20, 2020

pmreyes2 commented Oct 20, 2020 •

edited

Loading

ljlamarche commented Oct 20, 2020

[ENH] Updating resen-base and resen-core #61

[ENH] Updating resen-base and resen-core #61

Conversation

asreimer commented Sep 11, 2020

Testing

asreimer Sep 11, 2020

Choose a reason for hiding this comment

asreimer left a comment

Choose a reason for hiding this comment

asreimer commented Sep 14, 2020

ljlamarche commented Sep 15, 2020 • edited by asreimer Loading

asreimer commented Sep 16, 2020 • edited Loading

pmreyes2 commented Sep 16, 2020

ljlamarche commented Sep 16, 2020

valentic commented Sep 16, 2020 via email

ljlamarche commented Sep 16, 2020

ljlamarche commented Sep 18, 2020

asreimer commented Oct 16, 2020 • edited Loading

ljlamarche commented Oct 19, 2020

pmreyes2 commented Oct 19, 2020 • edited Loading

asreimer commented Oct 20, 2020

ljlamarche commented Oct 20, 2020

pmreyes2 commented Oct 20, 2020 • edited Loading

ljlamarche commented Oct 20, 2020

ljlamarche commented Sep 15, 2020 •

edited by asreimer

Loading

asreimer commented Sep 16, 2020 •

edited

Loading

asreimer commented Oct 16, 2020 •

edited

Loading

pmreyes2 commented Oct 19, 2020 •

edited

Loading

pmreyes2 commented Oct 20, 2020 •

edited

Loading