Record of trying to get arcgis 2.0.0 to work with pangeo

[eeholmes]

@timhaverland-noaa I did some rainy weekend banging on the arcgis image to try to get it work work with xarray and rasterio.

My conclusion Separate images for arcgis might be best for now untill the arcgis widget is upgraded to allow jupyterlab > 2. Current version is 4 and 2 is just so old.

Nonetheless, I did get an image with xarray, rasterio, gdal, and geopandas finally built. It starts up in classic notebook, so you have to manually change to /lab if you want to use that.
ghcr.io/nmfs-opensci/container-images/arcgis-pangeo:xarray
It is on the xarray branch and in the images/arcgis-pangeo folder.

• /images/test

Tried to update the FROM jupyter/base-notebook:lab-2.2.9 image to jupyterhub=2.0.0 since I could not get 1.5.0 to use \lab as start up. Tried everything.

Only thing that worked was updating jupyterhub to 2.0.0 but that meant copying all the start up scripts into the image since the ones with lab-2.2.9 image will not work with jupyterhub=2.0.0

• /image/arcgis-pangeo (branch arcgis-pangeo)
  
  Tried to add arcgis=2.0.0 to pangeo image with Python 3.9. No luck. The

RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager@2 && \
    jupyter labextension install arcgis-map-ipywidget@2.0.0

part needs old ipywidget and nodejs and I just couldn't get it to build.

Also the jupyterhub version is getting downgraded to 1.5, which will likely cause problems. Though tbd since maybe only the version in the base env is important.

• /image/arcgis-pangeo branch main

Tried to add arcgis:2.0.0 to Openscapes image. Same problem as for the pangeo image. The labextension installation is throwing an error

• /image/arcgis-pangeo branch xarray

I also tried to install xarray, rasterio to the image

[eeholmes]

Well this built but not sure how useful it is. I worry the packages are old since the base image is so old.

Link: ghcr.io/nmfs-opensci/container-images/arcgis-pangeo:latest

# ancient version by needed for Python 3.9
FROM jupyter/base-notebook:lab-2.2.9

RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager@2 && \
    jupyter labextension install arcgis-map-ipywidget@2.0.0

# conda is ancient in this old version of base-notebook
RUN conda install -n base conda=22.11.1
RUN conda update -n base conda
# faster solver in the more recent conda releases
RUN conda config --set solver libmamba

RUN conda install -n base --yes -c esri 'arcgis=2.0.0'
RUN conda install -n base --yes -c conda-forge xarray dask netCDF4 bottleneck
RUN conda install -n base --yes -c conda-forge rasterio gdal~=3.6 geopandas

RUN conda clean --all -f -y

COPY apt.txt /tmp/apt.txt

USER root

RUN echo "Installing Apt-get packages..." \
    && apt-get update --fix-missing > /dev/null \
    && apt-get install -y apt-utils wget zip tzdata > /dev/null \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
	
RUN echo "Installing helpful packages..." \
	&& apt-get update --fix-missing > /dev/null \
    # Read apt.txt line by line, and execute apt-get install -y for each line in apt.txt
    && xargs -a /tmp/apt.txt apt-get install -y \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* 

USER ${NB_USER}

RUN fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"

[eeholmes]

Things that didn't work #1 Build off Openscapes image

# Python 3.9 image based off pangeo/base-image:2023.01.13 on May 16, 2023
FROM openscapes/python:f577786

# install additional package...

USER ${NB_USER}

# conflict with juyterlab=2.2.9
RUN conda remove jupyterlab-link-share --name notebook 

# make sure we install in the notebook environment which is default in the
## This would not solve
# Openscapes image
COPY environment.yml arcgis-environment.yml
RUN conda env update --name notebook -f arcgis-environment.yml
RUN rm arcgis-environment.yml

SHELL ["/bin/bash", "-c"]
RUN source activate notebook \
  && jupyter labextension install @jupyter-widgets/jupyterlab-manager@2 \
  && jupyter labextension install arcgis-map-ipywidget@2.0.0

environment.yml

name: arcgis
channels:
  - conda-forge
  - esri
dependencies:
  - jupyterlab=2.2.9
  - jupyterhub=2.0.0
#  - conda=22.9.0
  - esri::arcgis=2.0.0
  - ipywidgets
  - nodejs

[eeholmes]

Things that did not work #2 . Build off pangeo image so try to install less stuff

FROM pangeo/base-image:2023.01.13
# pango base will look for environment.yml and apt.txt and install

# activate notebook env that pangeo base creates
RUN conda init zsh && conda init bash

USER ${NB_USER}

# broken here
RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager@2 && \
    jupyter labextension install arcgis-map-ipywidget@2.0.0


WORKDIR ${HOME}

environment.yml

name: arcgis-pangeo
channels:
  - conda-forge
dependencies:
  - python=3.9
  - pangeo-notebook
#  - esri::arcgis=2.0.0
## Utilities
  - pip
  - nodejs # for installing jupyterlab-widgets
  - ipywidgets
##
  - jupyterlab=2.2.9
  - jupyterlab-git
  - jupyter-resource-usage

[eeholmes]

Absolutely horrific attempt to upgrade the arcgis image built off jupyter/basenotebook:lab-2.2.9 to jupyterhub>1 so that the image starts in \lab by default. Technically, you should be able to set the config so that it does, but I could not get any settings to work. Tried everything and then I saw that others had the same problem. I had to upgrade to jupyterhub>2. This works but awful awful awful because all the start scripts that were in the original image had to be replaced. Some of this is not needed. But new start-singleuser.sh needed since the one in the image is not compatible with jupyterhub>1.

FROM jupyter/base-notebook:lab-2.2.9
#ENV JUPYTERHUB_SINGLEUSER_APP="jupyter_server.serverapp.ServerApp"

USER ${NB_UID}

RUN conda install --yes 'jupyterhub=2.0.0' && \
    conda clean --all -f -y

RUN conda install --yes -c esri 'arcgis=2.0.0' && \
    conda clean --all -f -y

#RUN conda install --yes jupyterlab-git jupytext && \
#    conda clean --all -f -y

RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager@2 && \
    jupyter labextension install arcgis-map-ipywidget@2.0.0

# I have to use this old base-notebook image (trust me), but it has a
# start-notebook.py that doesn't include "lab"
    
# Configure container startup
ENTRYPOINT ["tini", "-g", "--"]
CMD ["start-notebook.sh"]

# Copy local files as late as possible to avoid cache busting
COPY start.sh start-notebook.sh start-singleuser.sh /usr/local/bin/
# Currently need to have both jupyter_notebook_config and jupyter_server_config to support classic and lab
COPY jupyter_notebook_config.py jupyter_server_config.py /etc/jupyter/
COPY start-notebook.d /usr/local/bin/start-notebook.d

# Fix permissions on /etc/jupyter as root
USER root

RUN mkdir /etc/jupyterhub
COPY jupyterhub_config.py /etc/jupyterhub/
RUN fix-permissions /etc/jupyterhub/

RUN fix-permissions /etc/jupyter/

# Switch back to jovyan to avoid accidental container runs as root
USER $NB_UID

WORKDIR $HOME

start-singleuser.sh

#!/bin/bash
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

set -e

# set default ip to 0.0.0.0
if [[ "$NOTEBOOK_ARGS $@" != *"--ip="* ]]; then
    NOTEBOOK_ARGS="--ip=0.0.0.0 $NOTEBOOK_ARGS"
fi

# handle some deprecated environment variables
# from DockerSpawner < 0.8.
# These won't be passed from DockerSpawner 0.9,
# so avoid specifying --arg=empty-string
if [ ! -z "$NOTEBOOK_DIR" ]; then
    NOTEBOOK_ARGS="--notebook-dir='$NOTEBOOK_DIR' $NOTEBOOK_ARGS"
fi
if [ ! -z "$JPY_PORT" ]; then
    NOTEBOOK_ARGS="--port=$JPY_PORT $NOTEBOOK_ARGS"
fi
if [ ! -z "$JPY_USER" ]; then
    NOTEBOOK_ARGS="--user=$JPY_USER $NOTEBOOK_ARGS"
fi
if [ ! -z "$JPY_COOKIE_NAME" ]; then
    NOTEBOOK_ARGS="--cookie-name=$JPY_COOKIE_NAME $NOTEBOOK_ARGS"
fi
if [ ! -z "$JPY_BASE_URL" ]; then
    NOTEBOOK_ARGS="--base-url=$JPY_BASE_URL $NOTEBOOK_ARGS"
fi
if [ ! -z "$JPY_HUB_PREFIX" ]; then
    NOTEBOOK_ARGS="--hub-prefix=$JPY_HUB_PREFIX $NOTEBOOK_ARGS"
fi
if [ ! -z "$JPY_HUB_API_URL" ]; then
    NOTEBOOK_ARGS="--hub-api-url=$JPY_HUB_API_URL $NOTEBOOK_ARGS"
fi
NOTEBOOK_BIN="jupyterhub-singleuser"

. /usr/local/bin/start.sh $NOTEBOOK_BIN $NOTEBOOK_ARGS"$@"

jupyter_server_config.py. I don't think the \lab settings do anything. Key is to go to jupyterhub>1 so \lab is default.

import subprocess
import os
import errno
import stat

c = get_config()  # noqa: F821
c.ServerApp.ip = '0.0.0.0'
c.ServerApp.port = 8888
c.ServerApp.open_browser = False
c.Spawner.cmd = ["jupyter-labhub"]
c.Spawner.default_url = '/lab'
c.Spawner.defaultUrl = '/lab'

# https://github.com/jupyter/notebook/issues/3130
c.FileContentsManager.delete_to_trash = False

# Generate a self-signed certificate
if 'GEN_CERT' in os.environ:
    dir_name = jupyter_data_dir()
    pem_file = os.path.join(dir_name, 'notebook.pem')
    try:
        os.makedirs(dir_name)
    except OSError as exc:  # Python >2.5
        if exc.errno == errno.EEXIST and os.path.isdir(dir_name):
            pass
        else:
            raise

    # Generate an openssl.cnf file to set the distinguished name
    cnf_file = os.path.join(os.getenv('CONDA_DIR', '/usr/lib'), 'ssl', 'openssl.cnf')
    if not os.path.isfile(cnf_file):
        with open(cnf_file, 'w') as fh:
            fh.write('''\
[req]
distinguished_name = req_distinguished_name
[req_distinguished_name]
''')

    # Generate a certificate if one doesn't exist on disk
    subprocess.check_call(['openssl', 'req', '-new',
                           '-newkey', 'rsa:2048',
                           '-days', '365',
                           '-nodes', '-x509',
                           '-subj', '/C=XX/ST=XX/L=XX/O=generated/CN=generated',
                           '-keyout', pem_file,
                           '-out', pem_file])
    # Restrict access to the file
    os.chmod(pem_file, stat.S_IRUSR | stat.S_IWUSR)
    c.ServerApp.certfile = pem_file

# Change default umask for all subprocesses of the notebook server if set in
# the environment
if 'NB_UMASK' in os.environ:
    os.umask(int(os.environ['NB_UMASK'], 8))

start-notebook.d

#!/bin/bash
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

set -e

wrapper=""
if [[ "${RESTARTABLE}" == "yes" ]]; then
    wrapper="run-one-constantly"
fi

if [[ ! -z "${JUPYTERHUB_API_TOKEN}" ]]; then
    # launched by JupyterHub, use single-user entrypoint
    exec /usr/local/bin/start-singleuser.sh "$@"
elif [[ ! -z "${JUPYTER_ENABLE_LAB}" ]]; then
    . /usr/local/bin/start.sh $wrapper jupyter lab "$@"
else
    echo "WARN: Jupyter Notebook deprecation notice https://github.com/jupyter/docker-stacks#jupyter-notebook-deprecation-notice."
    . /usr/local/bin/start.sh $wrapper jupyter lab "$@"
fi

[timhaverland-noaa]

Hi Eli thanks for all these efforts and sorry it's taken me so long to review. In my testing I found that the lynchpin for building the labextensions is openssl 1.1.1. That's why I had to explicitly install nodejs 18.16.0 ha637b67_1. If I went to nodejs 18.17 openssl would jump to 3.x and the labextension builds would fail.