- Blackbalsam is an open source data science environment with an initial focus on COVID-19 and North Carolina.
- Architecture
- About
- Next
Data Science is impeded by operatoinal obstacles we can address. The discipline draws on numerous skill set but not everyone has all of the skills, and not everyone has those skills in the same degree. Once someone has created an interesting result, asking a collaborator to laboriously reproduce the steps the first researcher took is lost time. Similarly, each participant in a team needing to acquire the same data set is a source of error and a waste of time. In view of these realities, ready availability of tools and data in a common digital environment accomplishes a number of aims simultaneously:
- Digital Lab: It creates a digital laboratory so that each individual is not posed with the obstacle of assembling all the required computational and data instruments themselves from scratch.
- Sharing: Living in the cloud has acclimatized many of us to instant messaging a URL to a coworker many times a day. In prior times, email and installing software locally was the norm. When one person creates an interesting analysis, it should be possible to message a collaborator with a URL to that analysis.
- Scale: Setting up an analysis on one's laptop is very good but training machine learning models and analyzing large data sets increasingly requires computational scale, access to specialized hardware and accelerators, and the modern tools to access that scale.
Blackbalsam was developed to assemble a pragmatic digital data science laboratory to meet those needs.
Blackbalsam's interface uses a JupyterHub notebook environment featuring artificial intelligence, visualization, and scalable computing capabilities. For computation, integration of the Jupyter environment with Apache Spark and Kubernetes allows users to dynamically create personal Spark clusters with user specified attributes. Plans to incorporate GPU nodes to enable deep learning scenarios are under way. These interface and compute capabilities are coupled to a tiered storage platform including networked filesystem, the Mino S3 compatible object store, and the Alluxio distributed memory cache. These provide access to COVID-19 data sets. The prototype runs at the Renaissance Computing Institute in an on premise cluster, is cloud ready, and is open source under the MIT License.
Authentication is provided via GitHub and OpenID Connect (OIDC). Whitelisted users can use their GitHub identity to login and start working immediately.
The Blackbalsam notebook includes Tensorflow, Keras, Gensim, PyTorch, scikit-learn, pandas, and numpy. Users can also easily create Spark clusters providing access to Spark's MLlib machine learning toolkit.
JupyterHub provides the interface to the environment presenting a notebook providing Python and R kernels.
The Blackbalsam notebook includes matplotlib, plotly, and seaborn. It also includes bokeh, yellowbrick, and ipyleaflet to handle more specialized needs including machine learning and geospatial visualization. This figure shows a Leaflet component in Jupyter:
And here's a county level US map using Pandas and Plotly:
The Blackbalsam notebook is instrumented to allow dynamically launching a customized, personal Apache Spark cluster of a user specified topology through the Kubernetes API. In this figure, we see the notebook for loading the Python interface to Blackbalsam and creating a four worker Spark cluster. After creating the cluster, it uses Spark's resilient distributed dataset (RDD) interface and its functional programming paradigm to apply a functional operator to each loaded article.
The mechanics of configuring and launching the cluster are handled transparently to the user. Exiting the notebook kernel deallocates the cluster. This figure shows the four 1GB Spark workers created by the previous steps.
Next, we create a Word2Vec word embedding using the provided Spark machine learning libraries:
The network filesystem (NFS) is used to mount shared storage to each user notebook at /home/shared.
The Minio object store provides an S3 compatible interface .
Minio supports distributed deployment scenarios which make it horizontally scalable. Minio also facilitates loading large objects into Apache Spark
Alluxio is a distributed memory cache interposed between multiple "under-filesystems" like NFS and analytic tools like Spark and its machine learning toolkit. It stores data in node memory, not only accelerating access but allowing failed workflows to restart and other interesting scenarios. It also supports using the Minio S3 object store as an under filesystem. Since Alluxio also supports an ACL based access control model, this creates some interesting possibilities for us to explore with regard to data sharing.
The first Blackbalsam instance is the RENCI COVID-19 platform. See the descriptionof the data corpus for information on how to request new data.
A Kubernetes periodic task exectues the data repository update script hourly.
Blackbalsam's design composes widely used open source systems including Docker, Kubernetes, Apache Spark, Jupyter and JupyterHub into a coherent environment.
- Kubernetes v1.17.4
- kubectl >=v1.17.4
- Python 3.7.x
- The Linux envsubst command
- Helm 2 (a JupyterHub dependency)
Create a GitHub OAuth app
Create a file in your home directory called .blackbalsam with contents like these:
jupyterhub_secret_token=<jupyter-hub-secret-token>
jupyterhub_baseUrl=/blackbalsam/
public_ip=<public-ip-address>
github_client_id=<github-client-id>
github_client_secret=<client-server-id>
github_oauth_callback=http://<your-domain-name>/blackbalsam/oauth_callback
minio_access_key=<minio-access-key>
minio_secret_key=<minio-secret-key>
Ensure you have kubectl configured to point to a Kubernetes cluster.
Clone the repository. Create a virtual environment, populate the environment, and run the installer.
git clone git@github.com:stevencox/blackbalsam.git
python3 -m venv blackbbalsam-env
source blackbalsam-env/bin/activate
cd blackbalsam
bin/blackbalsam up
After a substantial pause, you should see output like this:
NOTES:
Thank you for installing JupyterHub!
Your release is named blackbalsam and installed into the namespace blackbalsam.
You can find if the hub and proxy is ready by doing:
kubectl --namespace=blackbalsam get pod
and watching for both those pods to be in status 'Ready'.
You can find the public IP of the JupyterHub by doing:
kubectl --namespace=blackbalsam get svc proxy-public
It might take a few minutes for it to appear!
Note that this is still an alpha release! If you have questions, feel free to
1. Read the guide at https://z2jh.jupyter.org
2. Chat with us at https://gitter.im/jupyterhub/jupyterhub
3. File issues at https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues
Then, go to https://{your-domain}/blackbalsam/ to visit the application.
For information on the command line management interface, see the help option:
$ bin/blackbalsam data help
bin/blackbalsam is the command line interface for a Blackbalsam data science cluster.
Each of these commands include up, down, status, and restart.
eg: bin/blackbalsam <command> [ up | down | status | restart ]
User Experience Services:
hub Manage the JupyterHub and associated proxy mappings, storage, etc.
Storage Services:
alluxio Manage Alluxio services and distributed workers.
minio Manage Minio S3 interface. Creates a service called 'minio'.
Proxy Services:
proxy Manage the programmable Ambassador edge proxy.
Data Services:
data Install the periodic data update task for this cluster.
data run Run the periodic data update task now.
data stop Stop the running data update task if one exists
Secrets:
secret Manage secrets based on environment conifguration.
General:
up Execute configurations and start all services.
down Stop all cluster services.
restart Stop and start all system components.
status Report on Kubernetes components and system services in detail.
nodes Display detailed usage and status for cluster nodes.
help Show this message.
From Wikipedia:
"Black Balsam Knob,[2] also known as Black Balsam Bald, is in the Pisgah National Forest southwest of Asheville, North Carolina, near milepost 420 on the Blue Ridge Parkway. It is the second highest mountain[3] in the Great Balsam Mountains. The Great Balsams are within the Blue Ridge Mountains, which are part of the Appalachian Mountains. It is the 23rd highest of the 40 mountains in North Carolina over 6000 feet.[4]"
Blackbalsam's authors are Steve Cox and PJ Linebaugh.
- AI & ML:
- Debug and fix the JupyterHub on Kubernetes bug preventing the use of multiple notebook profiles.
- Create a separate AI notebook and configure GPU targeting.
- Purchase and integrate GPU hardware.
- Persistence:
- Configure and test Alluxio to Minio interfaces
- Configure and test Alluxio to NFS interfaces
- Configure and test Spark Alluxio interface
- Tools:
- Create and document notebooks with examples of key features
- Track user demand to prioritize and incorporate new capabilities.
- Infrastructure:
- Get trusted certificates for the site and a development instance.
- Establish a blackbalsam-dev namespace for development testing.
- Crete an automated build and test pipeline.
- Purchase and deploy new hardware.
- Move management tools to Helm 3.
