Skip to content

Latest commit

 

History

History
278 lines (210 loc) · 11.2 KB

README.md

File metadata and controls

278 lines (210 loc) · 11.2 KB
Raw

MMLSpark Microsoft Machine Learning for Apache Spark

MMLSpark provides a number of deep learning and data science tools for Apache Spark, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK) and OpenCV, enabling you to quickly create powerful, highly-scalable predictive and analytical models for large image and text datasets.

MMLSpark requires Scala 2.11, Spark 2.1+, and either Python 2.7 or Python 3.5+. See the API documentation for Scala and for PySpark.

Table of Contents

Notable features

  • Create an deep image classifier with transfer learning (example:305)
  • Fit a LightGBM classification or regression model on a biochemical dataset (example:106), to learn more check out the LightGBM documentation page.
  • Deploy a deep network as a distributed web service with MMLSpark Serving
  • Use web services in Spark with HTTP on Apache Spark
  • Train a deep image classifier on Azure N-Series GPU VMs (example:401)
  • Use Bi-directional LSTMs from Keras for medical entity extraction (example:304)
  • Create a text analytics system on Amazon book reviews (example:201)
  • Perform distributed hyperparameter tuning to identify Breast Cancer (example:203)
  • Easily ingest images from HDFS into Spark DataFrame (example:301)
  • Use OpenCV on Spark to manipulate images (example:302)
  • Train classification and regression models easily via implicit featurization of data (example:101)
  • Train and evaluate a flight delay prediction system (example:102)

See our notebooks for all examples.

A short example

Below is an excerpt from a simple example of using a pre-trained CNN to classify images in the CIFAR-10 dataset. View the whole source code as an example notebook.

...
import mmlspark
# Initialize CNTKModel and define input and output columns
cntkModel = mmlspark.CNTKModel() \
                    .setInputCol("images").setOutputCol("output") \
                    .setModelLocation(modelFile)
# Train on dataset with internal spark pipeline
scoredImages = cntkModel.transform(imagesWithLabels)
...

See other sample notebooks as well as the MMLSpark documentation for Scala and PySpark.

Setup and installation

Docker

The easiest way to evaluate MMLSpark is via our pre-built Docker container. To do so, run the following command:

docker run -it -p 8888:8888 -e ACCEPT_EULA=yes microsoft/mmlspark

Navigate to http://localhost:8888/ in your web browser to run the sample notebooks. See the documentation for more on Docker use.

To read the EULA for using the docker image, run
docker run -it -p 8888:8888 microsoft/mmlspark eula

GPU VM Setup

MMLSpark can be used to train deep learning models on GPU nodes from a Spark application. See the instructions for setting up an Azure GPU VM.

Spark package

MMLSpark can be conveniently installed on existing Spark clusters via the --packages option, examples:

spark-shell --packages Azure:mmlspark:0.13
pyspark --packages Azure:mmlspark:0.13
spark-submit --packages Azure:mmlspark:0.13 MyApp.jar

This can be used in other Spark contexts too, for example, you can use MMLSpark in AZTK by adding it to the .aztk/spark-defaults.conf file.

Python

To try out MMLSpark on a Python (or Conda) installation you can get Spark installed via pip with pip install pyspark. You can then use pyspark as in the above example, or from python:

import pyspark
spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
            .config("spark.jars.packages", "Azure:mmlspark:0.13") \
            .getOrCreate()
import mmlspark

HDInsight

To install MMLSpark on an existing HDInsight Spark Cluster, you can execute a script action on the cluster head and worker nodes. For instructions on running script actions, see this guide.

The script action url is: https://mmlspark.azureedge.net/buildartifacts/0.13/install-mmlspark.sh.

If you're using the Azure Portal to run the script action, go to Script actions → Submit new in the Overview section of your cluster blade. In the Bash script URI field, input the script action URL provided above. Mark the rest of the options as shown on the screenshot to the right.

Submit, and the cluster should finish configuring within 10 minutes or so.

Databricks cloud

To install MMLSpark on the Databricks cloud, create a new library from Maven coordinates in your workspace.

For the coordinates use: Azure:mmlspark:0.13. Ensure this library is attached to all clusters you create.

Finally, ensure that your Spark cluster has at least Spark 2.1 and Scala 2.11.

You can use MMLSpark in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:

https://mmlspark.blob.core.windows.net/dbcs/MMLSpark%20Examples%20v0.13.dbc

SBT

If you are building a Spark application in Scala, add the following lines to your build.sbt:

resolvers += "MMLSpark Repo" at "https://mmlspark.azureedge.net/maven"
libraryDependencies += "com.microsoft.ml.spark" %% "mmlspark" % "0.13"

Building from source

You can also easily create your own build by cloning this repo and use the main build script: ./runme. Run it once to install the needed dependencies, and again to do a build. See this guide for more information.

R (Beta)

To try out MMLSpark using the R autogenerated wrappers see our instructions. Note: This feature is still under development and some necessary custom wrappers may be missing.

Learn More

Contributing & feedback

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

See CONTRIBUTING.md for contribution guidelines.

To give feedback and/or report an issue, open a GitHub Issue.

Other relevant projects

Apache®, Apache Spark, and Spark® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.