MMLSpark provides a number of deep learning and data science tools for Apache Spark, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK) and OpenCV, enabling you to quickly create powerful, highly-scalable predictive and analytical models for large image and text datasets.

MMLSpark requires Scala 2.11, Spark 2.1+, and either Python 2.7 or Python 3.5+. See the API documentation for Scala and for PySpark.

Salient features

  • Easily ingest images from HDFS into Spark DataFrame (example:301)
  • Pre-process image data using transforms from OpenCV (example:302)
  • Featurize images using pre-trained deep neural nets using CNTK (example:301)
  • Use pre-trained bidirectional LSTMs from Keras for medical entity extraction (example:304)
  • Train DNN-based image classification models on N-Series GPU VMs on Azure
  • Featurize free-form text data using convenient APIs on top of primitives in SparkML via a single transformer (example:201)
  • Train classification and regression models easily via implicit featurization of data (example:101)
  • Compute a rich set of evaluation metrics including per-instance metrics (example:102)

See our notebooks for all examples.

A short example

Below is an excerpt from a simple example of using a pre-trained CNN to classify images in the CIFAR-10 dataset. View the whole source code as an example notebook.

.
import mmlspark as mml
# Initialize CNTKModel and define input and output columns
cntkModel = mml.CNTKModel().setInputCol("images").setOutputCol("output").setModelLocation(modelFile)
# Train on dataset with internal spark pipeline
scoredImages = cntkModel.transform(imagesWithLabels)
.Copy the code

See other sample notebooks as well as the MMLSpark documentation for Scala and PySpark.

Setup and installation

Spark package

MMLSpark can be conveniently installed on existing Spark clusters via the --packages option, examples:

Spark-shell -- Packages Azure: MMLSpark :0.9 PySpark -- Packages Azure: MMLSpark :0.9 Spark-submit Azure: MMLSpark :0.9 MyApp.jarCopy the code

HDInsight

To install MMLSpark on an existing HDInsight Spark Cluster, you can execute a script action on the cluster head and worker nodes. For instructions on running script actions, see this guide.

The script action url is: mmlspark.azureedge.net/buildartifa… .

If you’re using the Azure Portal to run the script action, go to Script actions → Submit new in the Overview section of your cluster blade. In the Bash script URI field, input the script action URL provided above. Mark the rest of the options as shown on the screenshot to the right.

Submit, and the cluster should finish configuring within 10 minutes or so.

Databricks cloud

To install MMLSpark on the Databricks cloud, create a new library from Maven coordinates in your workspace.

For the coordinates use: Com. Microsoft. Ml. Spark: mmlspark: 0.9. Then, under the Advanced Options, use https://mmlspark.azureedge.net/maven for the repository. Ensure this library is attached to all clusters you create.

Finally, ensure that your Spark cluster has at least Spark 2.1 and Scala 2.11.

You can use MMLSpark in both your Scala and PySpark notebooks.

SBT

If you are building a Spark application in Scala, add the following lines to your build.sbt:

```scala resolvers += "MMLSpark Repo" at "https://mmlspark.azureedge.net/maven" libraryDependencies += "Com. Microsoft. Ml. Spark" % % "mmlspark" % "0.9" ` ` `Copy the code

Contributing & feedback

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

See CONTRIBUTING.md for contribution guidelines.

To give feedback and/or report an issue, open a GitHub Issue.

Other relevant projects

Apache ®, Apache Spark, And Spark® are either registered and Verified as a result of the Apache Software Foundation in the United States and/or other countries.