What's Happening

ONNX Standardization for Neural Networks

At the Build conference this year, Microsoft announced that ONNX support would be rolled out through many of their Cloud and Edge Machine Learning products.

In December of last year, Microsoft announced that ONNX 1.0 was production ready. The ONNX standard is a specification that developers can implement their Neural Networks against that will work with any ONNX compliant tooling regardless of vendor. ONNX is developed and supported by a community of partners including AWS, Microsoft, and Facebook. Currently, outside of the Microsoft ecosystem, there is support for Tensorflow, CoreML, PyTorch, MXNet, and Caffe2. (A full list is available here) Microsoft has released support for the Cognitive Toolkit (CNTK), ML.NET, Azure Cognitive Services, and Azure Intelligent Edge devices.

What exactly is ONNX?

The ONNX website describes it as an open format to represent deep learning models. With ONNX, AI developers can more easily move models between state-of-the-art tools and choose the combination that is best for them. ONNX is developed and supported by a community of partners.


ML.NET leverages Tensorflow, Caffe2, and CNTK to do training and provides a C# API for developing and using Neural Networks. ML.Net v 0.3 includes support for exporting and importing to the ONNX-ML format. This allows the consumption of an ONNX model trained in the cloud which can be deployed to a cloud service or an Intelligent Edge device running Windows ML.


CNTK was one of the very first deep learning toolkits to support ONNX. With CNTK 2.3.1 the ability to export an ONNX model and import an ONNX model requires just a couple of lines of code and works with both C# and Python.

The Microsoft Cognitive Toolkit (CNTK) is an open-source toolkit for commercial-grade distributed deep learning. It describes neural networks as a series of computational steps via a directed graph. CNTK allows the user to easily realize and combine popular model types such as feed-forward DNNs, convolutional neural networks (CNNs) and recurrent neural networks (RNNs/LSTMs). CNTK implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers. — Microsofts Cognitive Toolkit Overview

Azure Cognitive Services

One of the Cognitive Services available is the Custom Vision Service. This service allows the developer to train a pre-created neural network for image classification with a custom image set allowing for the easy creation of domain specific vision services. The Custom Vision Service now includes support for exporting into the ONNX format, which allows the neural network to be used on other technology stacks and platforms that support ONNX. This support is available today with a tutorial available here.

Azure Intelligent Edge Devices

ONNX models can be imported on edge devices running Windows allowing for predictions outside the cloud and when in a disconnected state. Any IOT device using Azure IoT Edge is now able to run ONNX models locally and take advantage of any hardware acceleration available, even devices as small as a Raspberry Pi. In addition, Microsoft released Windows ML which allows .NET developers to load ONNX models and run those in a local application such as a Modern app or a UWP application.

Windows ML is an API for on-device evaluation of trained deep learning and machine learning models. It is built to help developers with scenarios where evaluation of machine learning models locally might be more advantageous, due to the lack of a reliable internet connection, latency before getting prediction results (particularly important for real-time applications) or data privacy considerations where your customers wouldn’t be willing to have their data leave the device.

ONNX Model Zoo

ONNX support makes it very easy to import and export models and has lead to the creation of the ONNX Model Zoo. The ONNX Model Zoo is a collection of pre-trained state-of-the-art models in deep learning, available in the ONNX format. Accompanying each model are Jupyter notebooks for model training and running inference with the trained model. The notebooks are written in Python and include links to the training dataset as well as references to the original paper that describes the model architecture.

What can you build with these models?