HTTPS SSH

Scala / Spark Neural Network framework

A demonstration configurable, scalable and testable Convolutional / Fully Connected Neural Network framework (CNN / FCN) written in Scala, tested using Tensorflow.

Produced by Mike Whitfield at Virtual Business Systems

Overview

Motivation

My background is in commercial IT, however over the last year I've gained a reasonable knowledge of Neural Networks through online courses and code experimentation. To verify understanding I developed this as a "proof of knowledge" project.

Architecture

The framework uses Breeze and Apache Spark.

Parallelism is achieved purely using Scala's parallel collections when running on a single machine and Spark RDD's when running on a cluster. As currently implemented, feed forward and back propagation of each entry within a minibatch is processed concurrently through all layers of the CNN. Within the FCN each mini batch is processed as a single pair of feature set and label DenseMatrix's, therefore there is no explicit parallelism within the FCN.

Implemented Features

Weight Initialization:

  • Gaussian
  • Xavier

Within the Fully Connected Network:

  • Activation Functions:
    • Relu
    • Softmax
    • Sigmoid
  • Cost Functions:
    • Cross Entropy
    • Squared Error
  • Regularization Functions:
    • L1
    • L2
    • Dropout
  • Optimisation Functions:
    • Simple Gradient Descent
    • Momentum
    • Adam

Within the Convolutional Network:

  • Convolutional layers with Relu activation and Simple or Adam optimisation
  • Max / Average Pooling Layers

Testing Strategy

I developed a small set of Python Scripts, run via a Jupyter Notebook that allows all layer outputs, weights /biases and gradients to be persisted when training a small dataset using Tensorflow.

In conjunction with "debug" extensions of the Scala framework's feed forward and back propagation components, this allows each layer of the network to be verified against the data produced by Tensorflow.

For a given network configuration, the python scripts cover:

  • The generation and persistence of small datasets
  • The persistence of gradients and values for all Tensorflow trainable variables within each iteration of the run
  • A run through an equivalent pure Numpy configuration to persist all intermediate layer outputs and verify the correctness of the run against weights and gradients output by Tensorflow

See the Wiki for a testing example.

Development Environment

The Scala framework was developed using release 4.6 of the Scala IDE

The Python scripts were developed using Anaconda.

What's next?

As I think I've learnt as much as I can from this exercise I will probably move onto something new, but I may continue to:

  • Implement Batch Normalisation within the FCN
  • Attempt to get the framework running as a library within a notebook

Contact the developer

  • Email mike@virtual-businesses.com
  • Linked In