Skip to content

A short introduction on deep learning solving a classical OCR problem. This project is a proof of concept performed for the Machine Learning laboratory on University Politehnica of Bucharest

License

Notifications You must be signed in to change notification settings

imironica/keras-without-a-phd

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Keras without a PhD

This repository contains a Keras version of 'TensorFlow and Deep Learning without a PhD' presentation from Google.

The lab is taking few hours to be ran and understood. The exercises take you through the design and optimisation of a neural network for recognising handwritten digits, from the simplest possible solution all the way to a recognition accuracy above 99%. It covers feed-forward (with one or more layers with different activation functions) and convolutional networks, as well as techniques such as learning rate decay and dropout. We also perform a comparison with other classical machine learning techniques, such as Nearest Neighbor, Support Vector Machines, Random Forests and gradient boosted trees.

Table of contents

  1. Project installation
  2. Project description
  3. Results and learnings

1. Project installation

[back to the top]

Installation steps (tested on Ubuntu)

Install GIT

sudo apt-get install git

Get the source code

mkdir keras-without-a-phd

cd keras-without-a-phd

git clone https://github.com/imironica/keras-without-a-phd.git

Install latest python packages from requirements.txt file

pip3 install -r requirements.txt

2. Project description

[back to the top]

This case study shows how to create a model for a classical OCR problem using th MNIST dataset.

The MNIST database of handwritten digits is splited in two main components:

  • Training set: 60,000 image of digits of size 28x28

  • Test set: 10,000 images

More details about the database may be found on http://yann.lecun.com/exdb/mnist/

Image examples from the MNIST dataset

3. Results and learnings

[back to the top]

We have tested a broad list of ML algorithms, starting from classical approaches (SVM, Nearest neighbors, Naive Bayes, Random forests, Boosted trees) to deep learning architectures (feed forward and convolutional neural networks). All presented scores were computed using the accuracy metric.

Results with classical ML approaches

Run 0.5_classical_ml.py

- Nearest neighbors (3): 0.9705
- Stochastic Gradient Descent: 0.8985
- Naive Bayes: 0.5558
- Decision Tree Classifier : 0.879
- Adaboost Classifier: 0.7296
- Gradient Boosting Classifier: 0.6615
- Random Forest Classifier: 0.9704
- Extremelly Trees Classifier: 0.9735
- Linear SVM with C=0.01: 0.9443
- Linear SVM with C=0.1: 0.9472
- Linear SVM with C=1: 0.9404
- Linear SVM with C=10: 0.931
- RBF SVM with C=0.01: 0.835
- RBF SVM with C=0.1: 0.9166
- RBF SVM with C=1: 0.9446
- RBF SVM with C=10: 0.9614

Results with Feed Forward Neural Network architecture

Scripts: 1.0_softmax.py, 1.1_sigmoid.py, 2.0_five_layers_sigmoid.py, 2.1_five_layers_relu.py, 2.2_five_layers_relu_lrdecay.py, 2.3_five_layers_relu_lrdecay_dropout.py

- One Softmax layer: 0.9187
- Two layers: Sigmoid -> Softmax: 0.9676
- Five sigmoid layers: 0.9745
- Five relu layers layers: 0.9755
- Five relu layers with learning rate decay: 0.9785
- Five relu layers with learning rate decay and dropout: 0.9795

Results with Convolutional Neural Network architecture

Scripts: 3.0_convolutional.py, 3.1_convolutional_dropout.py, 4.0_five_layers_relu_lrdecay_batchnorm.py, 4.1_convolutional_dropout_batchnorm.py, 4.2_convolutional_dropout_batchnorm_maxpool.py

- 3 convolutional layers + flatten + softmax: 0.9862
- 3 convolutional layers + flatten + dropout + softmax: 0.99
- 3 convolutional layers + batch normalization + flatten + dropout + softmax + learning rate decay: 0.9948

About

A short introduction on deep learning solving a classical OCR problem. This project is a proof of concept performed for the Machine Learning laboratory on University Politehnica of Bucharest

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages