Improving The Accuracy Of Your Neural Network

Photo by Preethi Viswanathan on Unsplash

Neural networks were inspired by neural processing that occurs in the human brain. Though they are a much watered-down version of their human counterpart (our brain), they are extremely powerful. Deep networks have improved computers’ ability to solve complex problems given lots of data. But there are various circumstances in which the accuracy of a network is below par for the task at hand.

In such scenarios, seeking out methods to improve the performance of the network may be vital for the success of the project. For the remainder of this article, I’m going to cover various techniques you can employ to build better neural networks.

Do you prefer to watch this tutorial? See Improving the Accuracy of your Neural Network

Regularization

Deep neural networks overfit a lot. This is because they must learn millions of parameters whilst the model is being built. As a consequence, deep neural networks are equipped with the capacity to completely memorize values from the training data, thus rendering it ineffective when required to generalize to new, unseen samples.

Identifying whether your model has overfitted is simple. If the training accuracy is much higher than the test accuracy, then you’ve overfitted. For example, the image below shows a model that has completely memorized the training set data but doesn’t quite perform as well on the test data.

Image generated by the author.

A solution to this problem is to regularize the model. Regularization, in essence, is a set of techniques used to prevent overfitting. Popular techniques in deep learning include: L1 or L2 normalization and introducing dropout layers.

Hyperparameter Tuning

Hyperparameter tuning is a necessity for any practitioner seeking to maximize their model’s performance. It’s the process of selecting the optimal set of hyperparameters for a learning algorithm. This procedure alone can significantly improve a model’s performance thus permitting it to make better generalizations when faced with unseen samples.

“Values selected as hyperparameters control the learning process, therefore, they are different from normal parameters since they are selected prior to training a learning algorithm.”
— Hyperparameter Optimization for Beginners

Obtaining the optimal hyperparameters is a case of trial and error: there’s no clear way to know what hyperparameters are best suited. With that being said, let’s have a look at some of the different hyperparameters that can be tuned:

Activation function —An activation function in a neural network defines how the weighted sum of the input is transformed into an output from a node or nodes in a layer of the network. [Source: Machine Learning Mastery]

Learn more about how to get the most out of your models through experiment standardization by checking out our latest Office Hours series on YouTube.

Number of neurons — In each layer, we define the number of neurons. They attempt to model the function of a biological neuron by computing the weighted average of a given input.

Learning rate — The learning rate determines how big of a step should be taken while moving towards a minimum of a loss function.

Batch size — The number of training samples being used in one forward and backward pass.

Epochs — An epoch defines the number of times a model will see the entire training data.

Add More Data

Data is at the heart of machine learning; to build a good model, you need to have good data. The amount of data required for deep neural networks to perform well is more than what’s required for traditional machine learning methods. If the data going in is garbage, no matter how much you have, the performance of your model will reflect this discrepancy.

“Garbage in, garbage out.”

The data acquisition process can be extremely expensive. Equipping yourself with various techniques such as scraping and data augmentation may be helpful when this is the case. But if data is available, you may be better off collecting more data when your deep network fails to improve on the test dataset, after performing some of the techniques proposed in this article.

Ensembling

We can improve the accuracy of our neural network by using it as one of the constituents that make up an ensemble algorithm. An ensemble describes the combination of multiple predictors being put together to be used as an individual predictor. Practitioners agree that the predictive performance of an ensemble algorithm is usually better than that of any one of the constituent algorithms alone.

Winning solutions in several machine learning competitions typically leverage ensembles. How an ensemble is created can vary but the three main techniques are:

• Bootstrap aggregation (Bagging) — Combining the predictions of various models that are each built using randomly sampled data with replacement from the training set.
• Boosting — Combining a set of weak learners into one strong learner to minimize training error.
• Stacked generalization (Stacking) — Combining the outcomes of several other learning algorithms.

Conclusion

Building deep networks that meet the desired generalization level for a specific task is an extremely iterative process. Defining a baseline model is plays a major role in this process. A baseline model will provide you with a point of comparison when using more advanced methods such as deep neural networks. This means you’ll be left with more context as to where your neural network is striving and where it’s failing at a certain task — which you could use to direct how you improve your model.

Thanks for reading.