Actually, it really is from scratch. I made this because the web is flooded with "from scratch" tutorials that use Keras, PyTorch, and SKLearn (smh...). That being said, numpy
and scipy
are used for computational purposes only. Hopefully, this repository can be helpful to someone.
Clone or download the repo and add it to your code. Please cite the author (Anthony Krivonos).
git clone https://github.com/anthonykrivonos/mlp.git
import * from neural_network.neural_network
...
nn = NeuralNetwork(input_size = 784)
# First activation and normalization layers
nn.add(Layer(512, Activation.sigmoid))
nn.add(RegLayer(Regularization.normalize))
# Second activation and normalization layers
nn.add(Layer(256, Activation.leaky_relu, 0.2))
nn.add(RegLayer(Regularization.normalize))
# Dropout layer
nn.add(RegLayer(Regularization.dropout, 0.3))
nn.add(Layer(output_size, Activation.softmax), verbose = True)
nn.train(X_train,
y_train_encoded,
batch_size = 1024,
epochs = 10,
verbose = True,
shuffle = True,
lr = 100,
loss_type = Loss.cross_entropy,
validation_split = 0.1,
use_best_weights = True,
dynamic_lr_rate = 0.8,
dynamic_lr_limit= 10,
dynamic_lr_epochs = 2)
Performance on MNIST dataset:
(Note: we'd love for others to commit their successfully trained NN's to the tests/ directory.)
Add densely connected layers with ease and customize them to your heart's desire.
nn = NeuralNetwork(input_size = 40000)
nn.add(Layer(128, Activation.sigmoid))
nn.add(Layer(64, Activation.relu))
...
- Sigmoid [1]
- RELU [1]
- Leaky RELU [1]
- RELU with Gaussian Noise [2]
- ELU [1]
- Linear
- Softmax [1]
layer_size = 8
nn.add(Layer(layer_size, Activation.sigmoid))
nn.add(Layer(layer_size, Activation.relu))
nn.add(Layer(layer_size, Activation.leaky_relu, 0.2)) # 0.2 is the leak
nn.add(Layer(layer_size, Activation.noisy_relu, 0.2)) # 0.2 is the Gaussian noise
nn.add(Layer(layer_size, Activation.elu, 0.2)) # 0.2 is the ELU coefficient
nn.add(Layer(layer_size, Activation.linear))
nn.add(Layer(layer_size, Activation.softmax))
- L1 Regularization
- L2 Regularization
- Dropout [3]
- Normalization
nn.add(RegLayer(Regularization.l1, 0.4)) # 0.4 is the regularization lambda
nn.add(RegLayer(Regularization.l2, 0.4)) # 0.4 is the regularization lambda
nn.add(RegLayer(Regularization.dropout, 0.5)) # 0.5 is the dropout rate
nn.add(RegLayer(Regularization.normalize)) # 0.5 is the dropout rate
Load weights with nn.load_weights('filename')
and save weights nn.save_weights('filename')
. Just ensure that the neural network retains its identical layer structure (for instance, you can't load a 128-input-size weight vector into a network that takes in an input size of 64).
Basic feedforward and backpropogation algorithms inspired by Ali Mirzaei's article on Medium. [4]
[1] Chigozie Nwankpa, et al. "Activation Functions: Comparison of trends in Practice and Research for Deep Learning." (2018).
[2] Yinan Li, et al. "Whiteout: Gaussian Adaptive Noise Regularization in Deep Neural Networks." (2016).
[3] Nitish Srivastava, et al. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting". Journal of Machine Learning Research 15. 56(2014): 1929-1958.
[4] Ali Mirzaei. "Implement a neural network from scratch with Python/Numpy β Backpropagation". Medium. (2019).