Pre-reqs for learning deep learning, unsupervised feature learning, neural networks

So, you are interested in understanding unsupervised feature learning, deep learning, convolutional networks, etc? You have tried various tutorials but you seem to have not understood what the author was talking about. This post lists most of the pre-reqs you will need to understand (not an exhaustive list).

Best starting place on the web is Machine Learning on coursera. It will get you started on what Machine Learning is all about. Next try to understand topics in http://ufldl.stanford.edu/ . I think the ufldl website isn't completely edited for the new version. So, feel free to browse both old and the new versions. If you are not so great in math, I would recommend this book as a good starting point to get basic calculus used in neural networks. This is also a good starting point to understand backpropogation.

This is the order in which I studied them. If you know some of them already, feel free to skip them.

Pre-reqs to learn (try to learn the theory and how they work in practice):

1. Linear regression (including square error terms, mean error terms, concept of regularlization, L1 and L2 distance).
2. Gradient Descent (intuition of difference between conjugate, stochastic, mini-batch, batch/regular gradient descent). Awareness of bfgs, lbfgs optimization algorithms (you don't need to understand the details).
3. Activation functions (sigmoid, tan h, ReLU) and their derivatives.
4. Logistic regression (binary and multi-class classification).
5. Softmax regression (multinomial logistic regression).
6. Artificial Neural Networks (including feed forward back-propagation).
7. SVM. (with intuition of different kernels)
8. Dimensionality reduction using LDA, PCA.
9. Autoencoder (including the idea of denoising autoencoder and sparse autoencoder).
10. Autoencoder with linear decoder (doesn't use a sigmoid like in regular autoencoder).
11. Greedy layer wise training in deep learning.
12. Stacked auto-encoder (how to combine multiple autoencoders with a softmax regression final layer).
13. Convolution operation, Pooling (max, mean, stochastic).
14. Convolutional Neural Networks.
15. Restricted Boltzman Machines.
16. Deep Belief Networks.

Convolution with filters obtained from Autoencoders (unsupervised learning) help Convolutional Neural Networks in unsupervised feature learning. Sometimes, convolution filters used are simple gabor filters.

Other interesting links:

1. Deep Belief Networks-Deep Boltzman Machines.