I continued my long reading in Neural Networks today. I'm now setup with Theano, and working through the DeepLearning.net tutorials... but wow, I was not prepared for the (lack of) speed. Hours and hours to train a network (I don't have an nVidia GPU, so everything is being done on the CPU); my very first tutorial run is still going 4 hours later.

After a whole afternoon of going through the tutorials I now have a feel for how networks are set up in Theano. I haven't made any progress on getting a good, general data-set for voice dictation training. I may have to resort to VoxForge, but that just doesn't seem to have a reasonable download mechanism. I'm still not confident that I can write Theano code, but at least I "get" it (a bit).

I've spent quite a bit of time reviewing tutorial code for Long Short Term Memory and Recurrent Neural Networks, which are the two ways that people seem to be tackling time sequences (keeping in mind that I want to play with voice dictation tasks). Also been reading more on Restricted Boltzmann Machines (which seem to be vector clumping tools, a kind of pre-classifier that pulls data-sets into similar pools of things) and Deep Belief Networks (a stack of RBMs that you top off with a layer that tries to match the RBM stack's result to known classes).

I'm going to have to figure out how to iterate on this stuff a lot faster; 4+ hours for an extremely simple tutorial-level run isn't going to make for a rapid development environment trying to build a real tool. Thing is, the improvements in accurracy in the sample I'm running aren't all that compelling for the cost; epoch 9 had a 2% error rate, epoch 106 has brought that down to 1%, but IIRC the MNIST test set is one of those sample data-sets where the hard ones are more "sure, the user meant that, but I wouldn't have got it right and would have asked for clarification rather than imagining myself to be confident that it's 1 or 7". So maybe I'll have my training functions stop after a given static error threshold.