Well it’s been a hard few weeks. I recently lost my granddad to terminal lung cancer and I want to take this opportunity to thank him for all his support, he’s the reason I was able to study for my masters, providing the funding for the tuition fees. He meant a lot to me and he is sorely missed.
The dissertation it’s self has come along a lot. A long good brainstorm with my brilliant supervisor Dr. Julie Greensmith resulted in a rework of the application and the methodology for the project.
I’ve been making the revisions and writing up the methodology over the past week or so and I’m making some good progress.
Some new techniques have been learned by myself in the process, such as how to manipulate Dictionary<>() class collections. These are essentially dynamically buffered associative arrays. But require certain methods of manipulation to work well. I’m now using them because I need an ID number to cross reference classified binary strings as I’ve removed the symbolic string at the end of the feature vector which gives the type i.e. normal or an attack. Anyway I had a sharp lessen in optimisation and how these collections are enumerated. Initially I was using for loops then using a LINQ .ElementAt() to loop through the Dictionary collections. I realised that this was not ideal but I have numerous nested loops and when using arrays within arrays (arrayception?) aka jagged arrays, it was easier for me to follow and manipulate them easier using the for loops.
This led to some serious problems. Firstly I hadn’t actually enumerated the collection straight off, so each time the loop executed the collection would enumerate and them try to find the elements using the LINQ expression. This meant when writing to file the 500,000 feature vectors were taking about 30 minutes to write. To give some context, I write to file in this program about 40 times, all different sized data sets but still enough to make this unworkable.
So I took my finger out and read up, used the foreach() loop which enumerates the collection first then works on looping and the same process took about 5 seconds. I felt like a muppet to say the least. But all clouds have a silver lining, I’ve now learned how to use and manipulate another collection better and I also learned more LINQ statments. So overall I’m going to chalk that one up as a win.
I’m going to be super busy over the next few weeks finishing off this dissertation thesis so I won’t be doing anymore entries until it’s handed in. But once it’s done I’m going to get the projects page finished and upload the source code and my thesis to there with a link to the Dissertation blog page.
So that’s all for now, I’d like to finish this off by once again thanking my granddad for his love and support, he will be deeply missed.