Optimisation of hill climbing algorithm in c# for training neural networks -

i have written small project in c# creates , trains neural networks. more details see previous question here: (https://scicomp.stackexchange.com/questions/19481).

the neural networks perform after enough training, realise self-written hill climbing algorithm may not perfect , i'm looking suggestions improvement. in particular, can reach local optimum less calls fitness evaluation function?

there don't seem many examples around web simple hill climbing algorithms in c#. there .net math library, prefer not have pay something.

the hill climbing algorithm runs on every weight , every bias in network train network, , run multiple passes. have looked propagation, seems applied single training example, have ~7000 examples in training data, , fitness function evaluates average performance of network on of them , returns continuous (double) score.

here current code:

    public static double improveproperty(ref double property, double startingfitness, int maxiters, random r, ref defs.network network, func<defs.network, double> fitnessfunction)     {         //record starting values         var lastfitness = startingfitness;         var lastvalue = property;         //randomise magnitude of change reduce chance          //of getting stuck in local optimums         var magnitude = r.nextdouble();         var positive = true;         var itercount = 0f;         var magnitudechange = 5;         while (itercount < maxiters)         {             itercount++;             if (positive)             {   //try adding positive value property                 property += magnitude;                 //evaluate fitness                 var fitness = fitnessfunction(network);                 if (fitness == lastfitness)                 {   //no change in fitness, increase magnitude , re-try                     magnitude *= magnitudechange;                     property = lastvalue;                 }                 else if (fitness < lastfitness)                 {   //this change decreased fitness (bad)                     //put property , try going in negative direction                     property = lastvalue;                     positive = false;                 }                 else                 {   //this change increased fitness (good)                     //on next iteration try                      //to apply same change again                     lastfitness = fitness;                     lastvalue = property;                     //don't increase iteration count                     //if change made                     itercount -= 0.9f;                 }             }             else             {   //try adding negative value property                 property -= magnitude;                 var fitness = fitnessfunction(network);                 if (fitness == lastfitness)                 {                     //no change in fitness, increase magnitude , re-try                     magnitude *= magnitudechange;                     property = lastvalue;                 }                 else if (fitness < lastfitness)                 {                     //this change decreased fitness (bad)                     //now know going in positive direction                      //and negative direction decreases fitness                     //so make magnitude smaller close optimum                     property = lastvalue;                     magnitude /= magnitudechange;                     positive = true;                 }                 else                 {                     //this change increased fitness (good)                     //continue in same direction                     lastfitness = fitness;                     lastvalue = property;                     itercount -= 0.9f;                 }             }             //check bounds prevent math functions overflowing             if (property > 100)             {                 property = 100;                 lastfitness = fitnessfunction(network);                 return lastfitness;             }             else if (property < -100)             {                 property = -100;                 lastfitness = fitnessfunction(network);                 return lastfitness;             }         }         return lastfitness;     } 

the fitness function expensive, should called little possible. i'm looking improvements in getting local optimum less calls fitness function. getting stuck in local optimum not of concern, have graphed fitness function against value of different weights , biases in network, , looks there between 1-3 local optimums in graph. if network remains @ same fitness few passes can add parameter function attempt restarting hill climbing random value.

this approach won't scale; trying evaluate expensive total fitness function several times small improvement in single parameter. entire reason why gradient-based methods @ optimization problem sample sample, or more usually, mini-batch mini-batch. fitness function decomposes sum of fitness functions on each sample (or batch), , allows compute small update , take step in right direction.

you should read on theory bit. there many online resources, instance this online book.

alternatively, if aren't interested in theory behind neural networks , want apply them, recommend not reinvent wheel , instead use 1 of many open source toolkits neural networks, instance torch7, caffe, pylearn2, lasagne, keras, theanets ...