Image Image Image




Post new topic Reply to topic  [ 106 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Mar 29, 2011 6:29 am 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
I tried a number of others, but these were the top three results...

71.4% correct in weka with (meta) Bagging+RandomTree for 500 iterations.
71.07% correct in weka with (meta) RotationForest+RandomTree for 500 iterations.
70.6% correct in weka with (meta) RandomCommittee+RandomForest for the default number of iterations.


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Oct 04, 2012 9:41 pm 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
I'm not sure if anybody cares about this still, but I was able to achieve a test data accuracy of 73.78% with an Encog MLP.


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Oct 04, 2012 10:03 pm 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
Taking the pure output (i.e. setting the highest output to 1 and all others to 0), I was able to achieve an accuracy of 77.55% against the test data using an SLFF Encog NN. Interestingly, this accuracy was achieved before the training error reached 0.15. Other levels of purification were tested with reduced results.


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Oct 04, 2012 10:38 pm 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
By using only the inputs defined by iamnobody's YAGGA2 results and 100% purification, I was able to achieve a 79.17% accuracy.


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Oct 04, 2012 10:56 pm 
Online
Junior member
User avatar

Posts: 35
Favourite Bot: Working on it...
Interesting, what's the layout for you MLP in Encog? I'm only getting around 65% using it.
How many hidden layers? How long are you training for?
E.g. right not i'm training on around 2/3 of the data and testing the last 1/3, if the error starts rising I assume it's beginning to over fit and stop training.
I'll keep playing around and see what I get.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Oct 04, 2012 11:08 pm 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
JP11 wrote:
Interesting, what's the layout for you MLP in Encog? I'm only getting around 65% using it.
How many hidden layers? How long are you training for?
E.g. right not i'm training on around 2/3 of the data and testing the last 1/3, if the error starts rising I assume it's beginning to over fit and stop training.
I'll keep playing around and see what I get.


For that, I think I was using 2 hidden layers, inputs * 1.5 in length.

I wasn't training to a specific point then stopping, I was looking at the accuracy of the test data every time a new lowest error rate for the training data was reached. Just trying to find the theoretical "best."


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 05, 2012 9:16 am 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
I'm still interested.

Quote:
Taking the pure output (i.e. setting the highest output to 1 and all others to 0), I was able to achieve an accuracy of 77.55%

accuracy = number of instances correctly predicted / total number of instances ?

SLFF = Single Layer Feed Forward ?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 05, 2012 9:27 am 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
spears wrote:
accuracy = number of instances correctly predicted / total number of instances ?

Sort of. Like this: SUM(1-Math.Abs(ouput(i) - ideal(i))) / number_of_comparisons

spears wrote:
SLFF = Single Layer Feed Forward ?

Yes. Although, after looking up the definition, I was mistaken in my understanding of the term. The NN tested did have a single hidden layer. I'll post the code tomorrow so you guys can find all the errors. ;)


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 05, 2012 12:45 pm 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
As promised. And, as usual, sorry about the VB. ;)

Code:
Public Shared Sub trainTURN(ByVal iterations As Integer)
           'I1,I2,I3,I4,I5,I6,I7,I8,I9,I10,I11,I12,I13,I14,I15,I16,I17,I18,I19,I20,I21,I22,I23,I24,I25,I26,I27,I28,I29,I30,I31,I32,I33,I34,I35,I36,I37,I38,I39,I40,I41,I42,I43,O1,O2,O3,O4
        '0,0,0.06,0,0,0,0.075,1,0.3,0,0,0,0,0,0.165,0,0.5,0,0,0,0,0,0,0,0,0,1,0.5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,O3
        '0,0,0.06,0,0,0,0.075,1,0.3,0,0.12,1,0.2,0.045,0.265,0,0.5,0,0,0,0,0,0,0,0,0,1,0.5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,O2


        Dim data() As String = System.IO.File.ReadAllLines("turn")

        Dim test_count As Integer = CInt(data.Length * 0.66)

        Dim set_input(test_count)() As Double
        Dim set_ideal(test_count)() As Double

        Dim comp_input(data.Length - 3 - test_count)() As Double
        Dim comp_ideal(data.Length - 3 - test_count)() As Double

        'Dim use_only() As Integer = { 5, 7, 13, 14, 27, 28, 34}
        Dim use_only() As Integer = {5, 7, 13, 14, 27, 34}

        For i As Integer = 1 To set_input.Length

            Dim si() As String = data(i).Split(",")
            Dim inputs(si.Length - 5) As Double
            Dim ideals(2) As Double

            For j As Integer = 0 To inputs.Length - 1
                If Array.IndexOf(use_only, j) >= 0 Then
                    inputs(j) = CDbl(si(j))
                Else
                    inputs(j) = 0
                End If
            Next

            ideals(0) = CDbl(si(si.Length - 4))
            ideals(1) = CDbl(si(si.Length - 3))
            ideals(2) = CDbl(si(si.Length - 2))

            set_input(i - 1) = inputs
            set_ideal(i - 1) = ideals

        Next

        For i As Integer = set_input.Length + 1 To data.Length - 1

            Dim si() As String = data(i).Split(",")
            Dim inputs(si.Length - 5) As Double
            Dim ideals(2) As Double

            For j As Integer = 0 To inputs.Length - 1
                If Array.IndexOf(use_only, j) >= 0 Then
                    inputs(j) = CDbl(si(j))
                Else
                    inputs(j) = 0
                End If
            Next

            ideals(0) = CDbl(si(si.Length - 4))
            ideals(1) = CDbl(si(si.Length - 3))
            ideals(2) = CDbl(si(si.Length - 2))

            comp_input(i - set_input.Length - 1) = inputs
            comp_ideal(i - set_input.Length - 1) = ideals

        Next

        Console.WriteLine(data.Length & " instances loaded.")
        Console.WriteLine(set_input(0).Length & " inputs.")

        data = Nothing


        Dim lowest_error As Double = Double.PositiveInfinity

        Dim r_network As New BasicNetwork()
        r_network.AddLayer(New Layers.BasicLayer(New ActivationTANH(), False, set_input(0).Length))
        r_network.AddLayer(New Layers.BasicLayer(New ActivationTANH(), True, CInt(set_input(0).Length * 1.5)))
        r_network.AddLayer(New Layers.BasicLayer(New ActivationSigmoid(), False, 3))
        r_network.Structure.FinalizeStructure()
        r_network.Reset()


        Dim train_data As New Encog.ML.Data.Basic.BasicMLDataSet(set_input, set_ideal)
        Dim comp_data As New Encog.ML.Data.Basic.BasicMLDataSet(comp_input, comp_ideal)

        'Dim r_teacher As New ResilientPropagation(r_network, train_data, 0.00001, 0.00001)

        Dim r_teacher As New ResilientPropagation(r_network, train_data)

        Console.WriteLine("Training...")

        r_teacher.Iteration()

        Dim comp_accuracy As Double = Double.NegativeInfinity

        Dim er As Double = 0

        Dim plevel As Double = 0.05

        For i As Integer = 1 To iterations

            r_teacher.Iteration()

            er = r_teacher.Error()


            If er < lowest_error Then

                Dim temp_accuracy As Double = 0
                For j As Integer = 0 To comp_input.Length - 1
                    Dim output(2) As Double
                    r_network.Compute(comp_input(j), output)

                    'Dim total As Double = 0
                    'For q As Integer = 0 To output.Length - 1
                    '    total += output(q)
                    'Next
                    'Dim newtotal As Double = 0
                    'For q As Integer = 0 To output.Length - 1
                    '    output(q) /= total
                    '    If output(q) < plevel Then output(q) = 0
                    '    newtotal += output(q)
                    'Next
                    'For q As Integer = 0 To output.Length - 1
                    '    output(q) /= newtotal
                    'Next

                    If output(0) > output(1) And output(0) > output(2) Then
                        output(0) = 1 : output(1) = 0 : output(2) = 0
                    ElseIf output(1) > output(0) And output(1) > output(2) Then
                        output(0) = 0 : output(1) = 1 : output(2) = 0
                    ElseIf output(2) > output(1) And output(2) > output(0) Then
                        output(0) = 0 : output(1) = 0 : output(2) = 1
                    End If

                    Dim immac As Double = 0
                    For q As Integer = 0 To output.Length - 1
                        immac += 1 - Math.Abs(comp_ideal(j)(q) - output(q))
                    Next
                    temp_accuracy += immac / output.Length
                Next
                temp_accuracy /= comp_input.Length

                If temp_accuracy > comp_accuracy Then comp_accuracy = temp_accuracy

                lowest_error = er
            End If

            Console.WriteLine(i & vbTab & er & vbTab & lowest_error & vbTab & comp_accuracy)

        Next

    End Sub


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Sat Oct 06, 2012 9:41 pm 
Online
Junior member
User avatar

Posts: 35
Favourite Bot: Working on it...
Thanks for the code, I'm using a more or less the same setup in java bar how the NN was configured.
My code is as so:
Code:
   public static void trainNetworkComplete(MLDataSet trainingSet, MLDataSet testSet,
         String networkName, int inputs, int hidden, int outputs) {
      String fileName = "/Users/***/Documents/Eclipse/***/Neural Networks/Networks/"
            + networkName + ".eg";
      
      // create a neural network, without using a factory
      BasicNetwork network = new BasicNetwork();
      network.addLayer(new BasicLayer(new ActivationTANH(), false, inputs));
      network.addLayer(new BasicLayer(new ActivationTANH(), true, (int)(inputs*1.5)));
      network.addLayer(new BasicLayer(new ActivationSigmoid(), false, outputs));
      network.getStructure().finalizeStructure();
      network.reset();

      // train the neural network
      //final Backpropagation train = new Backpropagation(network, trainingSet);
      final ResilientPropagation train = new ResilientPropagation(network, trainingSet);
      
      long start = System.currentTimeMillis();
      
      ErrorCalculation.setMode(ErrorCalculationMode.RMS);

      boolean training = true;
      int errorcounter = 0;
      double error = 1;
      int epoch = 1;
      
      while (training) {
         train.iteration();
         System.out.println("Epoch #" + epoch + " Error:" + train.getError());
         epoch++;
         
                       // every x instances i save the network   
                       // network error/accuracy is also calculated every x iterations
         if (errorcounter<25) {
            errorcounter++;
         } else {
            EncogDirectoryPersistence.saveObject(new File(fileName), network);
            
            //double er = train.getError();
            double rms = network.calculateError(testSet);
            //double correct = Evaluate.getPercentCorrect(testSet, networkName);
            
            if (rms<error) { //> for correct, < for rms
               //error = correct;
               error = rms;
               errorcounter = 0;
            } else {
               training = false;
            }
         }
      }

      network = (BasicNetwork) train.getMethod();

      // save network
      System.out.println("Saving Network...");
      EncogDirectoryPersistence.saveObject(new File(fileName), network);
      System.out.println("Network Saved.");

      long end = System.currentTimeMillis();
      System.out.println("Trained in " + (end - start) / 1000 + "s.");

      Encog.getInstance().shutdown();
   }


It seems more or less the same as your code c2008, in a different language, but that shouldn't make a difference..?

Using attributes I6 I8 I14 I15 I28 I35 like I think you did, i'm now getting around ~70% accuracy.
I stop training once I the error on the test set starts to increase, or I can set it to stop if the accuracy decreases. Both methods get similar results.

Trying to find where I'm going wrong.. might try run it for a while and see where I get!

Here's what I'm getting just now:
Code:
Neural Network Evaluation:
      PF:    PC:    PR:
AF:  [4.975, 4.67, 0.0] 9.64
AC:  [3.655, 55.13, 5.685] 64.47
AR:  [0.914, 15.13, 9.746] 25.79
      9.54   74.93  15.43  69.85
Input weight analysis:
63  1  7437  90  2  69 
RMS: 0.37360411166639884


PS my "Input weight analysis" isn't too helpful I've found, but I've not updated the code yet.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Sun Oct 07, 2012 8:36 pm 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
I'd suggest:

1) Try my code and see if you reproduce the results I did.
2) Try calculating accuracy every iteration.
3) Was my code for calculating accuracy correct?
4) Are you purifying the NN output?


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Oct 08, 2012 1:35 am 
Online
Junior member
User avatar

Posts: 35
Favourite Bot: Working on it...
Ok, had a couple of translation errors but more or less copied the code into java.
Note: I have edited the turn file and removed all 'meta' data, e.g. I1,I2,I3,...,O4 and ..x,x,x,O3.
Code:
   public static void c2008Train(int iterations) {
      
      ArrayList<String> data = new ArrayList<String>();
      
      try
      {
         String path = "lalala/turn";
         
         FileInputStream fstream = new FileInputStream(path);
         DataInputStream in = new DataInputStream(fstream);
         BufferedReader file = new BufferedReader(new InputStreamReader(in));
         
         System.out.println("Data loading...");
         
         String line;
         while ((line = file.readLine()) != null)
         {   
            data.add(line);
         }
         file.close();
         
         // Print file currently being processed
         System.out.println("Data loaded.");
         
      
      }
      catch (Exception e) {System.err.println("Error: " + e.getMessage());}         
      
      int test_count = (int) (data.size()*0.66);
      
      double[][] set_input = new double[test_count][];
      double[][] set_ideal = new double[test_count][];
      
      double[][] comp_input = new double[data.size()-test_count][];
      double[][] comp_ideal = new double[data.size()-test_count][];
      
      double[] use_only = {5,7,13,14,17,34};
      
      System.out.println("Converting data...");
      
      for (int i=0; i<set_input.length; i++) {
         
         String[] si = data.get(i).split(",");
         double[] inputs = new double[si.length-3];
         double[] ideals = new double[3];
         
         for (int j=0; j<inputs.length; j++) {
            boolean use = false;
            for (double x : use_only) {
               if (x==j) {
                  use = true;
               }
            }
            if (use) {
               inputs[j] = Double.parseDouble(si[j]);
            } else {
               inputs[j] = 0;
            }
         }
         
         
         ideals[0] = Double.parseDouble(si[si.length-3]);
         ideals[1] = Double.parseDouble(si[si.length-2]);
         ideals[2] = Double.parseDouble(si[si.length-1]);
         
         set_input[i] = inputs;
         set_ideal[i] = ideals;      
            
      }
      
      for (int i=set_input.length; i<data.size(); i++) {
         
         String[] si = data.get(i).split(",");
         double[] inputs = new double[si.length-3];
         double[] ideals = new double[3];
         
         for (int j=0; j<inputs.length; j++) {
            boolean use = false;
            for (double x : use_only) {
               if (x==j) {
                  use = true;
               }
            }
            if (use) {
               inputs[j] = Double.parseDouble(si[j]);
            } else {
               inputs[j] = 0;
            }
         }
         
         ideals[0] = Double.parseDouble(si[si.length-3]);
         ideals[1] = Double.parseDouble(si[si.length-2]);
         ideals[2] = Double.parseDouble(si[si.length-1]);
         
         
         comp_input[i - set_input.length] = inputs;
         comp_ideal[i - set_input.length] = ideals;      
            
      }
      
      System.out.println("Data converted.");
      
      System.out.println(data.size() + " instances loaded.");
      System.out.println(set_input[0].length + " inputs." );
      
      data = null;
      
      double lowest_error = Double.POSITIVE_INFINITY;
      
      BasicNetwork r_network = new BasicNetwork();
      r_network.addLayer(new BasicLayer(new ActivationTANH(), false, set_input[0].length));
      r_network.addLayer(new BasicLayer(new ActivationSigmoid(), true, (int)(set_input[0].length*1.5)));
      r_network.addLayer(new BasicLayer(new ActivationSigmoid(), false, 3));
      r_network.getStructure().finalizeStructure();
      r_network.reset();
      
      BasicMLDataSet train_data = new BasicMLDataSet(set_input, set_ideal);
      BasicMLDataSet comp_data = new BasicMLDataSet(comp_input, comp_ideal);
      
      ResilientPropagation r_teacher = new ResilientPropagation(r_network, train_data);
      
      System.out.println("Training...");
      
      
      r_teacher.iteration();
      
      double comp_accuracy = Double.NEGATIVE_INFINITY;
      double er = 0;
      
      for (int i=1; i<iterations; i++) {
         
         r_teacher.iteration();
         
         er = r_teacher.getError();
         
         if (er < lowest_error) {
            
            double temp_accuracy = 0;
            for (int j=0; j<comp_input.length; j++) {
               double[] output = new double[3];
               r_network.compute(comp_input[j], output);
               
               if (output[0] > output[1] && output[0] > output[2]) {
                  output[0] = 1; output[1] = 0; output[2] = 0;
               } else if (output[1] > output[0] && output[1] > output[2]) {
                  output[0] = 0; output[1] = 1; output[2] = 0;
               } else {
                  output[0] = 0; output[1] = 0; output[2] = 1;
               }
               
               double immac = 0;
               
//               if (Arrays.equals(output, comp_ideal[j])) {
//                  immac = 1;
//               }
               
               for (int q=0; q<output.length; q++) {
                  immac += 1 - Math.abs(comp_ideal[j][q] - output[q]);
               }
               temp_accuracy += immac / output.length;
               
//               temp_accuracy += immac;

            }
            temp_accuracy /= comp_input.length;
            
            if (temp_accuracy > comp_accuracy) {
               comp_accuracy = temp_accuracy;
            }
            
            lowest_error = er;
         }
         
         System.out.println(i + ": " + er + " " + lowest_error + " %" + comp_accuracy*100.0);
         
      }
      
      System.out.println("Saving Network...");
      String fileName = "/lalala/"
            + "Turn" + ".eg";
      EncogDirectoryPersistence.saveObject(new File(fileName), r_network);
      System.out.println("Network Saved.");
      
      Encog.getInstance().shutdown();
      
      Evaluate.evaluateNetwork(comp_data, "Turn");

   }


I did manage to reproduce c2008's code and using that technique also get ~79% accuracy.
However, I think the reason for such a great result is in the method of how the accuracy is calculated.

Using the network trained with the above code I got ~79%, and using my own evaluation get around ~69%. The commented code around c2008's current accuracy part, gets more or less the same results as my normal evaluation method.
I also purify the outputs.

I think the error is in the code:
Quote:
SUM(1-Math.Abs(ouput(i) - ideal(i))) / number_of_comparisons


This sum will always be 1 or 3.
For example:
If the predicted output is [0,0,1] and the ideal is [0,0,1] then the sum is 3.
If the predicted output is [0,0,1] and the ideal is [0,1,0] then the sum is 1.
If the predicted output is [0,0,1] and the ideal is [1,0,0] then the sum is 1.

This means that even when the prediction is wrong, the value is >0 and this adds to accuracy, wrongly.

Basically, the accuracy is increasingly with both correct classifications and incorrect classifications. When there is a correct classification, it counts as +1, when there is an incorrect, it counts as +0.33.

I could be wrong, I hope I am!


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Oct 08, 2012 2:08 am 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
Drats. You're right.


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 12, 2012 12:24 am 
Offline
Level1 member
User avatar

Posts: 40
Favourite Bot: own
I'm just starting out to use ML for action prediction and am currently running FANN on the provided test set (will post them as soon as the calculations are finished). I played a bit with WEKA to reproduce the results of OP and I was able to achieve a similar performance. However, when looking at the probability prediction output, I realized that tree-based methods often have very screwed distributions, e.g. sth like [0.83, 0.17; 0], with many values are similar over many instances. Obviously this is due to the inner mechanisms of the decision tree and makes sense, however, I'd assume that a respective usage might cause problems in real world usages as you assume that e.g. he never performs a certain action instead of assuming it for a small %. I guess the underlying issue is that different actions and their mis-prediction have different costs associated with it. For example, assume we are thinking about 3betting QJs and trying to predict whether villain folds/calls/raises. If we predict a fold and get a call, this is not awefull as we still have equity. If we however face a 4b and need to fold, we give up our equity completely. Hence, I wonder whether current accuracy figures like %predicted_correctly or RMS are really useful in the poker domain or whether we need to tweak it a bit to capture the issue described above.
Furthermore, I wonder if anyone has compared more advanced techniques like DTs/ANNs with a very simple statistic-based approach based on HUD-stats. For instance, lets consider the 3-b case again where we predict how opponents react to our 3-b. Currently, I use the stats from HM/PT for this spot; given we dont have enough on a player, it gets weighted and merged with a "default opponent" which is the stats combined of all players. I wonder if there are experiences in terms of grade of improvement of advanced techniques compared to this simple one.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 12, 2012 12:59 am 
Offline
Level1 member
User avatar

Posts: 40
Favourite Bot: own
I finished the creation of an ANN using FANN. Here are the steps I performed:

1. Downloaded the data file of OP.
2. Converted the file into FANN format (and removed O4; only using O1-O3).
3. Split the file into a learn set and a test set. The learn set contains the first 2170 instances, the test set the remaining 790 instances
4. Used the learning set with the following parameters (this took about half an hour)
- 1 hidden layer, 20 nodes
- FANN_TRAIN_RPROP
- FANN_SIGMOID_SYMMETRIC fro both activation functions
- 100000 epochs
5. Stored the model and applied it to the test set.

Result:
Test Result MSE: 0.064978 (no %correct available)

I wonder why its so low, as it seems to calculate the right figure - according to fanns website:

If d is the desired output of an output neuron and y is the actual output of the neuron, the square error is (d - y) squared. If two output neurons exists, then the mean square error for these two neurons is the average of the two square errors.

When training with the fann_train_on_file function, an error value is printed. This error value is the mean square error for all the training data. Meaning that it is the average of all the square errors in each of the training pairs.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 12, 2012 9:04 am 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Zoobie wrote:
RMS should be something like :


Code:
double globalError = 0;
foreach (dataRow in spearsData)
{
   double[] output = network.Compute(input);
   for (i = 0; i < outputsCount; i++)
   {
      double e = output[i] - desired[i];
      globalError += e * e;
   }
}

double rms = Math.Sqrt(globalError / (dataLength * outputsCount));


So that would make your rms error 0.255. CART's is 0.38. I guess that might be possible. How many correct predictions do you make?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 12, 2012 5:58 pm 
Offline
Level1 member
User avatar

Posts: 40
Favourite Bot: own
I exported the test set results into excel and performed the following steps:
1. for each output: if output < 0, set output = 0
2. normalize each output: output = output / sum(outputs)
3. set correct classification: if max(output) == expected_output then 1 else 0
4. calculate %correctly classified and RMS according to the formula of spears

Results:
%correctly classified = 58,5%
RMS = 21,1%

Comparing this with the other results posted here, I see that the correctly classified figure is worse, while the RMS is better. I wonder how to interpret these results. In my understanding, we should focus on minimizing RMS. Lets assume we have a guy who randomly folds or raises 50% of the time. A simple prediction strategy that always predicts FOLD or RAISE will have 50% correctly classified instances. A perfect predictor, however, would also have the same accuracy in terms of correctly classified instances, but have a lower RMS. Furthermore, in spots where one action is dominant (e.g., folding in case of facing a raised and 3-bet pot instead of coldcalling/cold4betting), the %correctly classified is not very helpful as even the standard assumption of assuming always the dominant action takes place has a large number of correctly classified instances. Is my thought process on focussing on RMS wrong or do I miss something?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 12, 2012 6:26 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
There are lots of ways of measuring error, eg
Code:
Time taken to build model: 254.61 seconds

Correctly Classified Instances         610               60.6362 %
Incorrectly Classified Instances       396               39.3638 %
Kappa statistic                          0.151
Mean absolute error                      0.2708
Root mean squared error                  0.4825
Relative absolute error                 79.5008 %
Root relative squared error            117.1899 %
Total Number of Instances             1006     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.287     0.148      0.409     0.287     0.337      0.612    O3
                 0.788     0.677      0.677     0.788     0.729      0.576    O2
                 0.255     0.047      0.358     0.255     0.298      0.677    O1
Weighted Avg.    0.606     0.479      0.577     0.606     0.585      0.595

=== Confusion Matrix ===

   a   b   c   <-- classified as
  76 181   8 |   a = O3
102 510  35 |   b = O2
   8  62  24 |   c = O1


And I think there could be more than one way of calculating RMS error, so we need to be careful comparing these numbers. I haven't time right now, but I will have a think about whether or not RMS error is the "best" measure.

Quote:
Lets assume we have a guy who randomly folds or raises 50% of the time. A simple prediction strategy that always predicts FOLD or RAISE will have 50% correctly classified instances. A perfect predictor, however, would also have the same accuracy in terms of correctly classified instances, but have a lower RMS.
Why?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Oct 12, 2012 7:00 pm 
Offline
Level1 member
User avatar

Posts: 40
Favourite Bot: own
Lets assume we have our ALWAYS_FOLD predictor and the OPTIMAL_PREDICTOR. Our test set looks as follows:

instance | result_F | result C |result R |
1 | 1 | 0 | 0
2 | 0 | 0 | 1

ALWAYS FOLD RMS:
Instance | prediction_F | prediction_C | prediction R | Error^2
1 | 1 | 0 | 0 | 0
2 | 1 | 0 | 0 | 3
RMS = 3/6 = 0.5

OPTIMAL RMS:
Instance | prediction_F | prediction_C | prediction R | Error^2
1 | 0.5 | 0 | 0.5 | 0.5
2 | 0.5 | 0 | 0.5 | 0.5
RMS = 1/6 = ,167


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Oct 16, 2012 8:39 pm 
Offline
Senior member
User avatar

Posts: 100
Favourite Bot: LGC
These good FANN results are interesting since I never got RMS < 38%
(%correct is a hopeless measure of accuracy).

@proud2bBot I'm trying to reproduce your RMS of around 20-25%, but
don't get anyway near this figure and suspect you've done something
wrong. Note that FANN doesn't report the MSE correctly if the output
activation function is 'symmetric' (this is a silly choice for this
problem, but for consistency with what you did I'll use that here,
together with your output normalization). You didn't mention any form
of input scaling, so I'll assume you used unscaled inputs.

Here's some code to train the net and compute the RMS.

Code:
#include <vector>
#include "fann.h"

using std::vector;
#define SQR(x) (x)*(x)

int main(int argc, char** argv)
{
    int   nlayers = 3;
    int   nhidden = 20;
    int   ninputs = 43;
    int   noutputs = 3;
    int   ntest_data = 790;
    int   max_epochs = 100000;
    float desired_error = 0;
    int   epochs_between_reports = 1000;
    struct fann_train_data* td;
    struct fann* ann;
   
    /*
     *  Train FANN.
     */
    ann = fann_create_standard(nlayers, ninputs, nhidden, noutputs);
    fann_set_activation_function_hidden(ann, FANN_SIGMOID_SYMMETRIC);
    fann_set_activation_function_output(ann, FANN_SIGMOID_SYMMETRIC);
    fann_set_training_algorithm(ann, FANN_TRAIN_RPROP);
    srand(24525);  // NB: FANN compiled with FANN_NO_SEED defined
    fann_train_on_file(ann, "turn.train.fann", max_epochs,
                       epochs_between_reports, desired_error);
    fann_save(ann, "fann.net");

    /*
     *  Compute RMS.
     *  Read test data set.  outputs are in the last 3 columns.
     */
    vector<vector<float> > test(ntest_data, vector<float>(ninputs+noutputs));
    FILE* fp = fopen("turn.test.fann", "r");
    char buf[80];
    fgets(buf, sizeof(buf), fp); // discard header line
    for (int r=0; r<ntest_data; r++)
        for (int i=0; i<ninputs+noutputs; i++)
            fscanf(fp, "%f", &test[r][i]);

    /*
     *  Compute RMS.
     */
    double mse = 0.0;
    for (int r=0; r<ntest_data; r++) {
        fann_type* pred = fann_run(ann, &test[r][0]);

        double sum = 0.0;
        for (int o=0; o<noutputs; o++) {
            if (pred[o] < 0) pred[o] = 0;
            sum += pred[o];
        }
       
        for (int o=0; o<noutputs; o++)
            mse += SQR(pred[o]/sum - test[r][ninputs + o]);
    }
    printf("Correct RMS = %g\n", sqrt(mse/ntest_data/noutputs));
}


To setup the datafiles in the correct format from spears's data, and
to run the code we do this:

Code:
ntotal=2960
ntest=790
ntrain=$(( $ntotal - $ntest ))

tail -$ntotal turn | cut -f 1-46 -d, | sed 's/,/ /g' > tmp
echo $ntrain 43 3 > turn.train.fann
head -$ntrain tmp >> turn.train.fann

echo $ntest 43 3 > turn.test.fann
tail -$ntest tmp >> turn.test.fann

./fann-test


The output is:

Code:
Max epochs   100000. Desired error: 0.0000000000.
Epochs            1. Current error: 0.0821667239. Bit fail 2170.
Epochs         1000. Current error: 0.0228235591. Bit fail 230.
Epochs         2000. Current error: 0.0206252988. Bit fail 166.
Epochs         3000. Current error: 0.0196281169. Bit fail 152.
...
Epochs        96000. Current error: 0.0151933907. Bit fail 96.
Epochs        97000. Current error: 0.0151830530. Bit fail 96.
Epochs        98000. Current error: 0.0151746431. Bit fail 95.
Epochs        99000. Current error: 0.0151641155. Bit fail 96.
Epochs       100000. Current error: 0.0151461568. Bit fail 97.
Correct RMS = 0.440573


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 106 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next


Who is online

Users browsing this forum: No registered users and 14 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: