Image Image Image




Post new topic Reply to topic  [ 106 posts ]  Go to page 1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: Machine Learning Performance Comparison
PostPosted: Tue Dec 15, 2009 8:31 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Attached is a csv file of fold/call/bet decisions on the turn against 43 input parameters. How does your favourite scheme do on this data?

- I used 2/3 of the data and 1/3 to test.
- There are 4 outputs - three binary and one nominal - choose either the three binary or the one nominal
- I've tried out lots of schemes in Weka including neural networks, but the best results I've found come from CART and NaiveBayesTree

scheme = CART
runtime = 3.11 secs
rms error = 0.38
% correctly classified = 69.6%

confusion matrix
a b c <-- classified as
96 164 5 | a = O3
69 551 27 | b = O2
0 41 53 | c = O1

Some probability distributions
0.2 *0.798 0.003
0.14 *0.477 0.384
0.14 *0.477 0.384
0.051 0.333 *0.616
0.051 0.333 *0.616
0.14 *0.477 0.384
0.159 *0.676 0.165
0.2 *0.798 0.003
0.2 *0.798 0.003
0.159 *0.676 0.165
0.2 *0.798 0.003
0.159 *0.676 0.165
0.2 *0.798 0.003
0.2 *0.798 0.003
0.159 *0.676 0.165
0.2 *0.798 0.003
0.2 *0.798 0.003
0.051 0.333 *0.616
0.2 *0.798 0.003
*0.593 0.407 0
0.2 *0.798 0.003
0.2 *0.798 0.003
0.14 *0.477 0.384


Attachments:
turn.zip [49.94 KB]
Downloaded 263 times
Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 16, 2009 1:29 am 
Offline
Senior member
User avatar

Posts: 149
Favourite Bot: khan
Did you try libsvm? You'd have to transform the data to their format but it's not too hard.
The thing is that they have a tool grid.py or easy.py which tries to find the best parameters for the svm for a certain kernel function to achieve the best accuracy through cross validation.
And you don't have to do almost any of the work.

P.S. In a similar experiment with weka I had similar results with CART. Didn't use libsvm though.

_________________
/* no comment */


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 16, 2009 12:04 pm 
Offline
Senior member
User avatar

Posts: 477
Favourite Bot: Donbot
Analysis Services, 33% holdout, RMS, time to generate: a few seconds

DT
O1 0.132
O2 0.332
O3 0.266
O4 not generated

NN
O1 0.122
O2 0.313
O3 0.250
O4 0.329

O4 matrix
ActualValue PredictedValue Count
O1 O1 6
O2 O1 7
O3 O1 1
O1 O2 91
O2 O2 559
O3 O2 166
O1 O3 5
O2 O3 52
O3 O3 89


0

_________________
I'm doing my part


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 16, 2009 12:29 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Impressive. These numbers are RMS error for regression on O1 O2 O3 seperately?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 16, 2009 2:58 pm 
Offline
Senior member
User avatar

Posts: 477
Favourite Bot: Donbot
spears wrote:
Impressive. These numbers are RMS error for regression on O1 O2 O3 seperately?


Its regression, not sure what else it would be? Only O4 is comparable with your RMS since the O1,O2 and O3 models will have a lower RMS thanks to less alternatives (binary instead of 3 choices). The RMS is ofcourse on the holdout/test data. If you want to try NN again, here is my formula for number of hidden nodes:
3 * sqrt({the number of input nodes} * {the number of output nodes})

Analysis Services will normalize the input and uses a standard topology.

_________________
I'm doing my part


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 16, 2009 3:56 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
0svald wrote:

Its regression, not sure what else it would be?

Classification.

I didn't see your O4 numbers when I looked this morning.

0svald wrote:
If you want to try NN again, here is my formula for number of hidden nodes:
3 * sqrt({the number of input nodes} * {the number of output nodes})

Thanks. Just a single hidden layer? I don't hold much hope as Weka NN is weak (anagram). I was hoping somebody might put this data through Fann and Neat.


Oswalds results as confusion matrix
Code:
       O1  O2   O3
O1     6    91   5
O2     7    559  52
O3     1    166  89

67% correct


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 16, 2009 4:08 pm 
Offline
Senior member
User avatar

Posts: 477
Favourite Bot: Donbot
spears wrote:
I didn't see your O4 numbers when I looked this morning.


Uhm, I think I added them shortly after writing the reply, sorry about the confusion.

Just a single hidden layer yes. I dont think other NNs will produce much different results. Perhaps if some of the inputs are combined to create more helpful inputs you would see an improvement.

Also I want to add that with large amounts of data I have had better results with DT than with NN, so that is also a factor when deciding algorithm.

0

_________________
I'm doing my part


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Dec 21, 2009 1:27 pm 
Offline
Senior member
User avatar

Posts: 172
Location: France
Favourite Bot: Lucy Liubot
I tried with Weka and have got same result as you. After trying everything, SimpleCart works the best with this data.
In Weka Explorer, how do you choose multiple output ? It seems I can only use one output.
spears wrote:
Thanks. Just a single hidden layer? I don't hold much hope as Weka NN is weak (anagram). I was hoping somebody might put this data through Fann and Neat.
I gave it a try with FannExplorer and I've got strange result (too good to be true IMO):

Reported mean square error: 0.116

Inputs: 43
Hidden neurons: 34
Outputs: 3

All default setting except the outputs neurons that are set on sigmoid symmetric.

Without that I got results in the range of 0.44 rms.


Attachments:
File comment: Data in Fann format.
turn-fann.zip [65.37 KB]
Downloaded 78 times
Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Dec 21, 2009 2:39 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
iamnobody wrote:
In Weka Explorer, how do you choose multiple output ? It seems I can only use one output.

Not sure I understand the question. If your output is non numeric (like O4) the Weka NN will create an output for each value. You can see this in the GUI
iamnobody wrote:
I gave it a try with FannExplorer and I've got strange result (too good to be true IMO):
Reported mean square error: 0.116
Inputs: 43
Hidden neurons: 34
Outputs: 3

All default setting except the outputs neurons that are set on sigmoid symmetric.

- mean square or rms error?
- % correct?
- confusion matrix?
- Is it trained on 2/3 and tested on 1/3 or what?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Dec 21, 2009 3:21 pm 
Offline
Senior member
User avatar

Posts: 172
Location: France
Favourite Bot: Lucy Liubot
spears wrote:
Not sure I understand the question. If your output is non numeric (like O4) the Weka NN will create an output for each value. You can see this in the GUI
Yes that's what I used (O4). How do you do if you wants to use the O1,O2,O3 numeric ?
spears wrote:
- mean square or rms error?
- % correct?
FannExplorer say exactly that: "Currently reported mean square error 0.116..."
I'm a newbie to machine learning and don't know the difference between mean square or rms. I was supposing rms was an acronym of "reported mean square".
spears wrote:
- confusion matrix?
I didn't see that in FannExplorer.
spears wrote:
- Is it trained on 2/3 and tested on 1/3 or what?
Yes.

It should be easy to try it yourself if you want. FannExplorer is a GUI like Weka and I gave you the data in the Fann format in my last post.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Dec 22, 2009 8:45 am 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
61% best mean square accuracy (78% mean) for all data in NEAT. 58% (76%) average for the best species.


Last edited by c2008 on Tue Dec 22, 2009 9:23 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Dec 22, 2009 9:20 am 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
c2008 wrote:
61% best mean square accuracy for all data in NEAT. 58% average for the best species.


How do I compare mean square accuracy to rms error? What does 58% average mean?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Dec 22, 2009 9:31 am 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
spears wrote:
How do I compare mean square accuracy to rms error? What does 58% average mean?


I'm not sure how to calculate RMS error. The mean square error would be (1-SQRT(0.61))^2, or 0.0479.

There are multiple genomes (NNs) in a species, 58% is the average mean^2 accuracy for the best species.


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Dec 22, 2009 10:42 am 
Offline
Senior member
User avatar

Posts: 166
Favourite Bot: ZBot
RMS should be something like :

Code:
double globalError = 0;
foreach (dataRow in spearsData)
{
   double[] output = network.Compute(input);
   for (i = 0; i < outputsCount; i++)
   {
      double e = output[i] - desired[i];
      globalError += e * e;
   }
}

double rms = Math.Sqrt(globalError / (dataLength * outputsCount));


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Dec 22, 2009 5:21 pm 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
I'm getting about 38% RMS error for the entire set using NEAT.


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Dec 22, 2009 9:17 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
c2008 wrote:
I'm getting about 38% RMS error for the entire set using NEAT.

Same error as CART then. What does "entire set" mean? Do you train all data everything and not just, say, 2/3?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Dec 22, 2009 10:05 pm 
Offline
PokerAI fellow
User avatar

Posts: 1115
Favourite Bot: Johnny #5
Yes, I'm lazy. ;)


Top
 Profile  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 23, 2009 1:16 pm 
Offline
Senior member
User avatar

Posts: 172
Location: France
Favourite Bot: Lucy Liubot
Rapidminer is the Nuts :mrgreen: It's a little harder to understand than Weka but it's totally worth it.

I didn't found better scheme than CART but after some fiddling I got better results. I'm using a SimpleValidation operator with a split ratio of 0.6666.

I used YAGGA2 to find the best inputs and got 71.83% accuracy, confusion matrix:
True: O3 O2 O1
O3: 113 70 0
O2: 137 543 47
O1: 3 21 53

Then, I used a grid parameter optimization to find the best parameter for CART and got 73.66% accuracy, confusion matrix:
True: O3 O2 O1
O3: 113 53 0
O2: 135 549 35
O1: 5 32 65

A stacking learner operator may be useful to improve result further.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 23, 2009 2:49 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
iamnobody wrote:
Rapidminer is the Nuts :mrgreen: It's a little harder to understand than Weka but it's totally worth it.

I didn't found better scheme than CART but after some fiddling I got better results. I'm using a SimpleValidation operator with a split ratio of 0.6666.

I used YAGGA2 to find the best inputs and got 71.83% accuracy, confusion matrix:
True: O3 O2 O1
O3: 113 70 0
O2: 137 543 47
O1: 3 21 53

Then, I used a grid parameter optimization to find the best parameter for CART and got 73.66% accuracy, confusion matrix:
True: O3 O2 O1
O3: 113 53 0
O2: 135 549 35
O1: 5 32 65

A stacking learner operator may be useful to improve result further.


Cool. Best inputs means best attributes/columns? When I started looking at ML Weka was much better than RapidMiner(aka Yale) but it looks like things have changed.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Dec 23, 2009 3:07 pm 
Offline
Senior member
User avatar

Posts: 172
Location: France
Favourite Bot: Lucy Liubot
spears wrote:
Best inputs means best attributes/columns?
Yes. YAGGA2 is a GA to weight the attributes that are found to be the most useful. Here is what I got after 100 generation and a population size of 10:
Code:
<?xml version="1.0" encoding="windows-1252"?>
<attributeweights version="4.6">
    <weight name="I1" value="0.0"/>
    <weight name="I2" value="0.0"/>
    <weight name="I3" value="0.0"/>
    <weight name="I4" value="0.0"/>
    <weight name="I5" value="0.0"/>
    <weight name="I6" value="1.0"/>
    <weight name="I7" value="0.0"/>
    <weight name="I8" value="1.0"/>
    <weight name="I9" value="0.0"/>
    <weight name="I10" value="0.0"/>
    <weight name="I11" value="0.0"/>
    <weight name="I12" value="0.0"/>
    <weight name="I13" value="0.0"/>
    <weight name="I14" value="1.0"/>
    <weight name="I15" value="1.0"/>
    <weight name="I16" value="0.0"/>
    <weight name="I17" value="0.0"/>
    <weight name="I18" value="0.0"/>
    <weight name="I19" value="0.0"/>
    <weight name="I20" value="0.0"/>
    <weight name="I21" value="0.0"/>
    <weight name="I22" value="0.0"/>
    <weight name="I23" value="0.0"/>
    <weight name="I24" value="0.0"/>
    <weight name="I25" value="0.0"/>
    <weight name="I26" value="0.0"/>
    <weight name="I27" value="0.0"/>
    <weight name="I28" value="1.0"/>
    <weight name="I29" value="1.0"/>
    <weight name="I30" value="0.0"/>
    <weight name="I31" value="0.0"/>
    <weight name="I32" value="0.0"/>
    <weight name="I33" value="0.0"/>
    <weight name="I34" value="0.0"/>
    <weight name="I35" value="1.0"/>
    <weight name="I36" value="0.0"/>
    <weight name="I37" value="0.0"/>
    <weight name="I38" value="0.0"/>
    <weight name="I39" value="0.0"/>
    <weight name="I40" value="0.0"/>
    <weight name="I41" value="0.0"/>
    <weight name="I42" value="0.0"/>
    <weight name="I43" value="0.0"/>
</attributeweights>


Would you disclose what theses attributes are ?


Top
 Profile E-mail  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 106 posts ]  Go to page 1, 2, 3, 4, 5, 6  Next


Who is online

Users browsing this forum: No registered users and 14 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: