Image Image Image




Post new topic Reply to topic  [ 106 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6
Author Message
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Sat Feb 02, 2013 3:16 am 
Offline
Level1 member
User avatar

Posts: 46
Favourite Bot: IvoAvido
I teste on a dataset of 16000 rows (flop situations).
I could verify that a large dataset like mine is less prone to overfitting. Actually in this case i could leave the NN training run for ever without ovefitting.
I got about 75% accuracy with both svm and NN but imho we should not look at the accuracy, the important factor is the probability estimation of the actions. With svm i got a better rms error but....it's about 100 times slower than NN for predictions because of high number of support vectors, the fact that we need to calculate probabilities and the fact that the problem is multi-class. So imho svm is not an option.

For NNs i used FANN with a c# written by me.
For svm i used libsvm

If someone wantso to compare with my results i can post my dataset.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Sat Feb 02, 2013 9:50 am 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
I'll have a go at your dataset if you post it. Hold a third back so we can use it to check on cheats.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Feb 05, 2013 5:56 pm 
Offline
Level1 member
User avatar

Posts: 46
Favourite Bot: IvoAvido
Here it is. The inputs are all normalized in the 0,1 range.
By the way it actually starts to overfit after about 2k epochs using NN.


Attachments:
flopDataset.zip [311.66 KB]
Downloaded 16 times
Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Feb 05, 2013 7:32 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Code:
scheme                 attributes   %correct   rms error   runtime (s)
simple cart                          all   74.6            0.3541           10
nb tree                           all   74.3            0.375           18
random forest - 10 trees   all   78.2            0.328            3
random forest - 50 trees   all   80.2            0.313           13
random forest - 100 trees   all   80.3            0.311           28
random forest - 200 trees   all    80.5            0.31                   59


Correctly Classified Instances        3330               80.532  %
Incorrectly Classified Instances       805               19.468  %
Kappa statistic                          0.6668
Mean absolute error                      0.2012
Root mean squared error                  0.3097
Relative absolute error                 49.3194 %
Root relative squared error             68.7839 %
Total Number of Instances             4135     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.772     0.137      0.773     0.772     0.772      0.893    C
                 0.977     0.158      0.85      0.977     0.909      0.966    F
                 0.325     0.032      0.634     0.325     0.43       0.85     R
Weighted Avg.    0.805     0.132      0.79      0.805     0.788      0.922

=== Confusion Matrix ===

    a    b    c   <-- classified as
1202  248  108 |    a = C
   42 1934    4 |    b = F
  310   93  194 |    c = R



66% train 34% test

But it looks to me that
playerisSB == playerisBB

and these attributes are always zero
TotalInvestedPreflop
RemainingPlayersFlop
AggInvested1
PassInvested1
BetsToCall
RemainingPlayers
PairRank1

Haven't found a better subset of attributes.

@franconero What rms error did you get?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Feb 07, 2013 3:51 am 
Offline
Level1 member
User avatar

Posts: 46
Favourite Bot: IvoAvido
Thanks for pointing out the bugs. I solved all except BetsToCall because i didn't heve time for that.
With FANN, 19 neurons in hidden layer, train RPROP , i get 0.355 with the bugged dataset, 0.351 with the fixed Dataset.
The dataset is created from the history of only one player.

Do you perform any parameter optimization for the random forests ?


Attachments:
flopDataset.zip [394.21 KB]
Downloaded 10 times
Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Feb 07, 2013 8:03 am 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Franconero wrote:
Do you perform any parameter optimization for the random forests ?

I used Weka, and the only parameter is number trees, which I roughly optimised. I did a few experiments with attribute selection and dimensionality reduction but could not find an improvement, but maybe more work will yield something.


Top
 Profile E-mail  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 106 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6


Who is online

Users browsing this forum: No registered users and 14 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: