Image Image Image




Post new topic Reply to topic  [ 106 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Nov 02, 2010 10:46 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Quote:
There are 4 outputs - three binary and one nominal - choose either the three binary or the one nominal

Did you leave O1, O2, O3 in and use them as inputs?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Nov 02, 2010 10:54 pm 
Offline
Level1 member
User avatar

Posts: 42
Favourite Bot: ZZ'
spears wrote:
Quote:
There are 4 outputs - three binary and one nominal - choose either the three binary or the one nominal

Did you leave O1, O2, O3 in and use them as inputs?


I used one nominal output and left out others...


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Nov 02, 2010 11:11 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
OK, I give up. How did you do it?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Nov 02, 2010 11:15 pm 
Offline
Senior member
User avatar

Posts: 125
Favourite Bot: wetbot
Quote:
Attached is a csv file of fold/call/bet decisions on the turn against 43 input parameters.

Are we predicting the action of an opponent here? If so, shouldn't you be most interested in a probability distribution of f/c/r rather than a classification system? If not, what methodology was used to determine which of f/c/r was "correct"?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Nov 02, 2010 11:34 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
wetbot wrote:
Are we predicting the action of an opponent here?

Yes
wetbot wrote:
If so, shouldn't you be most interested in a probability distribution of f/c/r rather than a classification system?

Yes

Weka outputs probabilities which seem OK for some classifiers but not others. I'm interested how you would get probabilities.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Nov 02, 2010 11:40 pm 
Offline
Regular member
User avatar

Posts: 84
Favourite Bot: myself
spears wrote:
Did you leave O1, O2, O3 in and use them as inputs?

:lol: In this case he should get 100%.
I already tried about 20-30 different parameters, but the best result is about 66%.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Wed Nov 03, 2010 12:57 am 
Offline
Senior member
User avatar

Posts: 125
Favourite Bot: wetbot
Quote:
wetbot wrote:
Are we predicting the action of an opponent here?

Yes
wetbot wrote:
If so, shouldn't you be most interested in a probability distribution of f/c/r rather than a classification system?

Yes


In that case, (minimizing) RMS error is going to tell you everything you want to know re whether a particular system is good or not. Unfortunately, people have been focusing on confusion matrices and "percent correctly classified" as indicative of the quality of a proposed predictive system, and some have not even reported RMS error at all. Given that, any system that outputs "1/0/0", "0/1/0", or "0/0/1" is just not going to be competitive with a decent system that reports a probability distribution. [As result, I no longer believe SVM would be appropriate for this task.]

Quote:
I'm interested how you would get probabilities.


I would start with multinomial logistic regression, which Panik also suggested in a post on 2/20/10. Unfortunately, he did not report RMS error for his result. Also, I assume he reported the highest percentage option as the result of the regression for his confusion matrix, though he doesn't say for sure. If he's still around, perhaps he can report back.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Nov 04, 2010 12:45 pm 
Offline
Level1 member
User avatar

Posts: 42
Favourite Bot: ZZ'
I was experimenting with my own NN... splitting the set 2000 for training and rest for testing with single output... based on the neuron numbers 76% accuracy with 4.7112 training time or cost of doubling the training time 81% accuracy has been achieved. The standard deviation of testing accuracy was 0.0080. I think this could be quite reasonable.... (this can stay here was moved to the alternative topic :xx19 )


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Nov 04, 2010 10:49 pm 
Offline
Regular member
User avatar

Posts: 64
Location: Edinburgh, Scotland flounderhead@gmail.com
Favourite Bot: Kryten
Gave it a quick try with SVM, but only get 64-65% accuracy on a 1/3 test set. Used libsvm, RBF kernel with different values for C and gamma. Also tried both scaling the attributes to [-1 1] and normalising to mean=0, std = 1 before training the SVM, but no difference.

In fact, 64% is about the same result I get when I simply don't use the input attributes at all :D
So any method that gives less than 70% accuracy only marginally improves upon rolling a weighted dice.
Well, I might be doing something wrong of course...


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Sun Jan 30, 2011 11:52 pm 
Offline
Senior member
User avatar

Posts: 356
Favourite Bot: OpenHoldem
wetbot wrote:
I would start with multinomial logistic regression, which Panik also suggested in a post on 2/20/10. Unfortunately, he did not report RMS error for his result. Also, I assume he reported the highest percentage option as the result of the regression for his confusion matrix, though he doesn't say for sure. If he's still around, perhaps he can report back.


I'm still around. I'll report RMSE as soon as I get a chance.

/edit: RMSE=.38389934. O1, O2, O3 are zero or 1, representing the outcomes 1,2,3. pO1, pO2, pO3 are the calculated probabilities of the outcome from my multinomial logit. n is the number of observations. I calculated MSE for each outcome (e.g MSE_O1=sum((O1-pO1)^2)/n), and found RMSE=sqrt((MSE_O1+MSE_O2+MSE_O3)/3).

z.z. wrote:
Correctly Classified Instances 1001 99.503 %
Incorrectly Classified Instances 5 0.497 %
=== Confusion Matrix ===

a b c <-- classified as
262 3 0 | a = O3
0 647 0 | b = O2
0 2 92 | c = O1


Please post your code.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Jan 31, 2011 1:14 am 
Offline
Senior member
User avatar

Posts: 356
Favourite Bot: OpenHoldem
Could we start a new thread for round 2 of this comparison? I'd like to propose some new criteria in addition to the runtime, RMSE, % correct, and confusion matrices we're currently using:

1. Attach your code, or directions so that your results are reproducible, as a text file. Different programs have different implementations of CART, neural networks, or regression.
2. Use leave-one-out cross-validation as a metric of accuracy. This takes a lot longer to run, but it means that we're all using the same data for training and testing. Here's an example of leave-one-out cross-validation, coded in R, which seems to be the place where all the cutting edge statistics and machine learning research happens:

Code:
library('tree')
turn<-read.csv("turn.csv")
dat<-turn[,-44:-46] #Remove extra response variables
output <- rep(NA, nrow(dat))
for (i in 1:nrow(dat)){  #Loop through each observation in the dataset
   testset <- dat[i,]
   trainset <- dat[-i,]  #Train on the whole dataset, EXCLUDING the current observation
   model <- tree(O4~.,data=trainset)
   output[i] <- as.character(predict(model, newdata=testset, type="class"))
}
output(tree, dat$O4) #Tabulate results
       O1   O2   O3
  O1  159   83   14
  O2  141 1611  461
  O3    0  208  283

% correct = 69.35810811

Leave-one-out cross-validation is a very strong way to measure the predictive power of a model, as well as to prevent over-fitting. I suggest we start using it to enforce some consistency in our results. To kick things off, I've already posted the results of this method for CART. I have results coming soon for SVMs, neural networks, linear discriminant analysis, and a few other techniques in R.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Jan 31, 2011 8:35 am 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Panik wrote:
Could we start a new thread for round 2 of this comparison?


Sure - go ahead:
- But I'm not convinced Leave One Out Cross Validation is actually practical for the longer run time schemes like Neural Networks. Maybe we should adopt your earlier suggestion and just decide on particular training and test sets.
- There have been some interesting points made about what constitutes the best measure of success. I guess we all agree that we want to generate a realistic probability distribution, but does rms error measure that? http://pokerai.org/pf3/viewtopic.php?f= ... ion#p37385, also ITT
- Maybe somebody would like to contribute some labelled data?


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Jan 31, 2011 2:29 pm 
Offline
Senior member
User avatar

Posts: 356
Favourite Bot: OpenHoldem
spears wrote:
Panik wrote:
Could we start a new thread for round 2 of this comparison?


Sure - go ahead:
- But I'm not convinced Leave One Out Cross Validation is actually practical for the longer run time schemes like Neural Networks. Maybe we should adopt your earlier suggestion and just decide on particular training and test sets.
- There have been some interesting points made about what constitutes the best measure of success. I guess we all agree that we want to generate a realistic probability distribution, but does rms error measure that? http://pokerai.org/pf3/viewtopic.php?f= ... ion#p37385, also ITT
- Maybe somebody would like to contribute some labelled data?


-I agree. Maybe we could do some kind of knn-cross validation, and make 10 or so "standard" 2/3-1/3 training and testing sets.
-I think we should keep using confusion matrices and % correct too, as some algorithms output probabilities and some don't.
-I was going to grab one of the datasets posted here, and make a table similar to yours. This data is labeled by opponent, which will add another layer of complexity.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Mon Jan 31, 2011 6:53 pm 
Offline
Level1 member
User avatar

Posts: 46
Favourite Bot: IvoAvido
Training: Epochs 778. Current Error: 0.133832
Testing on validation set.
RMS = 0.374508 MSE = 0.140934
Confusion Matrix
O1 O2 O3 <-- expected
O1 55 38 8
O2 40 525 140
O3 1 58 85
70% correct.

Using NN (FANN) . I created a custom network , 3 layers , the second layers has 3 groups (a group for each output) , each group of the second layer is completely connected to the input layer but each output neuron is connected to only one group of the second layer.
To calculate RMS, first i normlalize p1 p2 p3 so that p1+p2+p3 = 1 then RMS is the square root of the average quadratic error
foreach row
squareError = squareError + (p1-r1)^2 +(p2-r2)^2 +(p3-r3)^2
end for
RMS = sqrt( squareError/ (numRows * 3))


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Tue Feb 01, 2011 12:33 am 
Offline
Senior member
User avatar

Posts: 356
Favourite Bot: OpenHoldem
It seems that ~70% is the highest accuracy possible with this dataset.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Thu Mar 03, 2011 11:57 pm 
Offline
Senior member
User avatar

Posts: 356
Favourite Bot: OpenHoldem
Ok, I have one more algorythm to post: Random Forests. Random forests are an ensemble of Classification and Regression Trees, and are amazingly accurate and robust. Here's the confusion matrix of a random forest, fit on the entire turn dataset:

Code:
OOB estimate of  error rate: 30.57%
Confusion matrix:
    O1   O2  O3 class.error
O1 137  160   3   0.5433333
O2  72 1681 149   0.1161935
O3  13  508 237   0.6873351

% correct = 69.4%

Here's the code I used to run this:
Code:
library('randomForest')
turn<-read.csv("turn.csv")
dat<-turn[,-44:-46] #Remove extra response variables
dat<-dat[-20]
model <- randomForest(O4~.,data=dat,ntree=5000,importance=TRUE)
model
varImpPlot(model,type=1)


Here are the huge advantages of random forests:
1. Random forests literally cannot over fit a model to your data. This is a HUGE advantage over other machine learning algorythms and especially relevant in poker where we intrinsically have a biased sample.
2. Random forests do NOT need cross validation. This means you can fit to your entire dataset, without splitting it into training and testing datasets, and it will generate an unbiased estimate of your error rate.
3. Variable selection. Random forests can be used to rank the importance of each variable in your dataset, as shown in the attached plot. This means you can run the algorithm once, eliminate some variables, and run it again for better results
4. Random forests can be run on datasets with large numbers of variables
5. Random forests can also be used to cluster data in unsupervised mode.
6. Random forests are accurate for estimating missing data, like in the case where hands are folder before the river...
7. Very little to tune, and thus little to screw up.

I strongly suggest you all check this algorithm out. You can read more about its advantages on Cross Validated

Here's a great graph it produces which can help you assess which variables are important in a model. In this case, variables such as I14, I15 etc. are clearly important.
Image


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Mar 04, 2011 10:49 am 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
@panik Apart from all the advantages you mention, Random Forests have a few disadvantages too:
- In this case the answer isn't as good as CART
- The generated classifiers can get huge. I had to stop using Random Forests on one problem because of this.

I like the importance plot. The results are similar to the YAGGA filter above. If you exclude the less important attributes does the result improve? And rather than just exclude some them, is it possible to weight their usage according to their importance?

btw Leo Breiman invented both CART and RF. Loved this paper http://projecteuclid.org/DPubS/Reposito ... 1009213726


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Mar 04, 2011 6:59 pm 
Offline
Senior member
User avatar

Posts: 356
Favourite Bot: OpenHoldem
One other advantage I forgot to mention (since it’s true of most classifiers), is that random forests can also output probabilities of class membership, which, as discussed previously, is often more useful than a binary classification.

@spears: I did a leave-one-out cross validation run with CART in R, and here are my results:
Code:
rtree   O1   O2   O3
   O1  159   87   14
   O2  141 1607  431
   O3    0  208  313
% correct = 70.2364865

As you can see, the increase in accuracy over a random forest is less than 1%. However, as you mentioned, CART takes much less time to run and outputs a much smaller model, because you are only dealing with one tree, rather than 5000. Importantly, CART models are very easy to understand.

I also tried 2 different algorithms with random forests. First, I re-ran my original model in 2 stages, with some variable selection in between. Here is a chart of variable importance, take from 100 runs of a random forest:
Image
Here is the output from the second run of the random forest, where I removed the least important 17 variables.
Code:
    O1   O2  O3 class.error
O1 159  139   2   0.4700000
O2  86 1664 152   0.1251314
O3  13  500 245   0.6767810
%correct=69.848649

It seems that the variable selection did not signifigantly improve the accuracy of the random forest, which after some theoretical reading, makes sense.

The second algorithm I tried was the “cforest” algorithm in the r package “party.” This algorithm implements a forest of conditional classification and regression trees, which is an improvement on the canonical implementation of random forest. As you can see, this model also performs very well (but takes a long time to compute).
Code:
       O1   O2   O3
  O1    7  293    0
  O2    0 1826   76
  O3    0  503  255
%correct=70.5405405

I guess the moral of the story is that it’s good to have a large toolbox of algorithms when solving difficult problems. Thanks for the paper, it was an excellent read… should be required reading for everyone.


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Mar 04, 2011 8:31 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Panik wrote:
As you can see, the increase in accuracy over a random forest is less than 1%.
It doesn't sound a lot, but it really matters.

Panik wrote:
Importantly, CART models are very easy to understand.
Well that's what the books say. But I have yet to build one myself I could understand in any reasonable timeframe.

Panik wrote:
I guess the moral of the story is that it’s good to have a large toolbox of algorithms when solving difficult problems.
http://www.no-free-lunch.org


Top
 Profile E-mail  
 
 Post subject: Re: Machine Learning Performance Comparison
PostPosted: Fri Mar 04, 2011 10:04 pm 
Offline
Senior member
User avatar

Posts: 356
Favourite Bot: OpenHoldem
spears wrote:
It doesn't sound a lot, but it really matters.

In that case, conditional inference tree forests seem to slightly outperform CART

spears wrote:
Well that's what the books say. But I have yet to build one myself I could understand in any reasonable timeframe.

Here's the CART model that gives 70.24% accuracy:
Image
If you knew what the inputs were, this diagram could probably go a long way towards explaining how this player behaves on the turn. (Left branch = "yes", right branch = "no")

spears wrote:

Thanks for the link, that's another good read.


Top
 Profile E-mail  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 106 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next


Who is online

Users browsing this forum: No registered users and 14 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: