Poker-AI.org

Hi.

Suppose you have something, some black box, which takes opponent's betting history, stats, any other info available and outputs a set possible opponent's hole cards. Suppose also you have showdown information.

How do you estimate the performance of this black box, to compare one against another?

Two obvious things are coming to mind:
1) opp's hand should be into the predicted range
2) the narrower the predicted range, the higher the score.

Any other ideas?

_________________
Бетономешалка! Мешает бетон!

Start by throwing X amount of hands into the black box to build a profile.

Then give the black box points depending on the certainty it predicts the right hand.
Lets say the black box narrow the possible hands down to: AA (30%), AK (45%), and KK (25%).
If the hand turns out to be AK then the black box gets 0.45 points.

The points could either be collected at each street or at the end of the hand.

Evaluate the black box over 1000 hands and add the points together. Make some changes to the black box and compare it's new score with the old score.

Ок, I got it.
And if the prediction is wrong? It can be zero points or some negative score. If negative, how much?

I also think about giving more points for right guessing in big pots and less for the small ones...

_________________
Бетономешалка! Мешает бетон!

You don't need to give it negative points.
Basically you're "range" needs to define probabilities for any possible holding. For everything that's not in the range it would therefore have 0% probability.
By probability make sure to define it as the probability that the opponent holds that hand, not from the player's perspective how often he would want to play it (so in the above example, the player might choose to ALWAYS play AA but overall the likelihood of opp holding AA is not 100%).

This has the benefit of assigning 0 points to wrong predictions and assigning higher points to narrower ranges (as the probability for each hand increases as the number of possible hands decreases). Should work pretty well out of the box.

_________________
Cheers.

If you give negative score when predicting wrong then you should think about how you handle the low likelihood predictions.

When you completely remove a hand from the range of you opponent then that is the same as giving it a 0% chance. If that results in a negative score then the 1-5% chances should be handled in a similair way.

If you add pot sizes to the score equation then remember that the pot sizes highly depends on which players are playing. If you score higher points for the right predictions in big pots then you might end up with a black box able to predict the hands well against loose agressive players (who tend to play bigger pots more often) and bad against everyone else. I don't know if this is good thing or bad thing.

You probably don't want to do it that way. That would be equivalent to minimizing the absolute error, which means the maximum likelihood model would perform better than the true model in some cases. For example, consider a situation where given your features the opponent has AA (51%) and KK (49%). If you use your method, then the model that predicts AA (100%) would receive an expected score of 0.51. However, the model that predicts the distribution perfectly would receive 0.51*0.51 + 0.49*0.49 = 0.2601 + 0.2401 = 0.5002.

Instead, you should use the squared error. That way, if your model predicts AA (100%), it would receive the score of 0.51*(0*0) + 0.49*(-1*-1) = 0.49. The true model would get 0.51*(-0.49*-0.49) + 0.49*(-0.51*-0.51) = 0.122451 + 0.127449 = 0.2499. You then evaluate the model on its ability to minimize the mean squared error.

That assumes of course that you have uniform samples, or at least enough samples that the bias is small. If you have a biased dataset that you use for evaluation, then the MSE will be biased as well. Since you're assuming you have perfect information, you can just sample uniformly over the holecards and that should be an unbiased sample.

Agree with longshot. In my book it's called the quadratic loss function. There is also the informational loss function.

I'm wondering about multiplying by the winnings/losses at showdown. An error associated with a win is something you want to reward, and an error associated with a loss is something you want to punish. I think this is called cost sensitive classification. There is something called cost sensitive learning too

That wasn't really in the scope of his system though, was it?

I suppose you really aren't interested in whether it's an error that's associated with a win or loss, so much as an error relative to the payoffs of the true model. So if you are against 3 possible hands each with some expected payoff for you: {H1 (50%, -5), H2 (10%, -5), H3 (40%, 0)], and your opponent model predicts incorrectly for H1 and H2, but perfectly for H3, e.g., {H1 (30%, -5), H2 (30%, -5), H3 (40%, 0)] then it doesn't matter at all since H1 and H2 are effectively the same strength. So being even 1% off in H3 is worse than 49% in H1 if the 49% is moved to H2.

All of this assumes you have some payoff function that can look ahead into the future and say, given complete info of the game, this is what you'll win. You could use HS or W% but I think those are going to be pretty poor predictors of the true payoffs of the hand given the opponent actions.

It's an interesting point though. It seems intuitive that you want to be more sensitive to error in certain hands.

You are right. I somehow neglected that a certain betting pattern could result in a range of hands rather than just one specific hand.

I'm assuming that we have the true payoff, just like we have the true showdown cards.

As I understand it you propose to compare
predicted cards with actual cards weighted with predicted payoff

Whereas I propose to compare
predicted cards with actual cards weighted with actual payoff

Oh, I see. So you just want to use the payoff as the error. Seems reasonable, and you could still just do mean squared payoff error as the loss function.

I'm a little curious though about the credit assignment problem. If you have a hand history with N opponent actions, do you break those N actions into N samples and give them equal weight in your evaluation? Seems like what you might want to do is randomly sample a hand history, then randomly sample a single action point to prevent bias towards showdowns.

No. I'm proposing that the error is calculated from the actual cards and the predicted probabilities, but it is somehow (I haven't figured out the details) weighted by the winnings/losses actually incurred.



You completely lost me. Have you changed the subject from testing hole card predictors?

Initially, I proposed to use pot size as weight. Not winning or losses - if we lost, but predictor worked well it is still good predictor.

_________________
Бетономешалка! Мешает бетон!

Yes, I know, and I've credited you above. And on reflection, I think it's probably a better idea than winnings/losses.