Poker-AI.org

Has anyone figured out some objective way to measure the relative playing strength of bots? The ELO rating system works pretty well for games like chess and table tennis, but it doesn't map onto poker very well for obvious reasons. ROI (total bets / total winnings) is another simple measure, but I see problems there, too. Any research papers on this topic?

I haven't come across much about it. Most of the research is centred around NE bots and the rating is the win rate. They typically play zillions of hands, duplicate, to get statistically valid results, and even then quote error margins.

Personally, I think poker research would be much more interesting and relevant if the competitions were run like skillbet http://en.wikipedia.org/wiki/Duplicate_ ... lBet_poker. I suggested it to UoA and got blown off

It's a tricky problem, for sure. Win rate doesn't seem like a good rating for the simple reason that the goal of poker isn't to win a lot of hands, but rather to win the most money. Of course a lot would depend on event structure, i.e., freezeout tournament or ring game? heads-up or multiway? limit or no-limit? ... and so on.

Duplicate matches require memory wipes of all bots in between matches, right? But how do event organizers verify the wipe? Do they inspect the source code of each bot before and after each match? Do they use the honor system?

Makes sense. Any interest in organizing an alternate competition?

I meant the amount won per zillion hands.


I think they reload a VM image and the bot software from CD

Interested but have no time. Its other great advantage is that you don't have to hand over your code.

Exactly. Hmmm