Poker-AI.org

Poker AI and Botting Discussion Forum
It is currently Mon Nov 13, 2023 5:42 pm

View unanswered posts | View active topics


Board index » Public Forums » AI Research

All times are UTC




Post new topic Reply to topic  Page 1 of 2
  [ 21 posts ]  Go to page 1, 2  Next
  Print view Previous topic | Next topic 
Author Message
corintio
 Post subject: My ANN AI strategy
PostPosted: Wed Oct 16, 2013 9:14 pm 
Offline
Regular Member
User avatar

Joined: Wed Oct 02, 2013 5:00 pm
Posts: 64
Hi All,

I'd like to outline here my AI implementation strategy, any feedback would be very welcome.

I'm basing my development on this paper: http://www2.cs.uregina.ca/~hilder/my_st ... Thesis.pdf

I'm developing it using OpenTestBed/Meerkat API. My approach is to use an ANN to actualy play poker (as opposed to just model opponents). Right now I'm collecting the following information from the game state as input to my ANN:

- Hand Strenght
- Hand Potential
- Amount to call
- Eligible pot
- My amount in pot
- # active players
- Stage (including pre-flop)
- My distance from button
- Chips (mine and from opponents)
- Some basic opponent modeling

and my ANN outputs the action that should be performed (fold, check/call, bet/raise)

As a proof of concept, I've trained my ANN with 10000 hands of FL-HU cash games played between two SimpleBots, with 5% error. Then I used a Genetic Algorithm to evolve this ANN so now it beats SimpleBot consistently.

My next step is to setup a tournament style game play so I can put all candidates in each generation of my GA to play against each other, hoping that it can evolve even further.

Also, I'm changing my ANN to play NoLimit games, and the output now is (fold, check/call, bet/raise low, medium, high, all-in)

Any comments?
Top
 Profile  
 
cantina
 Post subject: Re: My ANN AI strategy
PostPosted: Wed Oct 16, 2013 9:25 pm 
Offline
Veteran Member

Joined: Thu Feb 28, 2013 2:39 am
Posts: 437
When you can beat Fell Omen 2 by a reasonable margin then you'll have something. Simplebot is a very weak player. I've tried using Neuro-Evolution of Augmenting Topologies (NEAT) to build strategies, without avail. I'd bet any evolutionary approach you take will fail in performance.
Top
 Profile  
 
corintio
 Post subject: Re: My ANN AI strategy
PostPosted: Wed Oct 16, 2013 10:28 pm 
Offline
Regular Member
User avatar

Joined: Wed Oct 02, 2013 5:00 pm
Posts: 64
Yeah, I know it's really weak, but as I said, this was a proof of concept, and also helped me putting everything together (opentestbed, NN, evolutionary algorithm...)

I tried to use Fell Omem 2, but it gives me an error when running on opentestbed, an exception about strategy not found. Is it a known issue? Is it possible to run Fell Omen 2 with OTB? Otherwise, what do you guys use to compete against it?
Top
 Profile  
 
cantina
 Post subject: Re: My ANN AI strategy
PostPosted: Wed Oct 16, 2013 11:12 pm 
Offline
Veteran Member

Joined: Thu Feb 28, 2013 2:39 am
Posts: 437
Yeah, you need the strategy files for FO2 (a few hundred megs worth). My point is, you're likely wasting your time with evolution in HUL.
Top
 Profile  
 
corintio
 Post subject: Re: My ANN AI strategy
PostPosted: Wed Oct 16, 2013 11:35 pm 
Offline
Regular Member
User avatar

Joined: Wed Oct 02, 2013 5:00 pm
Posts: 64
My plan is to play HU NoLimit, and I'm too close to check if this approach (ANN/Evolution) is valid or not. Nevertheless, thanks for the advice.

But whatever approach I use, I will need FO2 running with OTB. Can you help me with that?

I downloaded FO2 from this site: http://www.deducer.org/pmwiki/index.php ... ndFellOmen, and it came with a folder called Strategy_FellOmen_2 with 73MB in zip files. Is this the correct distribution?

The error I get is:

Code:
...
turn Hand Strength : 0.5497584541062802 turn potential: 0.08984263233190272 turn negative potential : 0.12560046865846514
action observed: Check
rollout Hand Strength: 0.5211594202898551 turn index: 84
turn file: p1_strategy83-10_5.txt
table : p1_strategy83-10_5.txt
trying Strategy_FellOmen_2/1.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/2.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/3.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/4.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/5.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/6.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/7.zip
reached end of zip file before table was found
Exception in thread "main" java.lang.NullPointerException
   at com.ibot.FellOmen_2_Impl.getActionTable(Unknown Source)
   at com.ibot.FellOmen_2_Impl.turnAction(Unknown Source)
   at FellOmen_2.turnAction(Unknown Source)
   at FellOmen_2.getAction(Unknown Source)
   at game.Dealer.playStage(Dealer.java:102)
   at game.Dealer.playHand(Dealer.java:66)
   at game.CashGameRunner.runGames(CashGameRunner.java:82)
   at game.CashGameRunner.runGame(CashGameRunner.java:46)
   at CashGameConsoleStarter.main(CashGameConsoleStarter.java:52)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)


Thanks,
Corintio
Top
 Profile  
 
cantina
 Post subject: Re: My ANN AI strategy
PostPosted: Thu Oct 17, 2013 12:06 am 
Offline
Veteran Member

Joined: Thu Feb 28, 2013 2:39 am
Posts: 437
I tried HUNL with NEAT too. You'll likely end up evolving push/fold players, due to the overwhelming effect of variance (push/fold players have the highest earning potential when luck is a factor). Just trying to save you months or years of wasted effort.

I remember the FO2 zip(s) being bigger, but I could be mistaken. I also don't remember dashes in the file names. You might look in the zips and see if something isn't amiss. If it can't find the strategy file it's either:
a) Incorrect encoding of the game state (i.e. filename).
b) The file is missing from the zip(s).
Top
 Profile  
 
spears
 Post subject: Re: My ANN AI strategy
PostPosted: Thu Oct 17, 2013 7:45 am 
Offline
Site Admin
User avatar

Joined: Sun Feb 24, 2013 9:39 pm
Posts: 642
I don't remember reading that paper before but it is quite interesting. I liked the phantom parasite idea.

I don't want to pour cold water on your project but I'm pessimistic. I only skimmed the paper so I may have missed something.
- FOLDER beats all evolved strategies :(
- 500 hand games are nowhere long enough. My reinforcement learning algorithm was still learning at 20,000 hands
- Fictitious play, CFRM, Nicolai's method all have the same fundamental idea in common.
- FP & CFRM model many more game states than Nicolai's method and are hence potentially more accurate
- FP & CFRM "know" villain's strategy and can therefore exploit it quickly. I don't think Nicolai's method "knows" villain's strategy.
- I'm not sure but I think Nicolai's method creates a pure rather than a mixed strategy. This is exploitable.
- Incremental Learning in ANNs is hard to get right
- ANNs are quite slow compared to the simple histogram approach used in CFRM. Tile coding might improve the histogram approach without the cost of ANNs

On the other hand, Dahl seems to have got something working with ANNs. http://www.nytimes.com/2013/09/08/magaz ... .html?_r=0

Hope this helps.
Top
 Profile  
 
spears
 Post subject: Re: My ANN AI strategy
PostPosted: Thu Oct 17, 2013 9:50 am 
Offline
Site Admin
User avatar

Joined: Sun Feb 24, 2013 9:39 pm
Posts: 642
Maybe this will help http://poker-ai.org/archive/www.pokerai ... a&start=60
Top
 Profile  
 
cantina
 Post subject: Re: My ANN AI strategy
PostPosted: Thu Oct 17, 2013 4:29 pm 
Offline
Veteran Member

Joined: Thu Feb 28, 2013 2:39 am
Posts: 437
spears wrote:
On the other hand, Dahl seems to have got something working with ANNs. http://www.nytimes.com/2013/09/08/magaz ... .html?_r=0

Touche. I'd like to hear more about his methods, if they're published anywhere. I'd also like to see any benchmarks he may have against known players. It's kind of wild he started in '97. :)
Top
 Profile  
 
spears
 Post subject: Re: My ANN AI strategy
PostPosted: Thu Oct 17, 2013 5:06 pm 
Offline
Site Admin
User avatar

Joined: Sun Feb 24, 2013 9:39 pm
Posts: 642
Quote:
It's kind of wild he started in '97.
At backgammon

A reinforcement learning algorithm applied to simplified two-player Texas Hold’em poker (Dahl, 2001)

http://www.ke.tu-darmstadt.de/lehre/arc ... istoph.pdf
http://www.google.com/patents/US20100069134

I have a paper copy somewhere. I don't remember being very impressed when I read it, but maybe I missed something. It was early in my poker life.
Top
 Profile  
 
corintio
 Post subject: Re: My ANN AI strategy
PostPosted: Fri Oct 18, 2013 1:28 am 
Offline
Regular Member
User avatar

Joined: Wed Oct 02, 2013 5:00 pm
Posts: 64
Thanks for the replies!

spears wrote:

On the other hand, Dahl seems to have got something working with ANNs. http://www.nytimes.com/2013/09/08/magaz ... .html?_r=0

I've read that article before. That was my inspiration... :)

I'm planning to use way more than 500 hands. Also, my plan is to create more than one ANN, tuned for different scenarios, maybe more than one for the same scenario. That would be my attempt to make it less exploitable. I'm also planning to do a better opponent modeling than Nicolai's implementation.

spears wrote:

- FOLDER beats all evolved strategies :(

I also didn't get it! Will try to see what happens in my simulations.

Nasher wrote:
You'll likely end up evolving push/fold players, due to the overwhelming effect of variance (push/fold players have the highest earning potential when luck is a factor).

To avoid this, I'm planning to experiment with different Fitness functions, giving different weights for different aspects of the gameplay (not only ranking and bankroll).

Well, let's see what lessons I can get from all of this :)

Thanks,
Corintio.
Top
 Profile  
 
corintio
 Post subject: Re: My ANN AI strategy
PostPosted: Fri Oct 18, 2013 1:32 am 
Offline
Regular Member
User avatar

Joined: Wed Oct 02, 2013 5:00 pm
Posts: 64
spears wrote:
Maybe this will help http://poker-ai.org/archive/www.pokerai ... a&start=60

Thanks, but I've read this whole thread before, a few times, and checked the sources for every code snippet: it seems that all of those issues are fixed in the source code (at least the ones related to FO2....)

Will try again latter. Right now I'm focusing on implementing support for Tournaments in opentestbed.

Thanks,
Corintio
Top
 Profile  
 
cantina
 Post subject: Re: My ANN AI strategy
PostPosted: Sun Nov 03, 2013 12:18 pm 
Offline
Veteran Member

Joined: Thu Feb 28, 2013 2:39 am
Posts: 437
spears wrote:
A reinforcement learning algorithm applied to simplified two-player Texas Hold’em poker (Dahl, 2001)

http://www.ke.tu-darmstadt.de/lehre/arc ... istoph.pdf
http://www.google.com/patents/US20100069134

I have a paper copy somewhere. I don't remember being very impressed when I read it, but maybe I missed something. It was early in my poker life.

I remember reading that some time ago and not understanding how it would be used. Do you think that's what he's using currently to train his NNs?
Top
 Profile  
 
spears
 Post subject: Re: My ANN AI strategy
PostPosted: Sun Nov 03, 2013 9:03 pm 
Offline
Site Admin
User avatar

Joined: Sun Feb 24, 2013 9:39 pm
Posts: 642
It's just a guess that seems to fit the available info. You know at least as much as I do now.
Top
 Profile  
 
Nose
 Post subject: Re: My ANN AI strategy
PostPosted: Tue Nov 05, 2013 3:45 am 
Offline
Regular Member
User avatar

Joined: Sat May 25, 2013 7:36 am
Posts: 73
Nasher wrote:
spears wrote:
A reinforcement learning algorithm applied to simplified two-player Texas Hold’em poker (Dahl, 2001)

http://www.ke.tu-darmstadt.de/lehre/arc ... istoph.pdf
http://www.google.com/patents/US20100069134

I have a paper copy somewhere. I don't remember being very impressed when I read it, but maybe I missed something. It was early in my poker life.

I remember reading that some time ago and not understanding how it would be used. Do you think that's what he's using currently to train his NNs?


given a few insights I would suggest you to not waste too much time on figuring out what Bastian thinks or does. a few years ago we worked at the same company as software developers. his first days/ weeks (which where intended for him to get an overview of the project's code he was assigned to) he spent "beautifying" the source code, meaning the replaced comment-separator-lines from

Code:
// ########################################################


to

Code:
///////////////////////////////////////////////////////////////////////////////////////////////////////


because that "allowed" him to understand the code quicker ...
Top
 Profile  
 
d4n13l
 Post subject: Re: My ANN AI strategy
PostPosted: Fri Nov 08, 2013 10:31 am 
Offline
Junior Member

Joined: Thu Nov 07, 2013 4:24 pm
Posts: 12
I am new to the forum, and might be missing some theoretical background... so treat my idea with caution. ;)

Dahl's machine sounds exciting, and it seems ANN could be a very effective technique after all. A good selection on input is probably crucial here, and yours seems like a good start. It is however unclear to me what exactly you mean with "stage". Is it purely the current stage of the game (preflop, flop, turn, river)? If so, then maybe including somehting like "betting sequence up to now" could also be of interest, if that is not inherent to any of your inputs yet.
This is where often where opponents weakness or strengthis revealed imo. Have you thought/what is the scientific community saying about that?
Top
 Profile  
 
spears
 Post subject: Re: My ANN AI strategy
PostPosted: Fri Nov 08, 2013 11:29 am 
Offline
Site Admin
User avatar

Joined: Sun Feb 24, 2013 9:39 pm
Posts: 642
Yeah, stage means preflop, flop, turn, river.

The problem with using betting sequence is that that there are so many of them it's slow/difficult to learn from them. I guess one approach would be to try to define some similarity metric over betting sequences, which I think is what the SARTRE guys did.

Another problem I see with this whole ANN/Reinforcement Learning approach is that what you really need is a frequency for each action given the current game state. Most RL approaches just compute an optimum action. Getting an optimum action and then "mixing it up a bit" just don't cut it.

And another thing - why is everyone so taken by ANNs? They aren't magic. There are lots of ML techniques out there and some are quite a lot faster than ANNs. Sutton & Barto suggest using tile coding for problems like this. They are just a cheap and cheerful approach to locally weighted learning, and work well in incremental learning.
Top
 Profile  
 
d4n13l
 Post subject: Re: My ANN AI strategy
PostPosted: Fri Nov 08, 2013 11:49 am 
Offline
Junior Member

Joined: Thu Nov 07, 2013 4:24 pm
Posts: 12
possible ANN attraction point: maybe simply because they are rather simply to grasp, and the concept seems convincing: get a number of input variables, have some computation, and map it onto several outputs.

And its pretty close to what most people do when playing poker, too. "Ok, I'm in that position, with that blind/stack ratio, no action before me, .... so I do that." --> several inputs to come up the the desired action.
Top
 Profile  
 
corintio
 Post subject: Re: My ANN AI strategy
PostPosted: Sat Nov 09, 2013 2:23 am 
Offline
Regular Member
User avatar

Joined: Wed Oct 02, 2013 5:00 pm
Posts: 64
d4n13l wrote:
And its pretty close to what most people do when playing poker, too. "Ok, I'm in that position, with that blind/stack ratio, no action before me, .... so I do that." --> several inputs to come up the the desired action.



Yeah, the problem is training your ANN... You'll need hand histories (lots of them) from players you want to mimic (hard to get), or try to "grow" your own ANNs using some evolutionary algorithm - hard to get it right, or too slow (months, maybe years...)...

With that said, I will continue pursuing this path in the (near) future, after I get my scraper up and running.
Top
 Profile  
 
d4n13l
 Post subject: Re: My ANN AI strategy
PostPosted: Sat Nov 09, 2013 12:18 pm 
Offline
Junior Member

Joined: Thu Nov 07, 2013 4:24 pm
Posts: 12
using an available scraper such as the OH framework offers does not cut it?

I suppose you have treid that - may I ask wyh you are working on your own scraper now?
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 2
  [ 21 posts ]  Go to page 1, 2  Next

Board index » Public Forums » AI Research

All times are UTC


Who is online

Users browsing this forum: No registered users and 2 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Group