Poker-AI.org

Re: My ANN AI strategy

2013-11-26T17:52:15+00:00

Quote:

using an available scraper such as the OH framework offers does not cut it?
I suppose you have treid that - may I ask wyh you are working on your own scraper now?

I'm reading a lot of OH code and documentation, but I'm not using it for 3 main reasons:

1) My bot will run on Mac (at least in the beginning), and if all goes well, I will port it to Linux (a cluster of Linux boxes). OH only runs on Windows....
2) I'm a full-time software developer, and my main motivation for creating the bot is learning new (or revisiting) technologies
3) Using my own proprietary scraper is way more secure (in terms of stealthiness) than using OH, which all casinos know about and are actively trying to prevent its usage.

Corintio

Statistics: Posted by corintio — Tue Nov 26, 2013 5:52 pm

Re: My ANN AI strategy

2013-11-09T12:18:01+00:00

using an available scraper such as the OH framework offers does not cut it?

I suppose you have treid that - may I ask wyh you are working on your own scraper now?

Statistics: Posted by d4n13l — Sat Nov 09, 2013 12:18 pm

Re: My ANN AI strategy

2013-11-09T02:23:22+00:00

d4n13l wrote:

And its pretty close to what most people do when playing poker, too. "Ok, I'm in that position, with that blind/stack ratio, no action before me, .... so I do that." --> several inputs to come up the the desired action.

Yeah, the problem is training your ANN... You'll need hand histories (lots of them) from players you want to mimic (hard to get), or try to "grow" your own ANNs using some evolutionary algorithm - hard to get it right, or too slow (months, maybe years...)...

With that said, I will continue pursuing this path in the (near) future, after I get my scraper up and running.

Statistics: Posted by corintio — Sat Nov 09, 2013 2:23 am

Re: My ANN AI strategy

2013-11-08T11:49:41+00:00

possible ANN attraction point: maybe simply because they are rather simply to grasp, and the concept seems convincing: get a number of input variables, have some computation, and map it onto several outputs.

And its pretty close to what most people do when playing poker, too. "Ok, I'm in that position, with that blind/stack ratio, no action before me, .... so I do that." --> several inputs to come up the the desired action.

Statistics: Posted by d4n13l — Fri Nov 08, 2013 11:49 am

Re: My ANN AI strategy

2013-11-08T11:29:45+00:00

Yeah, stage means preflop, flop, turn, river.

The problem with using betting sequence is that that there are so many of them it's slow/difficult to learn from them. I guess one approach would be to try to define some similarity metric over betting sequences, which I think is what the SARTRE guys did.

Another problem I see with this whole ANN/Reinforcement Learning approach is that what you really need is a frequency for each action given the current game state. Most RL approaches just compute an optimum action. Getting an optimum action and then "mixing it up a bit" just don't cut it.

And another thing - why is everyone so taken by ANNs? They aren't magic. There are lots of ML techniques out there and some are quite a lot faster than ANNs. Sutton & Barto suggest using tile coding for problems like this. They are just a cheap and cheerful approach to locally weighted learning, and work well in incremental learning.

Statistics: Posted by spears — Fri Nov 08, 2013 11:29 am

Re: My ANN AI strategy

2013-11-08T10:31:11+00:00

I am new to the forum, and might be missing some theoretical background... so treat my idea with caution.

Dahl's machine sounds exciting, and it seems ANN could be a very effective technique after all. A good selection on input is probably crucial here, and yours seems like a good start. It is however unclear to me what exactly you mean with "stage". Is it purely the current stage of the game (preflop, flop, turn, river)? If so, then maybe including somehting like "betting sequence up to now" could also be of interest, if that is not inherent to any of your inputs yet.
This is where often where opponents weakness or strengthis revealed imo. Have you thought/what is the scientific community saying about that?

Statistics: Posted by d4n13l — Fri Nov 08, 2013 10:31 am

Re: My ANN AI strategy

2013-11-05T03:45:28+00:00

Nasher wrote:

spears wrote:

A reinforcement learning algorithm applied to simplified two-player Texas Hold’em poker (Dahl, 2001)

http://www.ke.tu-darmstadt.de/lehre/arc ... istoph.pdf
http://www.google.com/patents/US20100069134

I have a paper copy somewhere. I don't remember being very impressed when I read it, but maybe I missed something. It was early in my poker life.

I remember reading that some time ago and not understanding how it would be used. Do you think that's what he's using currently to train his NNs?

given a few insights I would suggest you to not waste too much time on figuring out what Bastian thinks or does. a few years ago we worked at the same company as software developers. his first days/ weeks (which where intended for him to get an overview of the project's code he was assigned to) he spent "beautifying" the source code, meaning the replaced comment-separator-lines from

Code:

// ########################################################

Code:

///////////////////////////////////////////////////////////////////////////////////////////////////////

because that "allowed" him to understand the code quicker ...

Statistics: Posted by Nose — Tue Nov 05, 2013 3:45 am

Re: My ANN AI strategy

2013-11-03T21:03:22+00:00

It's just a guess that seems to fit the available info. You know at least as much as I do now.

Statistics: Posted by spears — Sun Nov 03, 2013 9:03 pm

Re: My ANN AI strategy

2013-11-03T12:18:04+00:00

spears wrote:

I remember reading that some time ago and not understanding how it would be used. Do you think that's what he's using currently to train his NNs?

Statistics: Posted by cantina — Sun Nov 03, 2013 12:18 pm

Re: My ANN AI strategy

2013-10-18T01:32:39+00:00

spears wrote:

Maybe this will help http://poker-ai.org/archive/www.pokerai ... a&start=60

Thanks, but I've read this whole thread before, a few times, and checked the sources for every code snippet: it seems that all of those issues are fixed in the source code (at least the ones related to FO2....)

Will try again latter. Right now I'm focusing on implementing support for Tournaments in opentestbed.

Thanks,
Corintio

Statistics: Posted by corintio — Fri Oct 18, 2013 1:32 am

Re: My ANN AI strategy

2013-10-18T01:28:31+00:00

Thanks for the replies!

spears wrote:

On the other hand, Dahl seems to have got something working with ANNs. http://www.nytimes.com/2013/09/08/magaz ... .html?_r=0

I've read that article before. That was my inspiration...

I'm planning to use way more than 500 hands. Also, my plan is to create more than one ANN, tuned for different scenarios, maybe more than one for the same scenario. That would be my attempt to make it less exploitable. I'm also planning to do a better opponent modeling than Nicolai's implementation.

spears wrote:

- FOLDER beats all evolved strategies

I also didn't get it! Will try to see what happens in my simulations.

Nasher wrote:

You'll likely end up evolving push/fold players, due to the overwhelming effect of variance (push/fold players have the highest earning potential when luck is a factor).

To avoid this, I'm planning to experiment with different Fitness functions, giving different weights for different aspects of the gameplay (not only ranking and bankroll).

Well, let's see what lessons I can get from all of this

Thanks,
Corintio.

Statistics: Posted by corintio — Fri Oct 18, 2013 1:28 am

Re: My ANN AI strategy

2013-10-17T17:06:15+00:00

Quote:

It's kind of wild he started in '97.

At backgammon

A reinforcement learning algorithm applied to simplified two-player Texas Hold’em poker (Dahl, 2001)

http://www.ke.tu-darmstadt.de/lehre/arc ... istoph.pdf
http://www.google.com/patents/US20100069134

I have a paper copy somewhere. I don't remember being very impressed when I read it, but maybe I missed something. It was early in my poker life.

Statistics: Posted by spears — Thu Oct 17, 2013 5:06 pm

Re: My ANN AI strategy

2013-10-17T16:29:02+00:00

spears wrote:

On the other hand, Dahl seems to have got something working with ANNs. http://www.nytimes.com/2013/09/08/magaz ... .html?_r=0

Touche. I'd like to hear more about his methods, if they're published anywhere. I'd also like to see any benchmarks he may have against known players. It's kind of wild he started in '97.

Statistics: Posted by cantina — Thu Oct 17, 2013 4:29 pm

Re: My ANN AI strategy

2013-10-17T09:50:20+00:00

Maybe this will help http://poker-ai.org/archive/www.pokerai ... a&start=60

Statistics: Posted by spears — Thu Oct 17, 2013 9:50 am

Re: My ANN AI strategy

2013-10-17T07:45:34+00:00

I don't remember reading that paper before but it is quite interesting. I liked the phantom parasite idea.

I don't want to pour cold water on your project but I'm pessimistic. I only skimmed the paper so I may have missed something.
- FOLDER beats all evolved strategies
- 500 hand games are nowhere long enough. My reinforcement learning algorithm was still learning at 20,000 hands
- Fictitious play, CFRM, Nicolai's method all have the same fundamental idea in common.
- FP & CFRM model many more game states than Nicolai's method and are hence potentially more accurate
- FP & CFRM "know" villain's strategy and can therefore exploit it quickly. I don't think Nicolai's method "knows" villain's strategy.
- I'm not sure but I think Nicolai's method creates a pure rather than a mixed strategy. This is exploitable.
- Incremental Learning in ANNs is hard to get right
- ANNs are quite slow compared to the simple histogram approach used in CFRM. Tile coding might improve the histogram approach without the cost of ANNs

On the other hand, Dahl seems to have got something working with ANNs. http://www.nytimes.com/2013/09/08/magaz ... .html?_r=0

Hope this helps.

Statistics: Posted by spears — Thu Oct 17, 2013 7:45 am

Re: My ANN AI strategy

2013-10-17T00:06:41+00:00

I tried HUNL with NEAT too. You'll likely end up evolving push/fold players, due to the overwhelming effect of variance (push/fold players have the highest earning potential when luck is a factor). Just trying to save you months or years of wasted effort.

I remember the FO2 zip(s) being bigger, but I could be mistaken. I also don't remember dashes in the file names. You might look in the zips and see if something isn't amiss. If it can't find the strategy file it's either:
a) Incorrect encoding of the game state (i.e. filename).
b) The file is missing from the zip(s).

Statistics: Posted by cantina — Thu Oct 17, 2013 12:06 am

Re: My ANN AI strategy

2013-10-16T23:35:13+00:00

My plan is to play HU NoLimit, and I'm too close to check if this approach (ANN/Evolution) is valid or not. Nevertheless, thanks for the advice.

But whatever approach I use, I will need FO2 running with OTB. Can you help me with that?

I downloaded FO2 from this site: http://www.deducer.org/pmwiki/index.php ... ndFellOmen, and it came with a folder called Strategy_FellOmen_2 with 73MB in zip files. Is this the correct distribution?

The error I get is:

Code:

...
turn Hand Strength : 0.5497584541062802 turn potential: 0.08984263233190272 turn negative potential : 0.12560046865846514
action observed: Check
rollout Hand Strength: 0.5211594202898551 turn index: 84
turn file: p1_strategy83-10_5.txt
table : p1_strategy83-10_5.txt
trying Strategy_FellOmen_2/1.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/2.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/3.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/4.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/5.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/6.zip
reached end of zip file before table was found
trying Strategy_FellOmen_2/7.zip
reached end of zip file before table was found
Exception in thread "main" java.lang.NullPointerException
   at com.ibot.FellOmen_2_Impl.getActionTable(Unknown Source)
   at com.ibot.FellOmen_2_Impl.turnAction(Unknown Source)
   at FellOmen_2.turnAction(Unknown Source)
   at FellOmen_2.getAction(Unknown Source)
   at game.Dealer.playStage(Dealer.java:102)
   at game.Dealer.playHand(Dealer.java:66)
   at game.CashGameRunner.runGames(CashGameRunner.java:82)
   at game.CashGameRunner.runGame(CashGameRunner.java:46)
   at CashGameConsoleStarter.main(CashGameConsoleStarter.java:52)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

Thanks,
Corintio

Statistics: Posted by corintio — Wed Oct 16, 2013 11:35 pm

Re: My ANN AI strategy

2013-10-16T23:12:45+00:00

Yeah, you need the strategy files for FO2 (a few hundred megs worth). My point is, you're likely wasting your time with evolution in HUL.

Statistics: Posted by cantina — Wed Oct 16, 2013 11:12 pm

Re: My ANN AI strategy

2013-10-16T22:28:18+00:00

Yeah, I know it's really weak, but as I said, this was a proof of concept, and also helped me putting everything together (opentestbed, NN, evolutionary algorithm...)

I tried to use Fell Omem 2, but it gives me an error when running on opentestbed, an exception about strategy not found. Is it a known issue? Is it possible to run Fell Omen 2 with OTB? Otherwise, what do you guys use to compete against it?

Statistics: Posted by corintio — Wed Oct 16, 2013 10:28 pm

Re: My ANN AI strategy

2013-10-16T21:25:28+00:00

When you can beat Fell Omen 2 by a reasonable margin then you'll have something. Simplebot is a very weak player. I've tried using Neuro-Evolution of Augmenting Topologies (NEAT) to build strategies, without avail. I'd bet any evolutionary approach you take will fail in performance.

Statistics: Posted by cantina — Wed Oct 16, 2013 9:25 pm

My ANN AI strategy

2013-10-16T21:14:03+00:00

Hi All,

I'd like to outline here my AI implementation strategy, any feedback would be very welcome.

I'm basing my development on this paper: http://www2.cs.uregina.ca/~hilder/my_st ... Thesis.pdf

I'm developing it using OpenTestBed/Meerkat API. My approach is to use an ANN to actualy play poker (as opposed to just model opponents). Right now I'm collecting the following information from the game state as input to my ANN:

- Hand Strenght
- Hand Potential
- Amount to call
- Eligible pot
- My amount in pot
- # active players
- Stage (including pre-flop)
- My distance from button
- Chips (mine and from opponents)
- Some basic opponent modeling

and my ANN outputs the action that should be performed (fold, check/call, bet/raise)

As a proof of concept, I've trained my ANN with 10000 hands of FL-HU cash games played between two SimpleBots, with 5% error. Then I used a Genetic Algorithm to evolve this ANN so now it beats SimpleBot consistently.

My next step is to setup a tournament style game play so I can put all candidates in each generation of my GA to play against each other, hoping that it can evolve even further.

Also, I'm changing my ANN to play NoLimit games, and the output now is (fold, check/call, bet/raise low, medium, high, all-in)

Any comments?

Statistics: Posted by corintio — Wed Oct 16, 2013 9:14 pm