Image Image Image




Post new topic Reply to topic  [ 13 posts ] 
Author Message
 Post subject: UI automation
PostPosted: Tue Feb 12, 2013 12:21 am 
Offline
Regular member
User avatar

Posts: 76
Favourite Bot: ...
I'm trying to build an idealistic UI automation library right now, and I'm kind of obsessed over this. Something smart that doesn't need any hard coded parts. A bit pointless, but I have the time... And I wonder if anyone has tried something like this before...

I'm thinking of having a "task queue" of "paths or trees of ui/input conditions". Where a "condition" would be something like "click at x/y on a certain window" which also has some prerequisite conditions like "window is visible" and how it's possible to reach all that from the current UI state. These paths are known and usually linear. When there are several tasks in the queue, a graph search algorithm can be applied to find an optimal path to reach all end conditions.

Now I'm trying to make things more complex. How to optimally handle all the cases like, possible pop-ups and "required to act in a sec", where the UI state change isn't the result of immediate user input, but which can be predicted (for ex. in a poker table, to make the bot cleverly wait for the action to reach me, before doing something else, when it's about to be my turn to act in any second)?
So I want to have a dynamically changing graph, not just having an incremental search going on it, but also have predictions for the changes and have all that somehow included in the initial search.

Any thoughts on this? I guess the main problem for me is finding things to read about this topic, a proper description or name for this kind of problem is missing. And no glue of which search algorithms can be used to solve this problem. All I know right now is that D* and it's derivatives can be used for incremental search. I guess I end up having the predictions "compiled" into a static graph somehow.


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Tue Feb 12, 2013 5:34 pm 
Offline
Regular member
User avatar

Posts: 87
Favourite Bot: Mine
Hi Kratt,

I am not exactly sure what you need but I am guessing this is for automating on whether its time for the bot to make a decision and deal with common pop up dialogs and situations. Each site is different obviously and you will have to program for all of the scenarios. Some of these scenarios are not obvious until they happen. My "UI" main event loop has to do with checking for an active raise button (meaning its my turn for a decision), an active "I am back" button (due to a bustout or lost connection). For HU I had to also determine if the villain had left the table so the bot is not just sitting there doing nothing. So my UI decision flow is: 1. check that villain is still seated (and not sitting out). 2. check for "I am back". 3. check for active raise button. 4. verify what street the bot is on. 5. run AI and click the right button. That is pretty much it. For various sites you will have to check for various "nag dialogs" like if you are on a waiting list, captcha dialogs, etc.

You will also have to automate each various situation that comes up like the "I am back" situation. For a lost connection you just click the button and continue. For a bustout you have to click the button and then automate a rebuy. Another situation that comes up frequently is leaving the table after a big win. I do not like to have more then 150BB at risk per table (not to mention my AI is designed for 100BB stacks) so I have it leaving the table and rejoining on 9 max games.

Hope this helps


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Tue Feb 12, 2013 6:59 pm 
Offline
Regular member
User avatar

Posts: 76
Favourite Bot: ...
Thank you for your input. What you described was "how you check if certain conditions are true, turn to act, nag, and so on". I have no trouble handling all that. Guess, I have trouble explaining myself :D. The word "idealistic" in my post should have been written in bold, meaning it's kind of unnecessary and bloat talk.

I'll give an example of the usage:

mouse.moveTo(0, 0);
desktop.getACalc().enter("2+2=");
desktop.getANotepad().write("hello");
/**/
table1.fold();
/**/
table2.call(amt_set_wager);
/**/
taskQueue.start(); //just to illustrate that several tasks are in a queue at once. task handling would happen in another thread.

Where different tasks would have different (time)costs and urgencies. And I want to build an engine that can handle all these situations intelligently, so it finds the best order or path how to complete the tasks. E.g. acting in poker tables would be most important, so that gets handled first. Or like a situation where you have 2 overlapping poker tables and you want to tick the quick fold check box in the table that's covered by the second table, but instead of bringing up the first table, you wait because you are about to act in the second table in any moment(is about to pop back up anyway). So you bring up the first table and do the quick fold after acting in the second table.


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Tue Feb 12, 2013 9:36 pm 
Offline
Regular member
User avatar

Posts: 87
Favourite Bot: Mine
ah ok. I see what you mean now. I only have two threads in my bot. The main thread handles all the table UI/AI actions (for all tables) and another one is just for table hopping/rebuys. I know some people have a separate thread for each table but I never saw the need for that. The best way I have found to handle all this was to just check for all the above scenarios on each table one at a time and in order without giving any particular table any priority.


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 2:52 am 
Offline
Regular member
User avatar

Posts: 76
Favourite Bot: ...
I didn't mean having a separate thread for each table, just one for the UI actions. Just saying, to be clear. Your way is practical, and the way to go (Y). I'll see if I can come up with something impractical :D


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 10:18 am 
Offline
Senior member
User avatar

Posts: 134
Favourite Bot: my (incomplete) bot
Kratt wrote:
Something smart that doesn't need any hard coded parts.

But then what about the pretty pretty pixel patterns? Besides I hear all the cool kids use screenshots...

This is the important stuff:
1. take screenshots
2. getRGB(x,y) -> System.out() -> Control+V -> Control+P
3. opponent usernames can't have characters that aren't on your keyboard
4. whats antialiasing?
5. grayscale is for pussies

_________________
TorChat ID: "irqnpurfeuebdwmd"


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 11:53 am 
Offline
Regular member
User avatar

Posts: 76
Favourite Bot: ...
I wouldn't consider pixel patterns to be something hard coded. Pixel patterns are alright, I'm not concerned with that. If I had an input device that eats screenshots somehow :D maybe then.


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 12:06 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Do you want to write something like this http://www.sikuli.org/


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 12:28 pm 
Offline
Regular member
User avatar

Posts: 76
Favourite Bot: ...
A little bit. But more like something that could write smart ass sikuli scripts for you, on the fly... let's put it this way then - multitasking in mind (play poker, chat on an IM, and draw dicks in paint) - it's about how to spend the least amount of energy(think number of input commands) on all that. So it's only about the actual input part, excluding everything about getting the UI states/or the UI "search". That's a separate thing, already neatly handled/ "hard coded" :D. Also, I'm not dealing with any screenshots and I don't intend to make this suitable for something really chaotic like website navigation.


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 1:48 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
You would need a state transition model of all the possible UI environments, not just one application.
http://en.wikipedia.org/wiki/State_diagram
http://en.wikipedia.org/wiki/Petri_net

Finding a path through the model from one state to another would be a graph search of some kind.
Key words: Backtracking, Graph Search, Parsing


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 2:04 pm 
Offline
Regular member
User avatar

Posts: 76
Favourite Bot: ...
Thanks spears! Now that connects a lot of dots. :)


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 2:37 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
Glad I could help someone anyway...


Top
 Profile E-mail  
 
 Post subject: Re: UI automation
PostPosted: Wed Feb 13, 2013 4:06 pm 
Offline
PokerAI fellow
User avatar

Posts: 1239
Favourite Bot: my bot
- State charts get chaotic quickly if you don't have them nested - "post a reply" is inside "logged into phpbb" is inside "in browser"

- This is an interesting idea you've had and I'm wondering about the applications. In principle a program should be able to generate a model of its environment automatically. I guess automated test tools already do something like that.


Top
 Profile E-mail  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ] 


Who is online

Users browsing this forum: Google and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: