Hello, it has been some time since I wrote on this... And I have had several set backs, but at last I have created the framework for my poker AI. The program pick random cards for each of the possible card positions, it calculates the best hand between the player and one opponent (This will be the AI). This is just the set up for the AI however.
At the current moment in time, the plan is to create a decision tree which will create a "decision score" and using fuzzy logic find out the correct way to play, whether it be fold, checking or raising (and by how much). This will be kept up-to-date from now and on the current situation of the program and the written dissertation.
Tuesday, 22 March 2011
Tuesday, 25 January 2011
Research Proposal
Hello! This is my final research proposal that I have sumitted.
Introduction
Poker has been played for years, the origins are disputed, but it has been about for a long time. It has been the past time and profession of many people all over the world. Poker first made it’s way into the mainstream, only in the past few years. Since the invention of the internet, all forms of poker have exploded since any one can play, and for very little.
Poker games have been created since the rise of the video games industry, with it possible to play other players online or even against an artificial intelligence (AI). Since these games can be played for no money they can be used like most other games, just played for entertainment purposes. However the games with AI are simple and just use basic betting strategies and are very repetitive and it becomes very simple to see the patterns in the way that they play and for a decent player, it is easy enough to exploit these weaknesses.
Real poker players change their strategy according to their opponent, rather than just stick to one formula. This is essential for any poker player to make a profit and avoid losing all their money very quickly. Obviously a computer program doesn’t need to worry about how much money they lose, as long as the user is having fun. The goal of this project is to make the computer think more like a human on the poker table and vary its style of play depending on opponent. Which leads us to the question “Can a no-limit Texas Hold’em poker AI learn from other players and change strategy accordingly?”
Motivation
Poker video games, even at an advanced level tend to be easily read and not pose a challenge or any sort of real learning to the game and the thought processes involved, even at the highest difficulties the games are not challenging enough. Simple randomising algorithms tend to be used rather than exact calculations on bluffing strategies and pot-odd calculations that a real professional player would do.
The term artificial intelligence is “the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.” According to the oxford English dictionary. When a program uses non-exact methods of working out what is the correct play in certain situations, this goes against the actual meaning of AI, as a human wouldn’t “randomly” do something just because they want to be beaten out of their money and for the other player to have fun.
The goal for this project is to create an AI agent; capable of playing Texas hold’em poker, and to play well and to be able to pose a challenge or even beat professional players. Obviously everyone is not a professional poker player, so difficulties should be taking into consideration.
Research Topic
The topics that will need to be researched for this project will be
Current methods
Game theory
AI methods
Current methods – The methods currently used in the industry to create challenging AI poker players
Game theory – The theory behind all strategies used in games of this nature, such as pot-odd calculations (working out if the strength of a hand is worth the risk of the money involved.), bluffing calculations (when to bluff, when not to bluff, how often to bluff, how much on each bet per bluff)
AI methods – The way that an AI agent is structured. Such as fuzzy logic, finite state machines.
The current methods will need to be researched as this will provide a background and a starting point for making the AI. Game theory is important for poker, as it provides the knowledge that a professional poker would have to know which action is the best, i.e. calling, betting, raising, checking or folding. The AI methods that could possibly be used will need to be researched and the best methodology selected for this project.
Research Question
The research question for this project is:
“Can a no-limit Texas Hold’em poker AI learn from other players and change strategy accordingly?”
The objectives are as follows:
To produce a working Poker agent.
For this agent to be able to play competently using game theory and betting strategies.
For the agent to be able to change it strategy depending on how it’s opponent plays.
Project Methodology
There are several different methods in which this can be accomplished. Since poker is a game of “Imperfect Information” (some of the cards are concealed), it cannot be treated the same way in which one would treat a chess AI, for example where all the information is available.
All poker decisions are made via a neural network. All the information must be taken into consideration to create a truly realistic poker AI, use the information work out the probabilities and use these to make the correct decision on what play to make.
The input layer would consist of the number of players left in the pot, how much is in the pot, the current bet, the previous bets, the position, the community cards and the hole cards. This is information that a professional poker player would use. The hidden layer would consist of the thought processes that human would do. Such as pot odd calculations, hand probabilities, using such algorithms as Bayes’ theorem, the Nash Equilibrium and the Monte Carlo method, the AI should be able to run complex computations, over and over so that all of the different possibilities can be worked out and each action given a percentage of what should be done. For example, 40% chance that the AI will fold and 60% on the AI to raise a bet. These will be used to work out the output layer, which will be either fold, check, call, bet or raise. Each action will be given an additional weight, depending on how the opponent has been playing up to this point. They will be separated into different categories. The main categories will include: Tight-Aggressive, Tight-Passive, Loose-Aggressive and Loose-Passive. (To clarify, a Tight player is one that only plays the possible hand, a Loose player is one that plays a lot of hands, good or bad, an Aggressive player will bet and raise, rather than check and call, and a Passive player will check and call, rather than bet and raise).
Project Evaluation
The main point that needs to be stated in this section is that the final product will be focused on the AI and the algorithms used. It will most likely have a simple UI. Graphics are not a priority, and will only be considered if time allows for it. The main aim of the construction of the AI is to use proven winning betting strategies to take advantage of opponent’s weaknesses and exploit them.
The first test that will be used on this program to see if can actually play poker and if the program will deal hands and the AI will use reasonable actions. The second test will be if the AI agent can use game theory and betting strategies, to find optimum method for its own hand. The third and final test will be to test if it can change its strategies according to the way that an opponent plays.
It will also need to be tested against several different types of opponents, so the AI will be tested against various people of different skill levels, and it will be able to take advantage of each player and win consistently. The main target for all of the tests will be that will more often than they lose. Even the world’s best poker players do not win on every single hand, as long as the AI agent makes a net profit, then it will be a success.
Issues
The same issues will apply here as they would with any project and these problems need to be dealt with as soon as they arise. Any issue that can be foreseen in advanced would drop the time taken to sort these inconveniences.
Resource Requirements
The project will need an access to academic resources such as books, white papers in the aspects of Artificial Intelligence and Poker strategies, as much as the normal research requirement.
There are no exceptional resources needed for this project.
References
lab Joy Woller (1996) The basics of Monte Carlo Simulations [Online] Available from :
http://www.chem.unl.edu/zeng/joy/mclab/mcintro.html [Accessed on 20/12/2010]
Unknown Author (Date unknown) [Online] Available from : http://pokerai.org/pj2/index.php?option=com_frontpage&Itemid=1 [Accessed on 24/12/2010]
Eliezer S. Yudkowsky (2003) An Intuitive Explantion of Bayes Theorem [Online] Available from: http://yudkowsky.net/rational/bayes [Accessed on : 19/12/2010]
Unknown Author (2010) CPRG Homepage [Online]Available from : http://poker.cs.ualberta.ca/ [Accessed on : 18/11/2010]
Roger A. McCain (Date Unknown) Game theory : An Introductory Sketch [Online] Available From : http://faculty.lebow.drexel.edu/McCainR//top/eco/game/nash.html [Accessed on : 18/11/2010]
Unknown Author (Date Unknown) [Online] Available from : http://www.gametheory.net/ [Accessed on : 18/11/2010]
Introduction
Poker has been played for years, the origins are disputed, but it has been about for a long time. It has been the past time and profession of many people all over the world. Poker first made it’s way into the mainstream, only in the past few years. Since the invention of the internet, all forms of poker have exploded since any one can play, and for very little.
Poker games have been created since the rise of the video games industry, with it possible to play other players online or even against an artificial intelligence (AI). Since these games can be played for no money they can be used like most other games, just played for entertainment purposes. However the games with AI are simple and just use basic betting strategies and are very repetitive and it becomes very simple to see the patterns in the way that they play and for a decent player, it is easy enough to exploit these weaknesses.
Real poker players change their strategy according to their opponent, rather than just stick to one formula. This is essential for any poker player to make a profit and avoid losing all their money very quickly. Obviously a computer program doesn’t need to worry about how much money they lose, as long as the user is having fun. The goal of this project is to make the computer think more like a human on the poker table and vary its style of play depending on opponent. Which leads us to the question “Can a no-limit Texas Hold’em poker AI learn from other players and change strategy accordingly?”
Motivation
Poker video games, even at an advanced level tend to be easily read and not pose a challenge or any sort of real learning to the game and the thought processes involved, even at the highest difficulties the games are not challenging enough. Simple randomising algorithms tend to be used rather than exact calculations on bluffing strategies and pot-odd calculations that a real professional player would do.
The term artificial intelligence is “the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.” According to the oxford English dictionary. When a program uses non-exact methods of working out what is the correct play in certain situations, this goes against the actual meaning of AI, as a human wouldn’t “randomly” do something just because they want to be beaten out of their money and for the other player to have fun.
The goal for this project is to create an AI agent; capable of playing Texas hold’em poker, and to play well and to be able to pose a challenge or even beat professional players. Obviously everyone is not a professional poker player, so difficulties should be taking into consideration.
Research Topic
The topics that will need to be researched for this project will be
Current methods
Game theory
AI methods
Current methods – The methods currently used in the industry to create challenging AI poker players
Game theory – The theory behind all strategies used in games of this nature, such as pot-odd calculations (working out if the strength of a hand is worth the risk of the money involved.), bluffing calculations (when to bluff, when not to bluff, how often to bluff, how much on each bet per bluff)
AI methods – The way that an AI agent is structured. Such as fuzzy logic, finite state machines.
The current methods will need to be researched as this will provide a background and a starting point for making the AI. Game theory is important for poker, as it provides the knowledge that a professional poker would have to know which action is the best, i.e. calling, betting, raising, checking or folding. The AI methods that could possibly be used will need to be researched and the best methodology selected for this project.
Research Question
The research question for this project is:
“Can a no-limit Texas Hold’em poker AI learn from other players and change strategy accordingly?”
The objectives are as follows:
To produce a working Poker agent.
For this agent to be able to play competently using game theory and betting strategies.
For the agent to be able to change it strategy depending on how it’s opponent plays.
Project Methodology
There are several different methods in which this can be accomplished. Since poker is a game of “Imperfect Information” (some of the cards are concealed), it cannot be treated the same way in which one would treat a chess AI, for example where all the information is available.
All poker decisions are made via a neural network. All the information must be taken into consideration to create a truly realistic poker AI, use the information work out the probabilities and use these to make the correct decision on what play to make.
The input layer would consist of the number of players left in the pot, how much is in the pot, the current bet, the previous bets, the position, the community cards and the hole cards. This is information that a professional poker player would use. The hidden layer would consist of the thought processes that human would do. Such as pot odd calculations, hand probabilities, using such algorithms as Bayes’ theorem, the Nash Equilibrium and the Monte Carlo method, the AI should be able to run complex computations, over and over so that all of the different possibilities can be worked out and each action given a percentage of what should be done. For example, 40% chance that the AI will fold and 60% on the AI to raise a bet. These will be used to work out the output layer, which will be either fold, check, call, bet or raise. Each action will be given an additional weight, depending on how the opponent has been playing up to this point. They will be separated into different categories. The main categories will include: Tight-Aggressive, Tight-Passive, Loose-Aggressive and Loose-Passive. (To clarify, a Tight player is one that only plays the possible hand, a Loose player is one that plays a lot of hands, good or bad, an Aggressive player will bet and raise, rather than check and call, and a Passive player will check and call, rather than bet and raise).
Project Evaluation
The main point that needs to be stated in this section is that the final product will be focused on the AI and the algorithms used. It will most likely have a simple UI. Graphics are not a priority, and will only be considered if time allows for it. The main aim of the construction of the AI is to use proven winning betting strategies to take advantage of opponent’s weaknesses and exploit them.
The first test that will be used on this program to see if can actually play poker and if the program will deal hands and the AI will use reasonable actions. The second test will be if the AI agent can use game theory and betting strategies, to find optimum method for its own hand. The third and final test will be to test if it can change its strategies according to the way that an opponent plays.
It will also need to be tested against several different types of opponents, so the AI will be tested against various people of different skill levels, and it will be able to take advantage of each player and win consistently. The main target for all of the tests will be that will more often than they lose. Even the world’s best poker players do not win on every single hand, as long as the AI agent makes a net profit, then it will be a success.
Issues
The same issues will apply here as they would with any project and these problems need to be dealt with as soon as they arise. Any issue that can be foreseen in advanced would drop the time taken to sort these inconveniences.
Resource Requirements
The project will need an access to academic resources such as books, white papers in the aspects of Artificial Intelligence and Poker strategies, as much as the normal research requirement.
There are no exceptional resources needed for this project.
References
lab Joy Woller (1996) The basics of Monte Carlo Simulations [Online] Available from :
http://www.chem.unl.edu/zeng/joy/mclab/mcintro.html [Accessed on 20/12/2010]
Unknown Author (Date unknown) [Online] Available from : http://pokerai.org/pj2/index.php?option=com_frontpage&Itemid=1 [Accessed on 24/12/2010]
Eliezer S. Yudkowsky (2003) An Intuitive Explantion of Bayes Theorem [Online] Available from: http://yudkowsky.net/rational/bayes [Accessed on : 19/12/2010]
Unknown Author (2010) CPRG Homepage [Online]Available from : http://poker.cs.ualberta.ca/ [Accessed on : 18/11/2010]
Roger A. McCain (Date Unknown) Game theory : An Introductory Sketch [Online] Available From : http://faculty.lebow.drexel.edu/McCainR//top/eco/game/nash.html [Accessed on : 18/11/2010]
Unknown Author (Date Unknown) [Online] Available from : http://www.gametheory.net/ [Accessed on : 18/11/2010]
Wednesday, 17 November 2010
Worksheet 2
Honours Project Worksheet — #2
My research question will be "Can a no-limit Texas hold'em AI learn from other players and change strategy accordingly?"or something to that effect.
The focus of the project will be to build an AI that can use betting strategies to out smart and out play opponents. When the final program has been built, it will, at first, play against other AI's (and if time allows, have the AI different "personalities") and then I wish to make it be able to play against human players.
Possible expansions:
Have the AI predict an opponents hole cards
Teach the player what it has done wrong
(I doubt I will have time to make both of these possible, but it would be possible to do for future expansions outside the honours project)
Final Idea & Work sheet 1
Honours Project Worksheet – #1
- State the topic of interest that you may wish to take forward into a project.
- Indicate the issues surrounding this topic that you may wish investigate.
1. I wish to make an AI the can not only play Texas hold'em poker but adapt to how the opponent plays, using various betting strategies and advanced game theory.
The goal of this would be to build a program so that it could beat a good player and a bad player, and (most importantly) win the most money out of them. It should see how the opponent plays (ie. Do they bet a lot? Do they play a lot instead of folding?), and use this to their advantage. For example, if the enemy is a "maniac" (Plays a lot of hands and constantly bets, raises and re-raises), then the AI should start playing only the top hands and only bet/raise when it is totally sure that it has the best hand.
2. The main problem I have at the moment is that I am not sure what to use as a framework. I don't want to spend a lot of time building just the set-up (ie. the dealing of card, the graphics etc), as this will take away from time I have to make the AI as good as possible.
The maths of working out good betting strategies is in nearly every single poker book and since I am an avid player myself, I have personal experience on which strategies work and why.
I also wish (If time allows), to have the AI "guess" the opponents hole cards, like what a professional would do. In reality, pro's use the information that has been given by the previous betting on each different betting level and puts them on a range of possible hands and the more bets are made the more this range can be shortened. It also would take into account the type of play the enemy is, and use this to determine the most likely cards they have.
As a side note, I do not wish to make the computer cheat in any way, shape or form. If everything goes well, it should only play with the hand it has been given and the bets.
Monday, 4 October 2010
Worksheet 2
Questions for Worksheet 2:
1. Formulate a research question for your project.
2. When developing a research question you must produce a question that can be answered by performing literature research, practical experimentation and critical analysis. The question should also focus the activities of your project.
What is the focus of your project?
What information do you need to find out about?
What Experimental work do you need to perform?
3. How will the information that you obtain in 2 allow you to answer the research question
2.The focus of my project would be to research and build a more challanging racing AI. By looking into different techinques used and seeing if any of them can be improved upon. I will need to find the current best way, according to the industry, of using slipstreaming and looking for potential problems and finding a way to solve them. My plan is to build a simple AI that simply follows the track and another AI that tried to overtake it using the slipstream caused by the car infront.
3. If I can experiment with different ways of using slipstreaming, I can time laps around the course and see which is the fastest. Since it is a racing game and lap times will be monitored, it will be easy to look into which was the quickest and why.
Thanks :D
Worksheet 1
Hello! My original idea for my hon's project is to look into Driving AI but in particular looking into how an AI would deal with slipstreaming and use it to it's advantage. Slipstreaming is when there is a car infront another car during a race. Since they are driving fast, the air pressure acts against the car slowing it down, however if the a car has something to break the air pressure infront it, such as another car, then they can actually move faster since there is less pressure slowing the car down.
I wish to look into this and see if there are problems with the current way of doing it, and why more racing games do not have this feature. I also wish to look into the different ways that this can be done, and see if it can be improved upon.
For thsi idea, I would build a program that would simply make 2 cars go around a track. (If time allows I wish to do a complex track so that thety AI would have to take into account the corners when trying to overtake, however if this is not realistic, I wish to do just an oval track). One car would travel at a constant speed around the track, where as the other car would try to use slipstreaming to try and get infront of it.
I hope this is a good enough description, if you wish to know more about my idea, comment away! :D Thanks!
I wish to look into this and see if there are problems with the current way of doing it, and why more racing games do not have this feature. I also wish to look into the different ways that this can be done, and see if it can be improved upon.
For thsi idea, I would build a program that would simply make 2 cars go around a track. (If time allows I wish to do a complex track so that thety AI would have to take into account the corners when trying to overtake, however if this is not realistic, I wish to do just an oval track). One car would travel at a constant speed around the track, where as the other car would try to use slipstreaming to try and get infront of it.
I hope this is a good enough description, if you wish to know more about my idea, comment away! :D Thanks!
Monday, 27 September 2010
First Post
This is going to be my blog am I going to use for my Honours project for my 4th year of Computer Games Technlolgy! I'll keep this up to date as much as possible! :D x
Subscribe to:
Posts (Atom)