Cards Game AI
Part 001: Why to put the Neural Nets on Poker?
Implementing reinforcement learning in such complex environment is not straightforward, but I can split the problem into smaller factors. For sure, it will take time. In the meantime, I will report the progress of the project and my observations here in the subsequent posts.
In general, I’ll have to think about the input data, build poker environment representation for neural network and think about neural architectures. Moreover, I’ll need to write the code — I have chosen Python, mainly with Numpy and TensorFlow. There’s also room for multiprocessing, genetic algorithms and many other interesting experiments. That’s a lot, but let’s start simple:
Would Neural Network (NN) understand a card game?
We have a deck of 52 cards, 13 ranks and 4 suits. To master any card game, we have to understand the cards. In poker (Texas holdem), a set of five cards makes a valid poker hand. There are flushes, straights, pairs and other sets ranked according to the game rules. The same ranks may have different strengths, for example, a pair of kings is stronger than a pair of jacks. It would be helpful to assign each hand a unique integer for comparison. Would NN recognize (classify) ranks and compare them properly?
The solution to the problem seems to be easy, and as I have observed, it’s quite interesting.
It’s possible to write an algorithm that would take the hand as an input and return its rank and the strength integer. I’ve done it and it took about one hour and 100 lines of Python code studded with conditions. I’m sure I’ll use that for training data preparation. So would the neural network manage to do the classification in any better way?
The solution to the problem requires some simple skills: sorting the cards, looking for pairs, triples, suits, etc. selecting the five strongest cards from seven and once again putting them in order for easy comparison — this is an outline of the strategy — a simple logic and even simpler mathematics. An average person who has been taught the rules would rank and compare almost every poker hand after a few minutes.
Believe it or not but even advanced players sometimes make simple mistakes in this simple task — and that’s because some hands are harder to compare. However, the correct algorithm will not make any mistakes. And what about AI?
It is known that neural nets are quite poor at mathematics and logic. Deep networks are great in modeling many phenomena and functions but can you remember the XOR perceptron problem? It almost stopped neural development in the 1960s ;) ! Now it is solved with the hidden layer.
Are you familiar with the paper “Neural GPUs Learn Algorithms” by Ł. Kaiser and I. Sutskever? The authors said that learning an algorithm from examples is a fundamental but hard task for neural nets. They showed a solution that is quite successful in some stated tasks but it does not generalize easily, and it’s quite complicated and hard to train. Our simple task is very similar: to learn the algorithm of ranking from given examples of hands.
So why do it with NN?
First, because it’s an interesting task. Second, and this is the point: because the NN would learn effective cards and hands representations and we could extend that task to a more general one where there are many players at the table and we do not know all the cards! Evaluation of hand strength in that scenario is a known hard problem of the poker game. Because of computational complexity, numerical approaches are not relevant here (Effective Hand Strength @wiki).
In my next post I’ll write about the tested neural architectures, achieved results and some interesting observations.