The AI that mastered Starcraft II

Artificial intelligence researchers at
DeepMind have mastered Pong, Chess, and Go but some games are on another level. The game of StarCraft II one of the most complicated video games ever created. It
is a real-time strategy game which means that you don't take turns you play and
you have to play fast. It was clear that StarCraft would be a great grand
challenge for us to tackle using an artificial intelligence, the techniques
we love namely neural networks, deep reinforcement learning and so on so
forth. My name is Oriol Vinyals and I'm the lead researcher of the AlphaStar
project. AlphaStar is an artificial intelligence system that plays the game StarCraft II. So the first element of StarCraft that's quite different is that
you don't get to see what's happening on the other side of the board that is
what we call imperfect information and is a key challenge that neither in Chess
or Go are present. Another key difference is that the pieces of Chess
or what we call units in StarCraft are created throughout the game, right, so you
start with very very few units and what you're doing is deciding which pieces
are you gonna invest in creating next.

And that's very important because at the
same time you must counter pieces or units that the other player is thinking
about without having perfect information. The first step of AlphaStar is can you
predict where a human is gonna click what's the most likely move so
essentially AlphaStar is gathering information from the game where all its
units are it's getting all this data into, not a program, but rather a
mathematical entity that is called neural network that is able to then
process all this information.

We never hard coded what this processing means
this was fully learned from, first imitation learning, trying to imitate
human moves and later on self play through the AlphaStar League and after
this processing many layers of computation of learned behavior at the top
there is essentially a single action that will be sent back to the game, and
so this action could be create a new unit create a worker, send this worker
to scout, move the camera to this location and that's essentially what AlphaStar is.

Every basically 20 milliseconds in the game and every
action you see here is a result of computation using this neural network. By January 2019 AlphaStar was able to beat two professional players under special test conditions. But people wanted to see if it could reach the GrandMaster level. Beating the best players at the full game. What's quite critical is that
humans are very creative at playing this game so once we went online with this
agent we faced all sorts of creative strategies that we could not possibly
have ourselves designed. So the very first game that we entered we were
playing against obviously someone we don't know, so it really was like a lot
of adrenaline and a great moment to let people essentially experience what it is
to play against such a strong AI. AlphaStar progressed by playing against
itself, against a range of human players and crucially by playing against other
AI agents. We have this league of agents where all the agents are doing self play
in a manner that is perhaps a bit more advanced than what has been done before
and there we did perhaps the most research advances in the last few months
in which we discovered that it's not sufficient to have only one agent that
its sole purpose is to beat itself and its past versions of itself.

It's very
important to add agents that find flaws in the main agents right, so we call
these agents exploiter agents and if you will it's kind of asking for a friend to
beat your current strategy and maybe this strategy that is going to beat you
is not a great strategy in isolation but it's certainly gonna help you become a
better player and that core idea plus then some extra developments in
imitation learning and reinforcement learning that we had in the project
created an artificial intelligence agent that can play the game of StarCraft II at
a GrandMaster level. Actually I used to play the game back
the 90s I played quite seriously the first version of the game one of the sad
things that happened is that I kind of waited too long to play against AlphaStar.
So the team was really like Oriol you should play, at some point I
decided okay I'll play against the agent I played when it was just strong enough
that there was no chance for me to win and I just basically lost the window for
me to kind of defeat the agent sadly so cool, victory not for me AlphaStar has broken into the top 200 players in Europe at StarCraft II
playing the full version of the game with all three alien races.

We are here not trying to solve StarCraft we're trying to utilize
StarCraft for the purpose of advancing AI research but be applied to any domain
that utilizes artificial intelligence in any of its components and enhance for
instance its robustness its capabilities to do imitation learning from experts
and other things that we deem super important and that StarCraft really lets
us test on one of the most complicated video games ever created. Success in the real world is a little more complicated computer games can readily provide the
data points an AI needs to master a challenge but not every problem provides
so much accurate feedback. The integration of AI into everyday life
will be a much longer battle.

Leave a Reply

Your email address will not be published. Required fields are marked *