Google DeepMind has built an AI agent called SIMA 2 that can navigate and solve problems across different 3D video game worlds — a step toward machines that can follow human instructions in complex environments and eventually control real robots.
The agent plays games like No Man's Sky and Goat Simulator 3 by watching the screen pixel by pixel and learning which keyboard and mouse inputs accomplish specific tasks. What makes SIMA 2 different from previous game-playing AI is that it doesn't chase preset victory conditions. Instead, it learns to follow instructions from humans — via text chat, voice commands, or even drawings on the screen.
Learning from failure
SIMA 2 was trained on footage of humans playing eight commercial games plus three custom virtual worlds. The agent learned to match inputs to actions, but the real breakthrough came from connecting it to Gemini, Google DeepMind's large language model. Gemini helps SIMA 2 understand what it's being asked to do, ask clarifying questions, and generate its own practice tasks.
We're a new kind of news feed.
Regular news is designed to drain you. We're a non-profit built to restore you. Every story we publish is scored for impact, progress, and hope.
Start Your News DetoxWhen SIMA 2 fails at something, Gemini offers tips. The agent tries again. Repeat this enough times, and it improves through trial and error — much like a human learning to cook by attempting the same recipe multiple times.
Researchers tested SIMA 2 in environments it had never encountered. In one experiment, they used another Google AI system called Genie 3 to generate entirely new virtual worlds from scratch, then dropped SIMA 2 into them. The agent could navigate and follow instructions in these unfamiliar spaces, suggesting the skills it learned weren't locked to specific games.
The robot question
The long-term goal is clear: use these agents to control physical robots. Joe Marino, a research scientist at Google DeepMind, argues that skills SIMA 2 has learned — navigating spaces, using tools, collaborating with humans — are essential building blocks for future robot companions. Even simple actions in games involve complex sequences of steps. Lighting a lantern, for instance, might require finding the right object, positioning yourself correctly, and executing the right command.
But there are real limitations. SIMA 2 struggles with tasks requiring many steps over long periods. It only remembers recent interactions (the team deliberately cut its long-term memory to make it more responsive). And it's still far worse than humans at using a keyboard and mouse.
AI researchers are split on whether this translates to better robots. Julian Togelius at New York University notes that previous attempts to train single systems across multiple games have mostly failed — controlling games from visual input alone is genuinely difficult. But he's cautiously optimistic about the robot application, though he points out that real-world physics don't work like video game rules. Matthew Guzdial at the University of Alberta is more skeptical, noting that most games use similar keyboard-and-mouse controls, and that real-world camera images are far messier than game graphics designed for human eyes.
Google DeepMind isn't stopping here. Marino mentioned plans to create an "endless virtual training dojo" where Genie 3 generates worlds for SIMA to learn in, with Gemini providing feedback. "We've kind of just scratched the surface of what's possible," he said.
