Would you get in a robotaxi trained by GTA V?
Can video games be used to train computers and tools of the future?
Issue #25 of LFG: Learning from Games explores how researchers used Grand Theft Auto V to train a computer to drive a virtual car. We’re still not quite ready for autonomous cars, but maybe GTA V (and VI?) can help us get there.
Hello JOMT Reader!
I started writing this newsletter to share with you all how video games can positively affect people’s well-being. Just recently, I shared some studies about how nursing students did better on tests after playing video games, how freedom of choice in video games leads to better learning results, and how skills gained while playing video games can be used in real-life.
While the focus on human health and education is important, there is a way video games can affect us indirectly, through the technology that we interact with everyday.
That technology, including Amazon and Netflix’s recommendation systems, or the plethora of AI models that are coming out of the woodwork, to robots like Boston Dynamics’ Spot, all required a mountain of high-quality data to work as well as they do. Talking about Spot, this 5-min video from Boston Dynamics perfectly summarizes how they approached training for Spot.
At the heart of training any model or robot is data. Quantity and quality of data are both equally important, especially for models and robots that interact with the real world. But getting good data can be time-consuming and challenging.
Take a look outside your window right now. Now imagine the same view at night time, or under different weather conditions, or in a different season. Or with construction workers, dog walkers, and commuters coming in and out of that view. Despite all of these changing conditions (and their combinations), you know that it’s still the same view out of your window. But models and robots don’t know that, at least not without some training.
To try and get all the data necessary for models and robots to understand what you understand as “the view out of my window”, you might set up a camera to record all the changes that happen. But to capture seasonal changes, you’d need at least a year’s worth of recordings. And that’s just to account for the seasons. What happens if an ambulance passes by during a rainy night?
Enter simulations — something we’re all very familiar with as gamers. These virtual worlds allow you to create any combination of world environments, in a much shorter time frame. That, in turn, means these simulated environments can be used to generate data that might take years in the real world.
Although simulations are great for creating situations that might be difficult to find in real life, they also tend to be boring. Boring in the sense that it isn’t very realistic.
Now enter video game worlds — more specifically, open-world games with (mostly) dynamic environments. Non-playable characters move around, the weather and time of day might change, and parts of the environment can change based on your actions. We’ll leave the discussion about realism to Alex Antra and Exploring The Games in the post below; for the purposes of training models and robots, realistic or not, open-world games are a great source of data.
The world has been talking about self-driving cars for several years now and while I don’t think we are ready to hand over the steering wheel, we can certainly think about ways to train such cars. Researchers in Italy tackled this training problem from a video game angle, using the world of Grand Theft Auto V.
For those of you who don’t know, Grand Theft Auto V (and its upcoming series successor GTA VI), is an open-world, immersive game set in the fictional world of Los Santos. One of the key activities in this game is the ability to drive around Los Santos in different (probably stolen) cars, causing mayhem and evading police.
Using the game environment and some modding tools, the researchers sought to generate data using GTA V and see how it performed in a simulated driving test.
Can GTA V be used to generate data?
The big question is if a video game world like GTA V can be used to generate data for training. After all, the world of pixels is very different from the real world that we live in. If we show a day time image of a scene, can a computer recognize a night time image of the same scene?
Here are some examples of the types of images that were used. The first set is a real world image of the same street during the day and at night.
A computer trained on real life images was pitted against one that was trained on similar day/night images taken in GTA V (below).
After training, the computers were shown new day time images and asked to identify the corresponding night time image. The computers were evaluated for accuracy in predicting the correct image among a top-5, top-10, top-20…top-50 list (the numbers in the horizontal axis in the graph below, called k, is how the researchers chose the top lists).
Surprisingly, the computer trained using both real-life and GTA V images was more accurate at identifying night time images when shown day time images, compared to computers trained using either type of image alone. The researchers also tested whether day time images could be identified when night time images were shown.
In the reverse case, the computer trained using GTA V images was more accurate at identifying day time images using night time images as a reference, compared to the computer trained using real-life images. As was the case for the day/night combination, the computer trained using both sets of images was more accurate than computers trained using either type of image alone.
The key takeaway? Images from video games can be used to train computers!
But how good is it able to navigate/drive?
Being able to identify images is one thing. But using this ability to drive around is quite another. We’re not ready to just load this onto a self-driving car and let it cause real-life GTA mayhem. Instead, we can continue using the world of GTA and evaluate whether these computers are any good at navigating around a simulated route.
The specifics of how the researchers tested and measured the computer’s ability to navigate around Los Santos involves a lot of complicated math. In very simplified terms, the computer was shown images of a route that it had never seen before and then asked to drive that route in GTA V.
For most of the paths, the computer did an exceptional job following a route based only on images of the route, as the green and orange lines almost completely overlap each other.
Not only can GTA V be used to train computers to drive, but it can also provide a safe environment to test those systems!
What’s the catch?
As flexible and safe virtual game worlds can be, it lacks much of the uncertainty and unpredictability of the real world. In a self-driving car, the car would be relying on sensors and cameras to take in information about its environment. Those sensors and cameras can experience interference from excessive noise (like a siren) or dirt on the camera lens. Neither of these interferences were tested in this game world system. So how well these computer models perform in a game world may not reflect how they perform on real-world streets.
But it’s a start. Combined with real-world data, I think we’re just beginning to scratch the surface of possibility in using video game worlds to help us in real life.
And it’s not just for self-driving cars. I could imagine using video game worlds and environments to train robots to help surgeons do complicated operations. Or, as I’ve mentioned in the past, use changes in our in-game behaviour as a way to predict mental health issues or other conditions.
I am looking forward to a future where my game-playing session ended with a short report on my health and things I should keep an eye on! And it all starts with supplementing computer model training with video games.
If you want to read more about this study, you can access the article here for free: https://arxiv.org/pdf/2502.12303
If you liked what you read, please consider giving this post a like and sharing it with your community!
When GTAV was released my friend was playing it all the time, but as he noticed, he can sit behind the wheels everytime, because he starts driving like a maniac :D
Interesting piece, it begs the question whether digital twinning and game engines will start to overlap more and more as these things need to be simulated.