The Google Genie world model can now simulate real streets with Street View

We’ve all pulled up Street View on Google Maps to show a friend what our childhood home looked like, or dropped that little person icon on the streets of Paris to see if we’ve booked a hotel in a cool place. Imagine being able to do that, but in a more immersive, interactive way that allows you to really simulate the road and its surroundings, and even do things like adjust the weather or see what it will look like in a “Day After Tomorrow” scenario.

That’s one of the goals of Google’s latest merger. Starting today, Google DeepMind is connecting Street View to Project Genie, the company’s general-purpose world model that can generate diverse, interactive environments. The new feature was introduced during the Google I/O developer conference.

“It is very powerful for both agents [and robotics] use cases and people to play with, and that’s always been the Genie theory,” Jack Parker-Holder, a research scientist at DeepMind’s open source team, told TechCrunch.

He gave an example of a new robot in London, which rarely sees the sun. Genie, Parker-Holder says, can simulate those rare moments when the sun shines, so that the rays don’t shock the robot when they do.

“At the same time, you can say, ‘I’m going to New York City, but not this time of year,'” he continued. “‘It’s going to snow. I want to see what that block looks like in the snow.'”

For 20 years, Google has been collecting Street View data from cars with cameras and people wearing “tracker backpacks.” The tech giant has collected north of 280 billion images in 110 countries and seven continents.

“With Street View, we have images from all over the world,” said Jack. “You can imagine how powerful it is to combine this rich source of real-world knowledge and data with the ability to simulate worlds.”

Google released its latest Genie 3 world model for preview last August and opened access to the tool to Google AI Ultra subscribers in the US in January, allowing customers to create interactive game worlds using text or image instructions. The goal is to use Genie in educational experiences, games, and robotics training.

The Genie 3 is already helping to power one of Waymo’s simulators to train its self-driving cars for “extremely rare events” like hurricanes or common elephants. Adding Street View data to that could help Waymo prepare to launch in more cities around the world.

Waymo has its simulator relied on to reach 11 US cities and is testing its AI driver in several others. The difference with the Genie, says Parker-Holder, is that all of that comes through the car’s eyes. Street View allows not only to simulate the world stuck in real space, but also to transform the view to other types of agents, such as a human or a robot.

Google is rolling out Street View in Genie to some Ultra users in the United States starting today, with access being rolled out over time. Global Ultra users will get access in the next few weeks, according to each company.

The researchers’ goal is to put this new skill in as many hands as possible, according to Diego Rivas, product manager at DeepMind. He cautioned that Street View and Genie in particular are still experimental, so there is much room for improvement in terms of accuracy.

In the samples the Google team showed me – including an underwater simulation of my neighborhood – the results are stunning and visible, but video game quality rather than photorealistic. The models also do not yet know physics, which means they do not yet understand cause and effect. For example, in the scene of the woman running through the snowy Joshua Tree, she ran through cacti and trees.

Compare that, he says, to Google’s image generator Nano Banana – which can now generate complete text in infographics – or its video generator Veo – which understands that paper boats drift in water, smoke disperses in the air, and fabric drags over forms.

Physics has no hard code for these models; they read it accurately over time at a glance, just as a living creature would.

“I think for this type of model, it’s probably six to 12 months behind video in terms of accuracy and quality, so I think that’s something we’re going to address,” Parker-Holder said.

Jonathan Herbert, director of Google Maps who started on the Street View team as a student 12 years ago, said Genie can’t yet reconstruct a street reliably. He thinks the real breakthrough is the advancement of local AI. When you turn 360 degrees, the AI ​​perfectly remembers and simulates the environment behind you. From then on, the model can build a new location on top of that.

“We’ve been thinking for a long time about how to build the best and richest model in the world on top of Street View data,” said Herbert. “It’s certainly been our vision to use Maps Data in new ways and new types of AI research for a very long time.”