Are brain waves the next incarnation of physical AI?

The frontier of virtual reality AI is a game of Jenga in a warehouse in San Leandro, California.

That store houses Encord, a company that builds data tools used to train AI models. Andrew Ceja is a pilot—the company’s name for its robot trainers—and he carefully pulls wooden blocks into a moving tower while wearing a headset with a camera that tracks what he sees. That alone is common in collecting data for a robot’s training, but this headset includes sensors that measure his brain waves as he carefully dismantles the tower.

Encord is one of a small but growing number of startups betting that the next real limitation to humanoid and warehouse robots won’t be architectural modeling but rather a severe lack of real-world physical training data. Rather than just helping robotics companies manage the data they have, Encord is building a business around generating the data they don’t.

The brain headset worn by Ceja was developed by Zander Labs, a German neuroscience startup that is betting that measuring brain activity – detecting mental states such as error, intention and surprise – can create very useful data for training models. Encord’s work with Zander is currently experimental; Encord says the goal is to create a set of tagged brain wave data, run it through customer robotics models, and test whether it actually improves performance before deciding whether to scale it up.

Lucas Gehrke, the Zander neuroscientist leading the work, says the amount of brain activity used at any given time during a given task gives clues to modelers trying to figure out when they need to use their models for maximum effort.

This is the “bleeding edge” of the effort to solve the robot data problem, according to Vineeth Velmurugan, Encord’s head of robot learning. A veteran of OpenAI’s robotics lab and Berkshire Grey, a warehouse automation company, Velmurugan joined Encord to build the company’s internal data generation team.

Encord was founded to help companies building machine vision applications interpret data and test models. As their customers—Velmurugan says he works with several leading robotics companies but is not authorized to name them—started using learning to manipulate the robot, managers realized they would have to generate the training data themselves, rather than simply manage it. “The details are not there,” said Velmurugan.

Betting that productive AI can do for robots what is done for chatbots continues to work on this same wall. LLMs are built on an online text, and so on. Finding the same raw materials for teaching neural networks about physical manipulation is a challenge: self-driving car companies are collecting them themselves, but that’s hard to measure. Training from video can work, but it lacks the reliability of real-world data. Velmurugan says it would take a data set something like five times the size of YouTube’s video envelope to break through—a measure that helps explain why data generation itself has become a business rather than just a research problem.

Feed your egocentric data needs

Companies building robot brains are now turning to two main sources: “Egocentric” video collected by workers wearing cameras, often augmented with additional camera angles and other metrics, and collecting data from remotely operated robots. Encord does both, drawing on egocentric data from several industries around the world, and using its San Leandro facility to test new methods, such as brain waves, or collect data sets around specific skills for fine-tuning.

When TechCrunch visited, pilots were using leader-following instruments — a pair of robotic arms, one of which is directly controlled by a human operator and mimics its movements — to generate data about tasks like pouring coffee from a pot into cups (very small) and stacking poker chips. “All the humanoid companies have asked for these pieces,” Velmurugan said.

Storage racks held boxes of fake flowers in vases, books, plastic vegetables, litter trays and scoops, bags and bundles of rope, the stock in trade for training fraudsters for homework.

At one of these stations, another pilot, Sofia Infante, directs robotic arms to connect and disconnect ethernet cables behind a server—the kind of work data center operators would like to do themselves, if only robots could use them with the necessary precision. Looking at the back of the controls, I was able to see why that’s still out of reach: The pins are much smaller than human fingers and don’t have the degrees of freedom we take for granted in our arms.

Another new data method developed by Encord uses a set of sensors strapped to the arm to detect electrical signals from the muscles. Video taken of a person’s hands changing objects usually doesn’t capture the entire hand, but Velmurugan hopes to create a 3D representation of where the hand is at any given time based on the arm’s sensors, creating a more robust understanding of the models.

Encord datasets contain literal descriptions of each video’s content—“the right hand tightens a bolt”—to aid LLM-based models in understanding what’s going on. Velmurugan estimates that this kind of dense annotation costs 100 times as much as “data junky ego” to train specific tasks, and costs only 20 times to produce, which is a good trade-off, on paper.

But “20 times more” is still real money, and that’s the catch: removing text from the Internet, the way LLM makers built their models by outsourcing to Stack Overflow and the rest of the web, external labs are less expensive. Generating physical training data does not, and that is the limitation of physical-AI-like LLM comparisons. This type of data must be generated, not just collected, and that changes the economics of building these models.

Velmurugan says progress is being made—with Encord’s visibility into systems across the industry, you can see startup labs and frontier labs figuring out what works and what doesn’t in developing physical AI models. That point—which sits among many robotics companies at once—is also part of Encord’s pitch. It can see which data strategies are gaining traction across the industry before a single customer knows it.

That will keep a dozen pilots at the Encord facility busy. Both Infante and Ceja are part of a growing body of work developing the building blocks of neural networks; they previously worked at Scale, another AI data annotation company, before joining Encord.

Ceja had worked at a waste management company when his interest in technology found him in charge of keeping a robotic waste filter running smoothly. Now, as the Jenga tower collapses, he says he enjoys the challenge of solving robot training tasks —“It’s something new every day!”