How many P’s are in Google? According to Google, there are two.
There’s also “one ‘r’ in the word ‘poop’,” Google’s AI Overview says, and two ‘d’s in the word journalism, however it spells: journadism. Google has at least identified one P in the US president’s last name, but spelled it as trpum.
You didn’t have to be a prophet to predict that Google’s AI-forward search overhaul would go awry. We have done this before. The first time Google added an AI Overview to Research, the feature ended up citing funny posts from The Onion and Reddit, advising people to eat rocks and put glue on their pizza.
This time around, as Google reiterates its commitment to making productive AI the core of its 29-year-old flagship product, it’s no surprise to see it stumble.
“Numbering has been a known challenge for LLMs, and we are working to address this issue,” Google told TechCrunch in an emailed statement.
These spelling mistakes may seem familiar. LLMs, the type of artificial intelligence that powers chatbots and other text-generators, are not designed to understand spelling. It’s been a joke for years that whenever a company unveils a new AI model, you have to ask them how many or in the name strawberry. These AI models — which can code an app in seconds, or solve problems that have stumped mathematicians for decades — are almost like kindergarten in spelling.
Google’s AI overview The woes reach beyond spelling mistakes. Google has already released an issue last week where a search for the word “apathy” will bring up what appears to be a dictionary definition of the word, only the definition shown is, “Understood. Let me know whenever you have new information or a question!” But these spelling mistakes are still funny because they are so difficult to eliminate.
As the researchers have previously explained when we asked about these spelling problems, AI does not see sentences as linguistic units made up of words and letters. Most LLMs are built on transformer models, which break text into tokens, which can be full words, syllables, or letters, depending on the model. Instead of “reading” like a human, the AI converts the text into numerical representations of itself, which are then contextualized to help the AI come up with a logical answer.
“LLMs are based on this transformer design system, which doesn’t really read the text. What happens when you put in information is that it’s translated into coding,” Matthew Guzdial, an AI researcher and assistant professor at the University of Alberta, told TechCrunch. “When it sees the word ‘the,’ it has the code for what ‘the’ means, but it doesn’t know about ‘T,’ ‘H,’ ‘E.’
The token-based architecture that powers LLMs like Google AI’s overview is inherently limited, and researchers weren’t optimistic that they could solve the spelling problem.
“It’s hard to answer the question of what exactly a ‘word’ should be for a language model, and even if we get human experts to agree on a complete token vocabulary, the models will probably still find it useful to ‘mix’ things up even more,” Sheridan Feucht, a PhD student studying large-scale interpretive modeling at Northeastern University, told TechCrunch. “My guess would be that there is no such thing as a suitable tokenizer for this type of complexity.”
This is not an urgent problem in the minds of researchers, as the use of LLMs does not reach their potential to spell. But these apparent failures help us remember that AI is not perfect, even if it can sometimes seem like an omniscient force beyond our understanding. We cannot blindly trust AI results without double-checking their accuracy.
If you shop through links in our articles, we may earn a small commission. This does not affect our editorial independence.
