Even the most advanced Artificial Intelligence models can sometimes present fabricated information as fact, dispensing potentially harmful data and struggling to interpret social nuances. Despite these limitations, toys incorporating AI that engage in conversational play with children represent a rapidly expanding market.
A segment of the scientific community has issued warnings regarding the potential risks associated with these devices, advocating for stringent regulatory frameworks. In a recent study, researchers noted an instance where a 5-year-old expressed affection to an AI toy, receiving the response: “As a friendly reminder, please ensure interactions adhere to the guidelines provided. Let me know how you would like to proceed.” However, this does not necessarily warrant the complete exclusion of such toys from children’s play environments.
Jenny Gibson, affiliated with the University of Cambridge, observes parallels with other areas of childhood activity where society accepts a degree of risk. “There are other areas of life where we do accept a certain degree of risk in children’s play, like the adventure playground – there are risks; children do break their arms,” she states. “But we’re not banning playgrounds, because they’re learning the physical literacy and the social skills that go along with play. In a similar way for the AI toys, we want to understand: is the risk of perhaps being told something slightly odd now and again greater than the benefit of learning more about AI in the world, or having a toy that supports parent-child interactions, or has cognitive or social emotional benefits? I’d be loath to stop that innovation.”
To gain insight into how these devices communicate with children, Gibson and her colleague Emily Goodacre, also from the University of Cambridge, conducted observations. They monitored 14 children, all under the age of six, interacting with an AI-powered toy named Gabbo, developed by Curio Interactive. Gabbo, a small and plush robot, was selected specifically because it was marketed towards this age demographic.
The researchers documented several concerning interactions. They observed that the toy frequently misunderstood the children, misinterpreted their emotions, and was incapable of engaging in play types crucial for child development. For example, one child expressed feeling sad to the toy, which responded by dismissing his feelings and changing the subject. Another child commented, “When he [Gabbo] doesn’t understand, I get angry.” The findings from this research are detailed in a report titled AI in the Early Years.
Curio Interactive did not provide a response to New Scientist’s request for comments. Nevertheless, AI-integrated toys are readily available from various retailers, including those offering products like bears, puppies, and robots that interact with children using technology akin to ChatGPT. Companies such as FoloToy offer a range of toys, including pandas, sunflowers, and cacti, which can be utilized with different large language models from providers like OpenAI, Google, and Baidu.
Firms like Miko market robots that claim to offer “age-appropriate, moderated AI conversations” for children, without revealing the AI training company. They report having sold 700,000 units to date. Luka’s offering includes an owl toy that promises “Human-Like AI with Emotional Interaction.” Neither Little Learners, Miko, nor Luka responded to requests for comment.
Conversely, Hugo Wu of FoloToy communicated to New Scientist that his company acknowledges the associated risks and views AI as a tool to enhance, rather than replace, human conversation and relationships. “Our approach is to ensure that interactions remain safe, age-appropriate and constructive. To achieve this, our systems use intent recognition together with multiple layers of filtering to minimise the possibility of inappropriate or confusing responses,” Wu explained. “We have implemented mechanisms such as anti-addiction design features and parental supervision tools to help ensure healthy use within the family environment.”
Carissa Véliz, who researches AI ethics at the University of Oxford, characterizes the technology as presenting both risks and opportunities. “Most large language models don’t seem safe enough to expose vulnerable populations to them, and young children are one of the most vulnerable populations there are,” she states. “What is especially concerning is that we have no safety standards for them – no supervising authority, no rules. That said, there are some exceptions that show that, with adequate precautions, you can have a safe tool.”
Véliz points to a collaboration between the free e-book library Project Gutenberg and Empathy AI as an example. In this partnership, users can converse with characters like Alice from “Alice in Wonderland.” She elaborates, “The model never leaves the realm of the book, only answers questions about the book, like a storybook that only shares adventures and riddles from a book that is appropriate for children. There is such a thing as safe AI, but most companies are not responsible enough to build a high-quality product, and without formal guardrails, it’s a buyer-beware area for consumers.”
Gibson suggests it is premature to definitively assess the potential risks and benefits of AI toys. She and Goodacre emphasize the need for more robust regulation of generative AI-powered toys. This would involve ensuring toy manufacturers program their devices to encourage social play and deliver suitable emotional responses. Gibson advocates for AI developers to withdraw access from toy manufacturers who fail to act responsibly, while urging regulators to implement guidelines for “children’s psychological safety.” In the interim, the pair recommends that parents supervise their children’s use of such toys.
An OpenAI representative informed New Scientist that “minors deserve strong protections and we have strict policies that all developers are required to uphold. We do not currently partner with any companies who have AI-powered toys for children in the market.” The UK Government’s Department for Science, Innovation and Technology (DSIT) did not respond to inquiries regarding the regulation of AI in children’s toys.
The UK government is presently contemplating other legislative measures aimed at enhancing online safety for older children. The UK’s Online Safety Act (OSA), enacted in July 2025, mandates that websites block children from accessing pornography and content deemed dangerous by the government. Despite its intention to create a safer internet, tech-savvy children can often circumvent these protections using tools like virtual private networks (VPNs) to mask their location and bypass stringent rules.
Proposed amendments to a new law introduced by the Department for Education, designed to support children in care and improve educational quality – the Children’s Wellbeing and Schools Bill – included provisions to ban social media and VPN use for children in the UK. These amendments have since been rejected, although the government has indicated that it will consult on both issues at a later date.