Summary
-
BBC study in partnership with the European Broadcasting Union reveals that AIs still fail seriously when summarizing news.
-
Gemini, from Google, was the system that recorded the most errors and inaccuracies in responses (76%).
-
Still, public trust in AI summaries has grown, the survey found.
A survey conducted by BBC in partnership with the European Broadcasting Union (EBU) revealed that the main artificial intelligence assistants on the market are still far from offering reliable journalistic summaries.
The study analyzed more than 3,000 responses generated by tools such as ChatGPT (OpenAI), Copilot (Microsoft), Gemini (Google) and Perplexity, concluding that 45% of responses contained significant errors — from incorrect information to problematic fonts.
According to the study, 31% of responses had serious source attribution problems, while 20% included serious inaccuracies, such as outdated data or invented facts. When minor slips are considered, the response rate with some type of error reached 81%. Gemini, from Google, was the worst rated, with flaws identified in 76% of responses, double the rate recorded among competitors.
The researchers pointed out that the main cause of errors lies in the way these systems deal with recent information and external sources. Gemini, for example, presented inaccuracies in 72% of responses, a number three times higher than that of ChatGPT (24%).
Some curious cases were cited: a response from ChatGPT even stated that Pope Francis was still alive weeks after his death, while Gemini assured that no NASA astronaut was ever trapped in space — something that has already happened.
The study is considered the largest of its kind ever carried out, with the participation of 22 public media outlets from 18 countries. It comes shortly after OpenAI admitted that its models tend to respond confidently even when they are unsure, behavior that encourages what experts call hallucinations — fabrications of false facts or details.
Despite the flaws, the use of chatbots to consume news has grown. A parallel survey by the Ipsos institute, also in partnership with BBCshowed that 42% of people in the UK, where the study was conducted, trust AI-generated summaries. Among young people under 35, the number reaches 50%. However, 84% said they would lose trust if they identified just one factual error.
Source: https://tecnoblog.net/noticias/ias-ainda-erram-muito-ao-resumir-noticias-mostra-estudo/
