Do you have knowledge about the recent offering from Google? They have integrated artificial intelligence into it, giving it a “supercharged” boost. However, this enhancement seems to have led to some unfavorable outcomes.
In the traditional version of Google, if you were to ask, “What is Mark Zuckerberg’s online worth?” you would receive a concise and accurate response: “169.8 billion USD”.
Contrastingly, when posing the same question to the “experimental” new iteration of Google search, the AI-generated response is as follows: Zuckerberg’s net value is “\(46.24 per hour, or \)96,169 per year. This is equivalent to \(8,014 per month, \)1,849 per week, and $230.6 million per day”.
Clearly, these figures do not align logically.
The erratic behavior of Google can be attributed to the scans it conducts, which are then presented by its AI. Over the course of nearly 11 months, the company has been trialing this new Google version, referred to as the Search Generative Experience (SGE), with volunteers. Recently, AI-generated responses have started appearing in the primary Google results, even for users who have not opted into the test.
While the fresh Google shows promise, it frequently fabricates facts, misinterprets queries, presents outdated information, and tends to ramble on without much coherence. Furthermore, experts have noted that the AI often elevates low-quality websites as reliable sources of information.
Typically, reviewing an unfinished product is not standard practice. However, given that Google’s potential has been under scrutiny for almost a year, the decisions made now will impact how people access information. The fundamental concept behind the ongoing AI revolution is that technology can potentially eliminate the need for extensive research by providing direct answers. If a corporation with the resources and computational power of Google struggles to make this concept work, who can?
SGE combines the functionalities of a robot with those of a search engine. In addition to the standard features, SGE provides clear answers to questions along with relevant links for further exploration.
SGE addresses the growing trend of individuals, including myself, turning to AI like ChatGPT for intricate queries or when browsing through multiple websites seems cumbersome. Onely, a search marketing firm, asserts that consolidating pros and cons, pricing details, and other relevant data in one location can reduce an individual’s overall research time by 10 to 20 times.
Given our diminishing attention spans, an all-knowing answer bot appears to be indispensable. However, Google still has room for improvement. While we anticipate challenging queries, the AI responses from Google often take an unreasonably long time to generate. Google needs to strike a balance in the fragile internet ecosystem, where its AI solutions may influence readers who opt for the easier route instead of conducting thorough research.
Crucially, the new Google must consistently and accurately address our queries. This aspect was the focus of my evaluations, which consistently revealed instances where the AI-powered Google performed below expectations.
Assessing Google’s AI-generated responses
Typically, when conducting a Google search, users seek concise information or relevant websites. The new Google tends to be verbose, which can be frustrating in daily use.
For instance, when inquiring about what converters consume, the AI response delved into an elaborate explanation about fictional robots not requiring food or drink but necessitating some form of fuel. In contrast, the older Google succinctly provided the answer I sought: Energon. This critical information is buried within the verbose responses from the new Google.
This issue extends beyond fictional scenarios. SE Ranking, an SEO-focused company, conducted tests with 100,000 search queries using SGE and found that the average response length was 3,485 characters — roughly a third of the length of this column. Google faces the challenge of determining when its AI should remain succinct; at times, SGE prompts users to click a “generate” option before providing an answer.
Accuracy is paramount in search results. Google claims that SGE surpasses ChatGPT in this aspect due to its up-to-date knowledge base. However, my exploration revealed that the new Google struggles with recent events. For instance, three days post the latest Academy Awards, a search for “Oscars 2024” erroneously indicated that the event was ongoing and listed some winners.
Nothing shakes confidence in Google’s AI solutions more than witnessing it confidently present fabricated information. This was evident when I requested recommendations for an award-winning series to write for The Washington Post, and the AI response provided a stranger’s name and directed me to a different website.
Furthermore, there were instances where SGE spontaneously generated information about non-existent facts. When inquiring about a San Francisco restaurant named Danny’s Dan Dan Noodles, the AI response detailed its purported “crazy wait times” and described its cuisine. The concerning aspect is that this fictional eatery, named after a personal favorite dish, prompted Google’s AI to invent information about it.
Current AI technology grapples with issues such as hallucinations related to real and fictitious topics. Despite the disclaimer stating that “Generative AI is experimental,” Google must address the challenge of discerning when to admit uncertainty.
Questionable Source Selection
In its quest to provide comprehensive answers, Google’s AI must determine the reliability of information sources. However, there are doubts regarding its judgment in this regard.
The earlier discrepancy in Zuckerberg’s net worth highlights this issue. While a knowledgeable researcher or traditional Google might direct users to Forbes’ list of billionaires, the new Google inexplicably referenced a dubious job board on ZipRecruiter for “Mark Zuckerberg Jobs”.
During my assessments, the prevalence of unreliable sources was evident. Following Onely’s recommendation, I inquired about the reliability comparison between Apple iPhones and Samsung phones. Instead of providing insights from reputable sources like professional journalists or repair experts, the AI cited random opinions from social media. This raises concerns about the authenticity and credibility of such information.
“Google SGE operates under a different framework compared to the conventional search engine we are familiar with today,” remarked Tomek Rudzki, Onely’s head of research and development.
SEO firms have attempted quantitative analyses of SGE’s performance, albeit constrained by Google’s stipulations on test accounts. Their findings underscore the disparity between the old and new versions of Google. Authoritas, an SEO software company, conducted tests on a thousand shopping-related terms in late March, revealing that the AI-generated responses often omitted the top traditional search result, occurring 77% of the time.
In SE Ranking’s analysis of 100,000 keyword searches, it was noted that platforms like Quora, LinkedIn, and Reddit featured prominently, with Quora being the most frequently linked source. However, the prevalence of these sources in academic contexts, such as eighth-grade term papers, raises concerns.
For searches on tech-related topics, SE Ranking identified simplilearn.com as the most-linked domain. This unfamiliar site describes itself as an “online boot camp”.
“This trend not only compromises the quality of search outcomes but also diminishes traffic and revenue for numerous small businesses, including affiliate websites,” highlighted Anastasia Kotsiubynska, head of SEO at SE Ranking.
Work in Progress
Google emphasizes that SGE is an opt-in experiment. Despite surpassing the target launch date in December, Google has not provided an update on the broader rollout of SGE. It is plausible that Google deems SGE unreliable, sluggish, or unprofitable, necessitating significant modifications in the future.
Despite the potential perception of lagging behind in the AI race, Google’s cautious approach is prudent. While Microsoft’s Bing introduced similar AI enhancements in February 2023, it is primarily recognized for its erratic behavior.
In an interview, Elizabeth Reid, a Google vice president overseeing SGE, characterized the project as an evolving endeavor.
“We’re dedicated to refining the user experience to ensure it meets our standards,” she explained. Various factors, such as latency, accuracy, and utility, are pivotal in this process. Reid emphasized the nuanced nature of this undertaking, where the effectiveness of AI varies across different scenarios.
Upon sharing the instances highlighted in this assessment, Reid assured that SGE’s hallucination rates are minimal and have shown a significant decrease since the launch in May. However, specific figures were not disclosed.
Acknowledging the challenges associated with this technology, Reid emphasized Google’s commitment to addressing these issues. The inclusion of links alongside AI-generated responses is crucial for users to verify information independently.
A suggestion for Google would be to disclose its accuracy data before expanding SGE to a wider audience. Given the substantial volume of daily searches, even a minute margin of error can result in a significant dissemination of misinformation.
Google’s focus also extends to expediting responses to queries while providing additional context. Reid highlighted the importance of swiftly addressing the core question and subsequently offering elaboration.
Google refuted external research findings on SGE, citing limitations in the scope of searches analyzed. However, proprietary data was not disclosed.
Reid clarified that SGE upholds the same standards as the traditional Google, striving to prioritize high-quality content. While a broader range of sources may emerge, the primary goal remains ensuring the prominence of reliable information.
Navigating the credibility of information sources poses a challenge for users. What instills confidence in Google’s current AI technology, referred to as LLMs (large language models), to handle this task effectively?
Acknowledging the imperfections inherent in AI, Reid emphasized Google’s commitment to a thoughtful and meticulous approach, given the paramount importance of the company’s reputation.
The implications of these advancements extend to the future of information dissemination.
