IntelligentAI: An Illusion of Automation
Turk Mechanical
The AI-driven drive-through system installed in Checkers and Carl’s Jr. fast-food outlets is not the epitome of complete automation, as commonly assumed. Contrary to widespread belief, as detailed by Bloomberg, the AI system heavily depends on a network of external contractors who regularly step in to ensure the seamless processing of customer orders.
Recent revelations to the US Securities and Exchange Commission by Presto Automation, the supplier of these drive-through systems, expose the use of “off-site agents” located in countries like the Philippines. These agents play a vital role in aiding the “Presto Voice” chatbots in more than 70% of customer interactions.
This substantial human involvement challenges the idea of flawless “automation” and highlights tech companies’ tendency to exaggerate the capabilities of their AI technologies while downplaying the significant human element involved in the background.
Shelly Palmer, the CEO of a technology consulting firm, astutely points out the prevalent hype surrounding AI, underscoring the lack of genuine comprehension about its functionalities. He remarks, “Everyone believes that AI is magical in some way.”
Evolving Narratives
In July, the SEC launched an investigation into Presto’s AI systems, particularly focusing on specific features, as reported by Bloomberg.
While the details of the inquiry remain undisclosed, noticeable shifts in Presto’s marketing strategies have coincided with the investigation.
Earlier, Presto’s website asserted in August that its AI could manage 95% of drive-through orders “without any human intervention,” a claim that contradicts the current operational truth. Subsequently, in November, the statement was revised to indicate 95% accuracy “without any restaurant or team interference,” aligning more closely with the actual process but still carrying an aura of deception.
This modification mirrors Presto’s broader shift to a new marketing approach highlighting “humans in the loop,” designed to support remote workers and ease the burden on in-house restaurant staff. It appears that the AI industry, overall, is adopting intermediary measures to appease dissatisfied customers.
During a recent earnings call, Presto CEO Xavier Casanova, as cited by Bloomberg, recognized the essential nature of human oversight in the system’s functioning, stating, “Our human agents enter, review, test, and correct orders. Human intervention will always be essential to ensure order accuracy.”
Recognizing the Limitations
The exaggerated hype surrounding AI often masks its real capabilities and the extensive human contributions needed for its advancement. Many tech companies prefer to minimize the reliance on millions of underpaid workers in developing countries to sustain the effectiveness of their AI systems.
Even OpenAI’s ChatGPT heavily relies on a multitude of “grunts” to facilitate the robot’s learning process. Nevertheless, the backstage reality is frequently shielded from investors who have collectively invested over $90 billion in the sector this year without a comprehensive understanding of the underlying dynamics.
Brian Dobson, a scientist at Chardan Capital Markets, emphasizes the significance of investors comprehensively grasping an AI firm’s capabilities and limitations, stating, “It underscores the importance of investors truly understanding what an AI firm can and cannot do.”
