New York City’s AI Chatbot Faces Criticism for Providing Misleading Information
In a recent development, an AI-powered chatbot designed by New York City to assist small business owners has come under fire for offering peculiar advice that misinterprets local regulations and encourages businesses to violate the law.
Initially flagged by tech news outlet The Markup last week, concerns were raised about the chatbot’s inaccurate responses. Despite these issues, the city has chosen to retain the tool on its official government website. Mayor Eric Adams has stood by this decision, admitting that the chatbot’s guidance is flawed in certain areas.
Launched in October with the aim of simplifying interactions for business owners, the chatbot utilizes algorithm-generated text to address inquiries related to navigating the city’s bureaucratic processes. While the tool carries a disclaimer acknowledging the possibility of producing incorrect or biased information, it also emphasizes that its responses do not constitute legal advice.
Critics have expressed worry over the flawed advice provided by the chatbot, underscoring the risks associated with government adoption of AI systems lacking adequate oversight.
Julia Stoyanovich, a computer science professor at New York University, remarked on the irresponsible deployment of untested software without proper supervision. She highlighted the city’s apparent disregard for accountability in this regard.
Despite ongoing efforts to rectify the situation, the chatbot continues to offer erroneous suggestions. For instance, it erroneously indicated that employers could terminate workers for reporting sexual harassment, concealing a pregnancy, or refusing to cut their dreadlocks. Moreover, it inaccurately stated that businesses could dispose of trash in black garbage bags without mandatory composting, contradicting city waste initiatives.
In a bizarre instance, the chatbot even advised that a restaurant could serve cheese that had been nibbled on by a rodent, stressing the importance of assessing the extent of damage caused by the rat and informing customers about the situation.
Microsoft, the provider of the chatbot’s AI services through Azure, assured that they are collaborating with city officials to enhance the tool’s accuracy in alignment with official documentation.
Mayor Adams defended the chatbot at a recent press conference, suggesting that encountering issues is a natural part of refining new technologies. However, critics like Stoyanovich criticized this approach as reckless and irresponsible.
The incident underscores the concerns raised by experts regarding the limitations of large language models like the one powering the chatbot. These models, trained on vast text datasets from the internet, are prone to producing inaccurate and nonsensical responses.
As public interest in AI chatbots grows, private companies have introduced their own versions, albeit with mixed outcomes. Recent cases involving companies like Air Canada, TurboTax, and H&R Block have highlighted the potential risks of relying on chatbots for critical information.
Jevin West, a professor at the University of Washington, emphasized the heightened responsibility faced by public entities when deploying such models. He stressed the need for caution to prevent users from facing legal consequences due to erroneous advice.
Experts recommend that cities deploying chatbots should limit the scope of inputs to minimize misinformation. Los Angeles, for example, has adopted a curated approach for its chatbots, avoiding reliance on extensive language models.
The challenges experienced by New York City’s chatbot serve as a cautionary tale for other municipalities, prompting a reevaluation of the purpose and utility of such tools. Suresh Venkatasubramanian from Brown University urged cities to carefully consider the implications of using chatbots as substitutes for human interaction, emphasizing the importance of accountability in technological implementations.
