In this biweekly publication, Roula Khalaf, the head of the FT, handpicks her preferred articles.
Recalls of groundbreaking technology products are not unusual occurrences. The introduction of new technology does not always go smoothly right from the start. However, it is rare for such incidents to involve a pedestrian being hit by the system and dragged across the street for 20 feet.
The recent catastrophic event that has disrupted operations at Cruise, the autonomous vehicle division of General Motors, represents a significant setback that advocates of self-driving cars have long dreaded. If Cruise and its competitors promptly take action and demonstrate that they have truly internalized the lessons learned, it may not necessarily lead to a multi-year setback for the advancement of robotaxis. Nevertheless, there is a risk that it could erode trust in the technology and trigger stringent government interventions.
In early October, one of the company’s vehicles ran over a pedestrian who had been hit by another car and was lying in its path. While the Cruise car did come to a stop, it proceeded to move an additional 20 feet as part of what the company described as “a safety maneuver” to ensure there was no ongoing danger, all while leaving the critically injured pedestrian trapped underneath. Authorities in California, who suspended the company’s operational permits two weeks ago, also alleged that Cruise staff initially failed to inform regulators about the subsequent movement of the vehicle. However, Cruise has refuted these claims.
The unfortunate truths about autonomous vehicles and, by extension, a significant portion of the artificial intelligence sector have been starkly illuminated by this incident. One key realization is that the competitive pressures surrounding potentially transformative technologies inevitably create stress. While Silicon Valley is known for rapidly deploying innovative technologies, more established markets prioritize the gradual development of safety cultures and protocols over time.
Cruise has been in competition in the U.S. market with Tesla and Waymo, two subsidiaries of Alphabet, in the race to establish robotaxi services, with parent company GM aiming for $1 billion in revenue by 2025. Now that all operations have been intentionally halted, Cruise has committed to a complete overhaul of its leadership and safety protocols. While this proactive step is promising, it comes after California authorities prohibited the company from operating. Rather than merely reacting to incidents, Cruise and its rivals must demonstrate their ability to proactively address public health and safety expectations.
The foundational technology behind today’s most advanced AI systems, deep learning, still lacks the sophistication to predict accidents like the one involving Cruise, which is a sobering reality. It may never be completely achievable. This incident serves as a stark reminder that machine learning systems are only as effective as the data they are trained on. Moreover, it is impossible to train them on every possible scenario that may arise, regardless of the volume of data available.
Cruise can utilize this incident as a learning opportunity for all its vehicles moving forward. The company estimates that such a specific incident is likely to occur only once every 10 to 100 miles of travel. However, there will always be new and unforeseen circumstances to contend with.
To regain public trust, Cruise and its competitors must demonstrate not only that their vehicles experience fewer fatalities than human drivers but also that they can avoid serious errors that a human driver would easily prevent. For current technology standards, this benchmark remains quite high.
The incident has also raised concerns about legislation. While there has been significant debate on how AI should be regulated, less attention has been given to who should oversee AI governance and what input ordinary citizens and their elected representatives at various levels of government should have on a technology that could significantly impact their daily lives.
Despite objections from municipal transit authorities, the mayor’s office, and citizen advocacy groups regarding insufficient testing of the vehicles, in the case of Cruise, approval from state regulators in California was deemed adequate to grant its robotaxis unrestricted access to San Francisco streets.
Granting more authority for city-level oversight could lead to a complex web of regulations that may impede the growth of autonomous vehicle manufacturers. However, the repercussions of the Cruise incident suggest that the regulatory framework in California may be inadequate.
The battle is far from over. From the external investigations into its technology and incident management to the deliberate recall of its vehicles, Cruise’s response mirrors crisis management best practices. The company must convince the public that this is a genuine turning point, not merely a lesson in damage control.
