ANZ Bank’s Experiment with GitHub Copilot
The trial conducted by the Australia and New Zealand Banking Group (ANZ Bank) involving GitHub Copilot has proven to be a valuable experience for the software engineers, leading to enhanced productivity and code quality within the organization. The successful integration of this AI programming assistant into their production workflows showcases the potential benefits of leveraging such innovative tools in the financial sector.
ANZ Bank, headquartered in Melbourne, undertook an internal trial of GitHub Copilot from mid-June 2023 to the end of July of the same year, involving 100 professionals out of their 5,000 workforce. This six-week trial comprised two weeks of training followed by four weeks of engaging in code challenges to evaluate the impact of GitHub Copilot, integrated with Microsoft Visual Studio Code, on the programmers’ efficiency, code quality, and software security.
The outcomes of the trial were documented in a report titled “The Impact of AI Tool on Engineering at ANZ Bank, An Empirical Study on GitHub Copilot within Corporate Environment.” The report, co-authored by Sayan Chatterjee, an ANZ sky engineer, and Louis Liu, the company’s engineering AI and data analytics capability area lead, references various studies on the productivity of Copilot programming.
Noteworthy findings from the study include the significant increase in productivity by more than 55% as reported by a Microsoft study, emphasizing the efficiency boost achieved through coding with an AI assistant. However, an ACM/IEEE study highlighted a trade-off with Copilot generating more code but potentially compromising on quality compared to human-built programs.
Despite acknowledging the inherent risks associated with AI tools concerning academic integrity, data security, and privacy, ANZ Bank conducted a thorough evaluation in collaboration with legal and security experts to mitigate potential challenges. The study primarily focused on assessing the impact of Copilot on code quality, security, developer productivity, and overall sentiment among participants.
Participants were tasked with solving Python coding challenges, with one group utilizing Copilot while the control group relied on traditional methods like online searches or Stack Overflow. The Copilot users demonstrated a 42.36% faster task completion rate and produced more reliable code with fewer errors, indicating a positive impact on productivity and code quality.
While the study did not yield conclusive data on code security, it noted that Copilot did not introduce significant vulnerabilities into the codebase. Participants appreciated the assistance provided by Copilot in code review, documentation generation, and testing, streamlining their development process and reducing debugging time.
Interestingly, the study revealed that Copilot was beneficial across all proficiency levels but particularly advantageous for expert Python programmers, showcasing its potential to enhance performance on complex tasks. Despite the positive feedback from users, the report suggests room for further improvement in Copilot’s functionality based on the participants’ feedback.
In conclusion, ANZ Bank is inclined to incorporate Copilot into its production processes based on the favorable outcomes observed during the trial. The report indicates a widespread adoption of Copilot within the organization, with ongoing research to delve deeper into its impact on performance. However, it is essential to consider the contrasting views within the research community regarding the impact of AI support on source code quality.
