Code analysis firm sees no major benefits from AI dev tool when measuring key programming metrics, though others report incremental gains from coding copilots with emphasis on code review.
Credit: Roman Samborskyi / Shutterstock
Coding assistants have been an obvious early use case in the generative AI gold rush, but promised productivity improvements are falling short of the mark — if they exist at all.
Many developers say AI coding assistants make them more productive, but a recent study set forth to measure their output and found no significant gains. Use of GitHub Copilot also introduced 41% more bugs, according to the study from Uplevel, a company providing insights from coding and collaboration data.
The study measured pull request (PR) cycle time, or the time to merge code into a repository, and PR throughput, the number of pull requests merged. It found no significant improvements for developers using Copilot.
Uplevel, using data generated by its customers, compared the output of about 800 developers using …