Where's the Shovelware? Why AI Coding Claims Don't Add Up
One dev crunched the numbers on AI coding -- and found absolutely 0 noticeable impact:
I discovered that the data isn’t statistically significant at any meaningful level. That I would need to record new datapoints for another four months just to prove if AI was speeding me up or slowing me down at all. It’s too neck-and-neck.
That lack of differentiation between the groups is really interesting though. Yes, it’s a limited sample and could be chance, but also so far AI appears to slow me down by a median of 21%, exactly in line with the METR study. I can say definitively that I’m not seeing any massive increase in speed (i.e., 2x) using AI coding tools. If I were, the results would be statistically significant and the study would be over.
That’s really disappointing.
Tags: productivity chatgpt github technology business culture work llms metr ai