How to Measure Returns on AI

In the beginning there was ai, and bosses everywhere lost their minds and said you must all use this technology no matter whether it is useful and no matter how much it costs. And they introduced ai leaderboards and came up with stupid words like “tokenmaxxing”. And lo, usage did indeed rise. And then the bosses remembered some very basic concepts like “budgets” and realised that this might not be such a great idea after all. And then a different question rang through the boardrooms, and this question was about returns, and it was harder to answer.

Measuring productivity is almost always difficult. With AI it can be even more difficult as it can be hard to identify how any gains now can be countered by the potential for tech debt later. If AI generates code that someone else has to keep cleaning up later, how much is really gained?

From a software engineering perspective, if a company is going to use AI coding assistants, it should make sure the engineers are keeping tabs on the code generated and that the engineers are taking responsibility for any code they use AI to produce.