On the Current State of Codegen

What is codegen worth?

I’ve been away for a few months working on starting my software consulting company; Quality Compute, and in that time the state of genAI in the industry has taken a few turns and seems heavily fractured.

Overall, there seems to be mass adoption by working career software engineers, but there is a lot more to the story. Driving the adoption is mandates from executives. This makes it hard to get a clear view of whether or not the adoption is a result of genuine desire among workers. I see both reports that Claude Code and other codegen products are useful and used regularly and reports that people are literally faking their token usage to conform to mandates. Many engineers also report that the code produced by codegen is of poor quality. Others, and I hesitate to refer to them as software engineers at this point, report that they are giving up on reading or understanding the code entirely, and rely on AI for not just code generation, but also review. They have effectively removed themselves from the software engineering process altogether. Whether or not they were ever part of it is another question entirely.

The only certainties that I see in this entire muddle is that software quality has taken a sharp nosedive and that the adoption has failed to materialize literally any compelling software products. The force multiplier of codegen has yet to be documented; it is still a hypothesis. The most compelling study on the effects of codegen on productivity is still the Metr study — that they seemingly desperately want people to forget about and dismiss — which showed that while senior devs working on familiar projects estimated a roughly 20% increase in velocity, they actually suffered a 20% reduction in velocity using codegen products. All other studies fail to represent real important work being done in software. Some simply ask participants to create a basic REST/CRUD web application, something for which templates have existed for decades and would be trivial for codegen to do because it’s almost certainly the primary type of data in the training set, while others don’t even attempt to quantify the impacts of codegen at all and read more as opinion pieces fabricated sometimes by nameless authors.

The greatest proponents of codegen, aside from the “founders” and “AI-engineers” littering social networks without a single compelling product to show for all their bluster, are the executives of NVIDIA, Anthropic, OpenAI, Meta, etc. Let’s remember that these are all people who do not work as software engineers, and haven’t for decades if they ever did. Let’s also keep in mind that these people have obvious financial incentives to push these products as hard as possible, as their investment in the adoption of these products is beyond obscene at this point. Additionally, their pitch is that codegen will take the job of engineers; replacing them entirely. Some of them are on record admitting that they aren’t sure humanity should continue to exist. Beyond that, they cannot seem to identify the utopia that mass AI adoption will create. Will we have “universal, high wages,” or will no one have a job? Will no one need a job? The only tasks that AI promoters seem genuinely interested in replacing are tasks which emphasize creativity; tasks humans want to and must do themselves for them to have any meaning or use; painting, coding, drawing, writing, cinema.

And all this is happening while usage fees are increasing, buildout is stalled or failing, and model improvement is lackluster to say the least. Most of the benchmarks used to hype the technology are available to the companies building the models ahead of time, leading to the models being built simply to game the benchmarks that are supposed to evaluate them. The simple fact is, however, that all this “AI” has yet to provide a single novel advancement in any field or produce a single software innovation or even compelling product.

Many are now reporting that it’s becoming increasingly difficult to justify spend on codegen giving rising costs and a lack of real benefit. Codegen, and much of modern “AI”, may see a day soon when they can’t afford to keep the lights on as customers flee high token costs which mostly create bloated, insecure, and intractable codebases. What happens to all these “engineers” when codegen actually goes away because these companies fail? Most users of codegen report that their skills have severely atrophied, and others say that they need to restrict the method of their codegen use in order to avoid this which eliminates the purported benefits of codegen in the first place. If using it to go faster means your skill degrades as your repository becomes a tangled mess that only AI can actually work in, and using it while retaining your skill requires you to heavily manage, scrutinize, research, and rework its output, then what are the actual benefits of these products?

I have a lot more to say; like how genAI can only reproduce subsets of the training set and genAI is just a tool for obscuring plagiarism. For now, however, I’m glad to see many more people thinking critically about this topic and often coming to similar conclusions. I am also dismayed though that many, especially laymen on codegen, are repeatedly parroting the unexamined talking point that codegen is obviously a useful new technology that will and has improved software engineering. I disagree, and I both haven’t seen any compelling evidence otherwise and have seen a mountain of evidence that it is degrading the practice and degrading the products.

Know that I’m not coming to the topic of AI from an outside position; I’ve literally studied these systems academically and have worked with machine learning algorithms since before I had a job in software. AI has many valuable uses. I simply don’t believe that generating pictures or code are among them.

Quality Compute is a human-first endeavor, as all my endeavors are. I am seeking clients that value skilled and passionate software engineers who are devoted to the best possible process and creating the best possible products.