Is this because the models improved, or the tooling around models improved (both visible and not visible to the end user).
My impression is that the base models have not improved dramatically in the last 6 months and incremental improvements in those models is becoming extremely expensive.
At the same time, we have the first really useful 1mm token context model available with reasonably good skills across the context window (Gemini Pro 2.5), and that opens up a different category of work altogether. Reasoning models got launched to the world in the last six months, another significant dimension of improvement.
TLDR: Massive, massive increase in quality for coding models. And o3 is to my mind over the line people had in mind for generally intelligent in, say, 2018 — o3 alone is a huge improvement launched in the last six months. You can now tell o3 something like: “research the X library and architect a custom extension to that library that interfaces with my weird garage door opener; after writing the architecture implement the extension in (node/python/go) and come back in 20 minutes with something that almost certainly compiles and likely largely interfaces properly, leaving touch-up work to be done.
I use LLMs every day, so I get my news from myself (I couldn't even name a Brooklyn journalist?). My experience so far is they are good for greenfield development (i.e. getting a project started), and operating within a well defined scope (e.g. please define this function in this specific place with these constraints).
What I haven't seen is any LLM model consistently being able to fully implement new features or make refactors in a large existing code base (100k+ LOC, which are the code bases that most businesses have). These code bases typically require making changes across multiple layers (front end, API, service/business logic layer, data access layer, and the associated tests, even infrastructure changes). LLMs seem to ignore the conventions of the existing code and try to do their own thing, resulting in a mess.
Def a pain point. Anecdotally Claude code and aider both can be of some help. My go to method is: dump everything in the code base into Gemini and ask for an architecture spec, then ask for implementation from aider or Claude code. This 90% works 80% of the time. Well maybe 90% of the time. Notably it can deal with cross codebase interfaces and data structures, in general, with good prompting.
Dumping it at Claude 3.7 with no instructions will 100% get random rewriting - very annoying.
My impression is that the base models have not improved dramatically in the last 6 months and incremental improvements in those models is becoming extremely expensive.