My experience with LLMs currently is that they can handle any level of abstraction and focus, but you have discern the "layer" to isolate and resolve.
The next improvement may be something like "abstraction isolation" but for now I can vibe code a new feature which will produce something mediocre. Then I ask "is that the cleanest approach?" and it will improve it.
Then I might ask "is this performant?" Or "does this follow the structure used elsewhere?" Or "does this use existing data structures appropriately?" Etc.
Much like the blind men describing an elephant they all might be right, but collectively can still be wrong. Newer, slower models are definitely better at this, but I think rather than throwing infinite context at problems if they were designed with a more top down architectural view and a checklist of competing concerns we might get a lot further in less time.
This seems to be how a lot of people are using them effectively right now - create an architecturr, implement piecemeal.
I’ve found it to never be able to produce the cleanest approach. I can spend 5 hours and get something very simple and I can even give it that as an example and it cannot extrapolate.
It can’t even write algorithms that rely on the fact that something is sorted. It needs intermediate glue that is not necessary, etc. massive noise.
Tried single allocation algorithms, can’t do that.
Tried guiding to exploit invariants, can’t find
single pass workflows.
The public training data is just bad and it can’t really understand what actually good is.
Define clean, what does that even mean? If you tell it to write efficient code for example, how would it know whether you're taking about caching, RAM, Big O, I/O etc unless you tell it?
This has been my experience too. It's largely about breaking things down into smaller problems. LLMs just stop being effective when the scope gets too large.
Architecture documentation are helpful too, as you mentioned. They are basically a set of rules and intentions. It's kind of a compressed version of your codebase.
Of course, this means the programmer still has to do all the real work.
In my experience when you have a problem that is small enough for an LLM to solve, you could just write the code directly. You don't have to produce a detailed spec first
If the LLM needs a detailed spec to solve the same problem then you're doing unnecessary work to produce the spec for the LLM first
This has been the problem with higher level natural language programming for years. I really wonder what people are doing if they don't see this core issue that precludes their use.
It makes me wonder if some people writing code just cannot think in terms of code?
I imagine it is very slow if you always have to think in a human language and then translate each step into programming language
When people describe being in flow state, I think what is happening is they are more or less thinking directly in the programming language they are writing. No translation step, just writing code
LLM workflows completely remove the ability to achieve that imo
CSS or Tailwind has always been a tough one for me. I have banks of flashcards to help me remember stuff, (align-items, justify-content, grid-template-columns, etc.). Even with all that effort and many projects of practice, though, I've never had things click.
LLM assisted programming, however? – instant flow state. Instead of thinking in code I can think in product, and I can go straight from a pencil sketch to describing it as a set of constraints, and then say, "make sure it's ARIA compliant and responsive", and 95% of the work is done.
I feel similarly about configuration heavy files like Nginx or something. I really don't care to spend my time reading documentation, I'd rather copy paste the entire docs into the context window and then describe what I want in English.
Also good for SQL. And library code for a one off tool or API. And Bash scripting.
Doesn't being able to only handle a single layer of abstraction as long as it is well defined and isolated essentially mean it can't handle abstraction at all?
The next improvement may be something like "abstraction isolation" but for now I can vibe code a new feature which will produce something mediocre. Then I ask "is that the cleanest approach?" and it will improve it.
Then I might ask "is this performant?" Or "does this follow the structure used elsewhere?" Or "does this use existing data structures appropriately?" Etc.
Much like the blind men describing an elephant they all might be right, but collectively can still be wrong. Newer, slower models are definitely better at this, but I think rather than throwing infinite context at problems if they were designed with a more top down architectural view and a checklist of competing concerns we might get a lot further in less time.
This seems to be how a lot of people are using them effectively right now - create an architecturr, implement piecemeal.