Human "thought" is the way it is because "electrical impulses" (wildly inaccurate description of how the brain works, but I'll let it pass for the sake of the argument) implement it. They are its mechanism. LLMs are not implemented like a human brain, so if they do have anything similar to "thought", it's a qualitatively different thing, since the mechanism is different.
Mature sunflowers reliably point due east, needles on a compass point north. They implement different things using different mechanisms, yet are really the same.
You can get the same output from different mechanisms, like in your example. Another would be that it's equally possible to quickly do addition on a modern pocket calculator and an arithmometer, despite them fundamentally being different. However.
1. You can infer the output from the mechanism. (Because it is implemented by it).
2. You can't infer the mechanism from the output. (Because different mechanisms can easily produce the same output).
My point here is 1, in response to the parent commenter's "the mechanism of operation dictates the output, which isn't necessarily true". The mechanism of operation (whether of LLMs or sunflowers) absolutely dictates their output, and we can make valid inferences about that output based on how we understand that mechanism operates.
So the answer is "it's not a guide to computer science". It's like a guide talking about how to get better at mental calculation being titled "guide to learning mathematics", or a guide to language learning being titled "guide to learning linguistics".
> This month I’ve been doing Inkhaven — a writing workshop in Berkley, California where you must publish a 500+ word blog post every day or they kick you out
I'm not familiar with that particular workshop, but it's a writing workshop, and it's 500 words. Now, if they'd said 1,000, you'd have a point, but it's only 500. This comment has 39 and it took me only minutes.
I find the idea of having to pump out posts every day more questionable, rather than the length. In my experience, this is the model of content farms, not people who write things when they know they have something excellent to share.
They are doing "the same thing" only from the point of view of function, which only makes sense from the point of view of the thing utilizing this function (e.g. a clerical worker that needs to add numbers quickly).
Otherwise, if "the parts are all different, and the construction isn't even remotely similar", how can the thing they're doing be "the same"? More importantly, how is it possible to make useful inferences about one based on the other if that's the case?
The more you try to look into the LLM internals, the more similarities you find. Humanlike concepts, language-invariant circuits, abstract thinking, world models.
Mechanistic interpretability is struggling, of course. But what it found in the last 5 years is still enough to dispel a lot of the "LLMs are merely X" and "LLMs can't Y" myths - if you are up to date on the relevant research.
It's not just the outputs. The process is somewhat similar too. LLMs and humans both implement abstract thinking of some kind - much like calculators and arithmometers both implement addition.
Without a direct comparison to human internals (grounded in neurobiology, rather than intuition), it's hard to say how similar these similarities are, and if they're not simply a result of the transparency illusion (as Sydney Lamb defines it).
However, if you can point us to some specific reading on mechanistic interpretability that you think is relevant here, I would definitely appreciate it.
That's what I'm saying: there is no "direct comparison grounded in neurobiology" for most things, and for many things, there simply can't be one. For the same reason you can't compare gears and springs to silicon circuits 1:1. The low level components diverge too much.
Despite all that, the calculator and the arithmometer do the same things. If you can't go up an abstraction level and look past low level implementation details, then you'll remain blind to that fact forever.
What papers depends on what you're interested in. There's a lot of research - ranging from weird LLM capabilities and to exact operation of reverse engineered circuits.
There is no level of abstraction to go up sans context. Again, let me repeat myself as well: the calculator and the arithmometer do the same things -- from the point of view of the cleric that needs to add and subtract quickly. Otherwise they are simply two completely different objects. And we will have a hard time making correct inferences about how one works based only on how we know the other works, or, e.g. how calculating machines work.
What I'm interested in is evidence that supports that "The more you try to look into the LLM internals, the more similarities you find". Some pointers to specific books and papers will be very helpful.
> Otherwise they are simply two completely different objects.
That's where you're wrong. Both objects reflect the same mathematical operations in their structure.
Even if those were inscrutable alien artifacts to you, even if you knew nothing about who constructed them, how or why? If you studied them, you would be able to see the similarities laid bare.
Their inputs align, their outputs align. And if you dug deep enough? You would find that there are components in them that correspond to the same mathematical operations - even if the two are nothing alike in how exactly they implement them.
LLMs and human brains are "inscrutable alien artifacts" to us. Both are created by inhuman optimization pressures. Both you need to study to find out how they function. It's obvious, though, that their inputs align, and their outputs align. And the more you dig into internals?
I recommend taking a look at Anthropic's papers on SAE - sparse autoencoders. Which is a method that essentially takes the population coding hypothesis and runs with it. It attempts to crack the neural coding used by the LLM internally to pry interpretable features out of it. There are no "grandmother neurons" there - so you need elaborate methods to examine what kind of representations an LLM can learn to recognize and use in its functioning.
Anthropic's work is notable because they have not only managed to extract features that map to some amazingly high level concepts, but also prove causality - interfering with the neuron populations mapped out by SAE changes LLM's behaviors in predictable ways.
You are making the false assumption that if output can be inferred from structure, the converse is true as well. Similarity in behaviour does not in any way, shape or form imply structural similarity. The boy scout, the migrating swallow, the foraging bee, and the mobile robot are good at orienteering. Do they achieve this goal in a similar manner? Not really.
Re: "I'm baffled that someone in CS, a field ruled by applied abstraction, has to be explained over and over again that abstraction is a thing that exists". Computer science deals with models of computation. You are making a classic mistake in confusing models for the real things they are capable of modelling.
> That's where you're wrong. Both objects reflect the same mathematical operations in their structure.
This is missing the point by a country mile, I think.
All navel-gazing aside, understanding every bit of how an arithmometer works - hell, even being able to build one yourself - tells you absolutely nothing about how the Z80 chip in a TI-83 calculator actually works. Even if you take it down to individual components, there is zero real similarity between how a Leibniz wheel works and how a (full) adder circuit works. They are in fact fundamentally different machines that operate via fundamentally different principles.
The idea that similar functions must mean that they share significant similarities under the hood is senseless; you might as well argue that there are similarities to be found between a nuclear chain reaction and the flow of a river because they are both harnessed to spin turbines to generate electricity. It is a profoundly and quite frankly disturbingly incurious way for anyone who considers themself an "engineer" to approach the world.
In case you have missed it in the middle of the navel-gazing about abstraction, this all started with the comment "Please stop comparing these things to biological systems. They have very little in common."[0]
If you insist on continuing to miss the point even when told explicitly that the comment is referring to what's inside the box, not its interface, then be my guest. There isn't much of a sensible discussion about engineering to be had with someone who thinks that e.g. the sentence "Please stop comparing [nuclear reactors] to [coal power plants]. They have very little in common" can be countered with "but abstraction! they both produce electricity!".
For the record, I am not the one you have been replying to.
"How would this impact people who rely on screen readers" was exactly my first thought. Unfortunately, it seems there is no middle-ground. Screen-reader-friendly means computer-friendly.
The US is the largest market for firearms, so the NRA can use the threat of boycotting a manufacturer within the states to prevent the technology gaining traction elsewhere.
To profit, they would first have to sell the goods. Who is actually in the market for a smart gun? Consumers aren't, surely. There is virtually no upside to your gun tracking you, at your own expense of buying a more complex piece of tech to boot. So that leaves something like (apparently) New Jersey where the government would compel purchases of smart guns because they were interested in the tracking. But eg. China simply don't allow citizens to purchase guns period. There may be some application to applying it to state-owned firearms to track military and police usage, but deploying that at Chinese scale would be an extremely expensive endeavour for what appears to be a solution in search of a problem. Not to mention the biometric lock concept, if implemented, is introducing an entire new axis of unreliability to a life-or-death tool.
Gun owners in the US probably wouldn't want their gun to be used against them in a home invasion, or by their child at a school. Seems like that could be a large-ish market. Especially if you can lobby regulators in favor of making it a requirement for all or some people.
You are right that gun owners wouldn't want those things, but they are unlikely to want a smart gun as a solution to those things.
They want the gun to be available to them, and not be under duress to use a fingerprint reader or pin pad or RFID ring to do it.
Responsible gun owners keep guns out of children's hands by locking them up or supervising them, and irresponsible ones aren't going to want to pay extra for smart features.
I think there's a very narrow range of smart features, something like a gun that is unlocked when removed from a holster, but locks up if it is dropped or grabbed, that might be interesting. That makes having the gun taken from an officer less of a threat, which might have an institutional appeal. Give it a 10-hour maintenance mode so that it can be used as a "nightstand gun" while automatically being locked if left idle for longer, and it would basically meet the needs of police both on and off-duty.
In my personal experience gun owners want mechanical foolproofness too. They want something that's not going to lock up or fail or discharge at the wrong time. Smart features just add a layer of complexity with fail possibilities to address a problem that many of them would prefer to be addressed differently anyway.
I think a country like Australia could be a good starting point for smart guns. Yes, not a very big market-around 8% of US population, with significantly lower rates of gun ownership-but culturally more open to gun control, with a much weaker gun rights lobby, and a marked political tendency towards surveillance and “nanny state” regulation
IIRC Australia doesn't have legal frameworks for gun ownership for the purpose of self defense, and there's no great implementation of smart guns in the first place.
A smart gun is like an AWS authenticated motor twisting ballpoint pen. Just no one ever seriously pays for such a thing, and it has not even been seriously made if it ever was actually conceived. Making it a requirement is basically out of question.
> Making it a requirement is basically out of question
Why? If there’s the political will, it is doable. There are Australian gun manufacturers (e.g. Lithgow Arms, owned by Thales)-and if none of them are willing to cooperate, the government can always start their own gun manufacturer. Indeed, Lithgow Arms was founded in 1912 as a government-owned arms manufacturer, and remained in public hands until the Australian federal government sold it to Thales in 2006.
Same reasons as why things like Clipper Chips isn't happening. It completely lacks technical basis, and even political consensus gets sketchy quick.
Post-war Commonwealth nations are generally bad at gun designs as well - UK tried once and produced an infantry rifle that will(not could) seriously injure its user if held and fired in left hand. So even if forced, the approved gun will be more of a theoretical product, and the smart gun mandate will just be a less politically viable alternative to total firearms ban.
The tech just isn't there; hand-held guns don't benefit from a computerized firing system at all. So any smart feature on human sized guns and less will be totally removable addons, and that completely defeats its purpose.
Many tanks and planes do have smart guns. Electronic firing control with additional software features that impede firing are beneficial and totally fine at that scale.
That's just a scope. Comes right off and the gun reverts to a regular M4.
Most(though not all) other smart gun attempts work in a similar fashion; the host gun works exactly as it were, except an extraneous metal bit inhibits firing. If the bit was removed or held down, it reverts to the original host gun and fires normally. As such, the extra bit is literally extraneous, irrelevant to the gun's working.
The correct answer is - all the designs so far aren’t great.
The military would love a smart gun to cut down on accidental discharges. Cops would love it to stop weapons being used against cops.
The issue is that it has to have a very high reliability (you don’t want it to fail to fire while a suspect is shooting at you). And not much point if it only works “sometimes” with unauthorized users.
This. There’s many countries that allow civilians firearms (e.g. Canada and much of Europe), but generally for hunting purposes and thus more likely to be rifles and shotguns than concealable handguns.
Most of the world doesn’t need that whole setup because:
- Our cultural baseline around firearms is completely different. Countries like Denmark, Sweden, Finland, Switzerland, Austria, and the Czech Republic have plenty of guns at home - and historically, a lot of them were actual assault rifles, not “looks-spicy” semiautos.
- We treat guns like weapons. They live in safes, not nightstands, and kids get taught safety early, the same way you’d teach them not to put a fork in a power supply.
The Swiss do have a lot of guns at home. However, you cannot carry (or even transport guns that are not discharged). Just take them at a shooting range - a popular pastime for Swiss people.
2.5 million years ago, a friend of mine
Made a tool from a stone and defended his tribe
It's technology, sorry for the technical term
It's a wheel then a fire and the rest is a blur
Where does the output come from if not the mechanism?
reply