I know very little about the protocol aspects of cellular communication, so can anyone explain how such a huge gaping security hole could come into existence?
Before 2G, networks used completely unencrypted analog voice. You could snoop on anyone's calls with a slightly-modified radio; at least until Congress heard about this and made it illegal to sell a radio that could be modified to do this[0].
2G was actually considered a huge bump up in security because you could encrypt the contents of calls. Albeit with hilariously insecure crypto mandated by the old ITAR regime[1]. IMSI catchers weren't part of their threat model, for the same reason why people only recently have realized that metadata is relevant to security.
[0] This law is still on the books, even though analog cellular is entirely dead. It's still a pain in the ass to properly comply with this for, e.g. software-defined radio.
[1] This is the same reason why DVD CSS was so easy to crack, and why we there used to be 10 different ways to strip SSL before we decided to stop serving old browsers entirely.
In the beginning of cell phones, security was too expensive. Telcos also like to do their own things, so GSM encryption wasn't built on best practices. And some countries forbid use of even GSM encryption.
Early mobile phone networks suffered from cloning, so work was done to improve verification of clients, but verifying the network wasn't seen as required. Telcos have been historically light on authentication and verification; so it's not surprising.
Adding to this the GSM A3/A8 algo were broken shortly after they arrived in the US. The only mitigating control was my boss in a wireless provider and the FBI meeting up with someone that was going to demo breaking it. They were advised what prison they would be relocating to and the demo was called off. Rinse and repeat. This was before the internet was popular or even widely used. The word eventually got out.
The networks are insecure by standard. They are designed such that they can have "lawful intercept" by government entities. The key material on the SIM card is readily transferred between the carrier and SIM/eSIM card manufacturers, which enables multiple levels of supply chain attacks if the material is mishandled.
IMSI-catchers are not considered a security hole by the carriers or the standards bodies. SUCI/SUPI was put in at the request of phone vendors, if I remember correctly, and is still the only piece of public key cryptography in the networks. Everything else is symmetric keys.
"Depending on national requirements, the CSP may be required to report the location of the Target at the beginning and
end of CS calls and PS and IMS sessions on a per warrant or per intercept basis. It may also be a national requirement
for the CSP to report the location of the Target [...]"
The telco trusts its own network. Telcos doesn't trust users, so users need to authenticate themselves, and devices need to be regulated. But under the traditional telco security model, the network doesn't need to authenticate itself to the devices.
Even today, building the necessary infrastructure for network trust management is also really, really hard across the many jurisdictions involved.
The phreaking [1] community was huge and becoming increasingly sophisticated long before mobile was even a thing. I think it's mostly that telecoms were traditionally discouraged from pursuing security. There's, at most, a minimal commercial incentive to it, and the government loves comms that can be easily spied on meaning you're going to get pushback from that side if you start aiming for security.
The idea to start using SMS for secure purposes was similarly probably never really about security, but an advertising/government driven effort given that it helps create a fairly reliable tracking identity for a person. It makes no sense otherwise to use SMS over something like a 2FA app which is completely cross platform, secure, free, and has basically 0 downsides relative to SMS, and a whole bunch of upsides. The only thing is that it's also anonymous.
Don’t 2FA apps have the major downside that if you lose the specific mobile device you installed it on you’re SOL, unless you have backup codes that are too technical for most. SMS gets you more human support since you pay your carrier, I can walk into my nearest teleco branch with my ID if I lose my phone and change the SIM to another phone. So most of the time unless your SIM is hijacked it’s a good proxy for being actually you.
Plus having to download another app adds friction to the signup process and most users aren’t going to bother, so for most it’s SMS 2FA or nothing. Since apps often want your phone number anyway for bot prevention, and users are used to verification codes, it’s not a big deal.
Also a tail end of other issues with 2FA apps (and SMS 2FA predates the nice ones anyway); in other countries there are devices other than iOS/Android to suggest an authenticator app for, limited network speeds and device storage, etc. Heck, I know people in the U.S. with full device storage who can’t download new apps without deleting some stuff. If you’re a random app and not a tech company SMS 2FA is just going to be much easier to implement.
The whole point of 2FA is that once you lose possession of your physical second factor, you lose access. If you can maintain access after losing the hardware, you've just added a second password. SIM swapping attacks have proven very effective at showing how easy it is for someone to bypass SMS 2FA. It's better than no 2FA, but it's the worst second factor out there.
If you don't want to lose access after losing your second factor, you don't want two factor authentication. Trying to make 2FA something it's not only muddies the waters and makes things annoyingly confusing.
I don't think "I know someone whose phone can't handle a 2MiB TOTP app" is a good reason not to offer real 2FA on a website. Sure, offer SMS codes for people who don't care much about security beyond ticking auditor boxes.
No curious reason for it coming into existence. It's software, it will have bugs and oversights. What's curious is that it and so many other problems of the cellular grid have been left untended to for almost three decades.
