Is the probability of lost data zero across eg. millions of documents?
I see there's a 30% redundancy per document, but I'm not sure every frame in a h265 file is guaranteed to have more than 70% of a qr code being readable. And if it's not readable, then that could mean losing an entire chunk of data.
I'd definitely calculate the probability of losing data if storing text with a lossy compression.
At some point I eyeballed a comparison to WhatsApp/Instagram/Snapchat growth, and IIRC although it was within the same order of magnitude, it still didn’t reach the rate of growth of those hypergrowth social apps.
A friend of a friend told me about an organization that has a steady income from existing products maintained by just enough engineers to keep the lights on, while the other 80% of the organization is building the “new version” that no customer asked for and that nobody is currently paying for. There’s one product that is used by more than 80% of customers that’s maintained by 2 developers and that the CEO isn’t aware even exists.
Ya I've been there. I even tried pitching to management that a small team of us wanted to move to the legacy product and iteratively improve it because it had customers and revenue and we could make an impact while the new product was under development. They said no. I left about 6 months after. 9 years later the legacy product is still running. I can't find any evidence that they launched a new one.
Sure, but for every efficiently run company, there’s another with 80% of its engineers working on a “new vision” with zero customers, while the revenue-generating software sits idle or attended by one or two developers…
And maybe this is intentional, rational strategy - why not reinvest profits in R&D? But just because an organization is large does not mean that it’s efficient.
The Co-Op (grocery store chain) was hacked around the same time in likely the same incident. It took three weeks for them to get food back on the shelves at my local store. I don’t understand how that’s even possible… what happened to all the meat and vegetables in the supply chain? They just stopped flowing? They rotted? Why couldn’t they use pen and paper? It’s unbelievable to me that a business would go three weeks without stocking inventory.
You could (and people did) run this in the pre-internet days with basically just phone calls and a desk to receive them. The problem is that by now this represents an incredible increase in manpower required overnight.
And you need a process to follow. You can't just have nearly 4000 supermarkets ringing up HQ at random and reading out lists of 1000 items each. Then what? Back when a supermarket chain did operate like that, the processes like "fill in form ABC in triplicate, forward two to department DEF for batching and then the forward one to department GHI for supplier orders and they produce forms XYZ to send to department JKL for turning into orders for dispatch from warehouses". And so on and so on. You can't just magic up that entire infrastructure and knowledge even if you could get the warm bodies to implement it. Everyone who remembers how to operate a system like that is retired or has forgotten the details, all the forms were destroyed years ago and even the buildings with the phones and vacuum tubes and mail rooms don't exist.
Of course you could stand up a whole new system like that eventually, but you could also use the time to fix the computers and get back to business probably sooner.
But I imagine during those 3 weeks, there were a lot of phone calls, ad-hoc processes being invented and general chaos to get some minimal level of service limping along.
I agree, although it seems like a failure of imagination that this is so difficult. The staff will have a good understanding of what usually happens and what needs to happen. What they are lacking is some really basic things that are the natural monopoly of "the system".
Perhaps we need fallback systems that can rebuild some of that utility from scratch...
* A communication channel of last resort that can be bootstrapped. Like an emergency RCS messaging number that everyone is given or even a print/mailing service.
* A way to authenticate people getting in touch using photo ID, archived employee data or some kind of web of trust.
* A way to send messages to everyone using a he RCS system.
* A way to commission printing, delivery and collection of printed forms.
* A bot that can guide people to enter data into a particular schema.
* An append only data store that records messages. A filtering and export layer on top of that.
* A way to give people access to an office suite outside of the normal MS/Google subscription.
* A reliable third party wifi/cell service that is detached from your infrastructure.
* A pool of admin people who can run OCR, do data entry.
Basically you onboard people onto an emergency system. And have some basic resources that let people communicate and start spreadsheets.
> Everyone who remembers how to operate a system like that is retired or has forgotten the details
Anyone who’s experienced the sudden emergence of middle management might feel otherwise :) please don’t teach those people the meaning of “triplicate,” they might try to apply it to next quarter’s Jira workflows…
I remember when I was a teenager working the register at a local store. The power went out one day, and we processed credit cards with a device that imprinted the embossed card number onto a paper for later reconciliation.
That wouldn’t work today for a number of reasons but it was cool to see that kind of backup plan in place.
In the UK the credit / debit cards I've had issued in the last few years have been flat, with details just printed, so that level of manual processing is presumably defunct here.
Don't forget chip & PIN is state of the art novel tech in the US. (From memory I think it was required here in the UK from Valentine's day^ in something like 2005.)
(^I remember the day better than the year because the ad campaign was something like 'I <3 PIN'.)
that is mostly because major US retailers sued Visa/Mastercard to make it not enforceable via lower interchange fees, since then they would have to change tens of thousands of point-of-sale systems at each one
In my case all the perishable shelves were empty - no fruit, no vegetables, no meat, no dairy. I checked every few days for multiple weeks and it wasn’t until three weeks after the incident I was able to buy chicken again.
It’s possible they were ordering some default level of stock and I just didn’t go at the right time to see it, but it sure looked like they were missing the inventory… when I first asked the lady “is the food missing because of the bank holiday?” and she said “no because of the cyber attack” I thought she was joking! It reminded me of the March 2020 shelves.
Interestingly Co-Op is so-called because it’s a cooperative business, which vaguely means it’s owned by its employees, and technically means it’s a “Registered Society” [0].
If you check CompaniesHouse [1], which normally has all financial documents for UK corporations, it points you to a separate “Public Register” for the Co-Op [2].
So, your comment has more basis in reality than simply being snark… the fact that “nobody is incentivized to care” is actually by design. That has some positive benefits but in this case we’re seeing how it breaks down for the same reasons nobody in a crowd calls an ambulance for someone hurt… it’s the bystander effect applied to corporate governance with diluted accountability.
I’m not following your logic. The co-op is designed for everyone to care _more_ because they are part-owners and because the organisation is set up for a larger good than simple profit-making.
In practice the distinction has long been lost both for employees and members (customers), but the intent of the organisational structure was not for nobody to care; quite the opposite
But there are millions of part-owners. Every “member” of co-op (i.e. a customer in the same membership program that just lost all their data to this hack) is an owner of it. Maybe the employees get more “shares” but it’s not at all significant.
And at the executive governance level, there are a few dozen directors.
There is a CEO who makes £750k a year, so it has elements of traditional governance. I’m not saying the structure is entirely to blame for the slow reaction to the hack, or that there is zero accountability, but it’s certainly interesting to see the lack of urgency to restore business continuity.
My family used to own a local market, and as my dad said when I told him this story, “my father would have been on the farm killing the chickens himself if that’s what he had to do to ensure he had inventory to sell his customers.”
You simply won’t get that level of accountability in an organization with thousands of stakeholders. And a traditional for-profit corporation will have the same problems, but it will also have a stock price that starts tanking after half a quarter of empty shelves. The co-op is missing that sort of accountability mechanism.
Exactly, the bystander effect. But it’s not strictly due to the large size. Other big companies get hacked too. But if they have a stock price then there’s an obvious metric to indicate when the CEO needs to be fired. It’s the dilution of responsibility combined with a lack of measurable accountability that causes the dysfunction.
The problem is that cutting IT and similar functions to the bone is really good for CEOs. It juices the profits in the short/mid term, the stock price goes up because investors just see line go up, money goes in, and the CEO gets plaudits. There's only one figure of merit: stock price. What you measure is what you get.
It's only much later that the wheels fall off and it all goes to hell. The hack isn't a result of the CEOs actions this quarter, it's years and years of cumulative stock price optimisation for which the CEO was rewarded.
And you can't even blame all the investors because many will be diluted and mixed though funds and pensions. Is Muriel to blame because her private pension, which everyone told her is good and responsible financial planning, invested in Co-Operative Group on the back of strong growth and "business optimisation intiatives"? Is she supposed to call up Legal and General and say "look I know 2% of my pension is invested in Co-Op Group Ltd and it's doing well, and yes I'm with you guys because you have good returns, but I'm concerned their supermarket division is outsourcing their IT too much, could you please reduce my returns for the next few years and invest in companies that make less money by doing the IT more correctly?"
There is a serious crisis of competence and caring all throughout society and it is indeed frightening. It’s this nagging worry that never goes away, while little cracks keep appearing in the mechanisms we usually take for granted…
Buying and distributing vegetables for stores is not remotely a simple thing. It includes statistical analysis with estimates of demand for every store, seasonal scheduling, weather awareness, complicated national and/or international logistics, plus accounting and payments.
Some or all of those may be broken during a cyberattack.
That’s a good point but perhaps you underestimate the ingenuity borne from constraints.
If you’ve got trucks arriving with meat that’s going to expire in a week, and all your stores have empty shelves, surely there is a system to get that meat into customer mouths before it expires. It could be as simple as asking each store, when they call (which they surely will), how much meat they ordered last week, and sending them the same this week. You could build out more complicated distribution mechanisms, but it should be enough to keep your goods from perishing until you manage to repair your digital crutch.
The suppliers will know and be able to predict what a large customer like M&S is likely to order. They will probably be preparing items before they are even ordered. And surely their must be some kind of understanding of what a typical store will receive.
How about the would-be victims don’t ship exploitable software to production? If that’s not possible, then maybe they should signup for an automated targeted hacking service to find the exploitable bugs before someone else does.
Your argument is straight out of the 1990s. We’ve moved beyond this as an industry, as you can see from the proliferation of bug bounty programs, responsible disclosure policies, CVE transparency, etc…
I’m the kind of person who is conscientious enough to make a handover document when I go on PTO, and the same traits that make me do that are the reason why I’d never use this. It lacks a personal touch; in fact it’s almost insulting (“I couldn’t be bothered to prepare a handover document so I asked this AI to do a shoddy job of it”). And the whole point of a handover document is to capture top-of-mind priorities that exist nowhere outside of your own head. An LLM won’t fix that (at least in the near-term before we’ve all got brain implants…)
Same on longer PTO doc, because there is always something weird going on that people should be aware of and no one should be able to say 'No one told me.' From that perspective, it is a useful CYA, but in my experience it genuinely helps everyone plan a little better.
That is the reason I think the Handover project has merit. What I personally hate however is the format ( video ), because I can see corporations adopting this out of sheer laziness. In other words, I don't wish this project be adopted by corporate America based on personal preferences, but I can kinda see them jumping on it.
reply