I’m genuinely curious on the point about reducing headcount because AI will be more efficient. I’ve seen it articulated here but other places too that a company will be able to have less engineers because each would be more productive. What if companies kept the same number of people engineers but now massively out produce what they used to? And I disagree with the example that this is like typewriters replacing typists. I think typists have a fixed number of things that need to be typed. Software is different - a company that has a better or more feature rich project could gain on their competitors.
Curious if anyone else thinks this. Maybe it’s just optimism but I’ve yet to be convinced that a company would want to maintain its productivity through trading engineers for AI if it had the same opportunity to grow its productivity through AI and maintaining headcount.
And to add-on, isn’t there some market dynamics we are avoiding here with this example? If I’m an AI company and really produced a principal level engineer, why would I sell it for less than the labor market is willing to bear? Wouldn’t I price it perhaps less than the market but not so dramatically less as to lose money.
You make a good point. The shrinking headcount is not necessarily tied to mass-firing. It's more likely tied to +10M newly trained engineers entering the job market every year, but only 50 positions being opened.
Over time, with each recessions, headcount will shrink at some companies, and will not grow back to pre prior levels. Over time, the line trends downwards
Additional Context: The Getty Villa which is on the border of Malibu and Pacific Palisades was the structure that was threatened by fire directly. This article is about the Getty Center which lies in Brentwood and fires did not reach it.
Center: 1200 Getty Center Drive, Los Angeles, CA 90049
Villa: 17985 Pacific Coast Highway, Pacific Palisades, CA 90272
The reason this article is likely appearing now is because the Getty Center proper is currently in a zone which is under evacuation orders:
"The Getty Center, situated in Brentwood, draws 1.8 million visitors annually and houses hundreds of centuries-old art pieces from renowned artists such as Van Gogh, Rembrandt and Monet.
But even though as of Saturday, the center was included in a mandatory evacuation zone as a result of the Palisades Fire expansion into Brentwood, the center insisted its campus is the "safest place possible" for its massive art collection."
The Getty has an endowment in the billions and an entire team devoted to this that is permitted to be on site, coordinates with local fire and police services, etc. The entire center is also built into the hillside, with fireproof vaults in the underground levels, so there is no real risk to any of the collections.
An observation on how this has become extremely common nowadays.
Here, at work, in real life. People just argue with whatever dumb thing they can come up with, for the sake of arguing, it makes them feel smart. It's really hard to have a meaningful conversation with them.
I go to a couple philosophical discussion groups and the occasional town hall meeting. People just can't get their imaginary needs satisfied.
"But that area seems unsafe"
"We could hire a security guard to be around"
"But what if the security guard is a criminal, like in that one episode of muh favorite tv show"
"We could do an extensive background check, work with companies that have a good reputation, ..."
"But what if they make all that up, I saw that in a movie"
And nothing. ever. gets. done.
Btw, I've even seen people get a small round of applause by their peers after making one of such arguments irl. This comes to mind: https://www.youtube.com/watch?v=kn200lvmTZc.
Yes: you can notice this, for example, in the announcements they put out while the area around the Getty Villa evacuated and then burned, where they pointed out that non-emergency staff had evacuated. Similarly, I think JPL always had (emergency) staff present, and their own firefighting resources.
Having heard about wildfire policies for some high-profile institutions in fire-prone areas, they'll often have their own procedures, in coordination with local authorities, which may not involve evacuating when others do, and may involve people coming to the site when others are evacuating.
I imagine they have a number of life-support systems to ensure the staff can hang out in the building for a while in case of a severe natural disaster.
I don't think I've personally ever been as disillusioned with technology more than with the recent rapid adoption of AI.
I think AI is massively negative on the environment via increased energy consumption to train large LLMs, on creative endeavors via the auto-generated art, etc. on labor via automation of jobs through AI agents, or the degradation of the job market, on public communication via the proliferation of fake videos/news, on personal relationships via AI significant others, on child development through easier cheating, etc, etc.
I get that there's a certain inevitability about technology. I also feel - and this could be a terrible analogy - that might have been what it was like to develop nuclear weapons. Better to develop and harness the technology for yourself if your competition is doing so too. That's the world and the market that we've constructed and it's rational to take advantage where you can.
I certainly don't pretend to stand on any moral high ground about what I would do if given the opportunity to cash out from AI skills if I had them.
But it all makes me very sad. I think I'll go read a book.
As an SRE who’s moved between a few companies recently, I’ve noticed a trend towards simpler infrastructure options rather than k8s. Most places I interviewed at used AWS with just fleets of EC2s managed through ASG and instance refresh.
Deployments are no downtime. It’s pretty nice and if you want to run with docker on the host instances you can but you can also just install the application using systemd.
How do you build ec2's? Funny thing is, everyone will have their own answer. Few will look alike. It's maybe simpler, but also bespoke.
There's different definitions of simpler. I've rarely felt like folk's have a great grasp when they cobble together a bunch of options. Things that maybe one engineer started as a simple project 5 years ago grow and sprawl. They rarely stay simple.
What this team gets out of the box here is not to be trivialized. You havent started talking to the ci/cd that ArgoCD deploys. How are load balancers setup & managed, and at what price (Traeffik here is basically free)? Well managed Postgres with backups & replication, Lets Encrypt... handled. Is it really simpler to keep DIY'ing bits and bobs? Are you confident your simple start will work reliably & won't need improvement? What is better about your pile "simple" bespoke solutions, and why not have a platform that integrates varying concerns with consistent management patterns & autonomic operators?
EC2s are provisioned with Terraform or if you want to bypass that, click-ops. Nothing custom is needed. You can define a small shell script that is known as userdata that runs when the EC2 boots. That's standard and used everywhere that EC2s are used.
If you want more customization - again not necessary - you can build via Packer. The thing is, you have to do that anyways with Docker or whatever your container runtime happens to be. Docker images need patches as well if you install dependencies like openssl, libpql, etc.
ArgoCD - ArgoCD is great but it's something that has to be separate managed and comes with a database, UI layer, etc. and it's own set of permissions. The whole point of ArgoCD is to have deployments specifically around GitOps for container-based applications. Unnecessary if you don't have K8s.
Load balancers are simpler with ALBs and route directly to the auto-scaling groups. Traeffik is something that you need to manage. Also, K8s typically requires the use of external load balancers to work in the cloud as it is which means there's a layer of complexity that was introduced. Typically, K8s through the External Load Balancer Controller will provision the LB and attach targets. So it adds complexity. Traeffik isn't free, you are paying for the compute in your cluster which has the overhead of the control plane.
Let's Encrypt is another layer of complexity that has to be managed. AWS has ACM which can dynamically create and renew your certifications without you having to do anything.
K8s as a cluster needs to be managed and introduces significant breaking API changes across minor versions (1.23-1.24, etc.). Postgres managed as a stateful application also breaks some of the benefits of K8s in that it needs to be managed as a StatefulSet rather than as a ReplicaSet which means you likely need different deployment mechanisms to avoid destroying it accidentally.
Ironically, this architecture doesn't need any customizations at all. You can provision it with a few modules in Terraform or you can click ops the entire thing. K8s on the other hand, requires pretty constant attention. You've now introduced the management of the cluster, the building of the image, management of ArgoCD, Traeffik as well as external load balancers.
Basically:
- Route53 for domain management
- ACM for cert management
- EC2s for LBs
- ASGs to manage EC2s
- Small shell script within userdata to configure the host vm
- RDS if you need a database
Avoiding the operational and actual costs of K8s.
I think that there are compelling reasons to use K8s:
- Lots of smaller services that need to interface with each other
- Product is deployed to multiple clouds or in a hybrid setting
This is what we do for GCE. Our CI builds a system image and then that's deployed across our datacentres. Worked perfectly for years, but still people want to switch to kubernetes. What can you do?
Oh definitely that happens and from my experience I don’t think there’s a lot that can be done unless there’s strong faith in engineering leadership to push back on the complexity.
> Serve traffic behind a load balancer that has a WAF
whats the cheapest non aws way to do this? cloudflare on everything? is there another option? just trying to learn whats out there. WAF mainly protects against ddos right?
There are also some projects like Coraza in the works: https://coraza.io/
Probably not what you're looking for if you want a cloud service to take care of everything for you, though, because of the question below (just thought that it might be useful to point out that anyone can run their own WAF if need be).
> WAF mainly protects against ddos right?
Typically WAF might be offered as a part of a larger cloud service that would include DDoS protection.
However, on its own, it is meant to filter traffic that might be harmful and attempt to exploit various vulnerabilities. A bit like an anti-virus in a sense, but for web requests. Some people argue that WAF solutions can be problematic because they encourage an attitude of "so what if there's a log4j vulnerability in the codebase, the WAF will take care of it" instead of making sure that the actual code is secure, but opinions are split there (defense in depth and the Swiss cheese model).
GP has some good suggestions. For implementation of these, Cloudflare is a decent first stop - though they are a little hostile to non-vanilla internet users. Their free plan offers sensible security (SSL termination, WAF, DDOS protection) out of the box, with a straight forward UI.
Network segregation for database (separate subnets) would be a config option wherever you're hosting (AWS/Google Cloud/etc.) said database/application.
It’s a feature of an LB that consolidates the actions of blocking ports except for the ones you are using, fail-fast on paths that scrapers tend to check (e.g. /wp-admin, /phpMyAdmin) so it doesn’t end up in normal request logging, set rate limits, fail-to-ban conditions, etc.
Seems like we’re recreating the Middle Ages financially a bit here doesn’t it?
Mobility is linked to previous generations more than your own. Wealth is more important than income because there are so few jobs that can actually build enough wealth to own a home.
I think Americans workers are willing to tolerate conditions of exorbitant wealth disparity for at least two reasons.
First, because Americans believe someday that they will be wealthy and they do not want to eliminate their chance to join the club[a]. In my opinion, this is the most satisfying explanation. Workers do not want to remove that ladder to the upper class because in doing so, they place a hard cap on their expected outcomes.
Second, American political culture is derivative of a property-rights focused political philosophy. One of the biggest influences on the Bill of Rights and it's explicit protection of property rights was John Locke's Two Treatises of Government. John Locke argues that a defining purpose of government is in the protection of property (life, liberty and property). Americans therefore, view wealth as at least in some part meritocratic and rich, having earned their extra property, are wary of retracting that property. The importance of protecting what's "theirs" is politically, socially and legally baked-in to society.
If Americans' belief in their own prosperity suffer and/or the relative importance of property diminishes in light of other rights (health care, equality, etc.), I think then we might begin to see changes for high-wealth compensation. But, I would argue as long as each of these remain strong, we aren't likely to see any changes.
I think Americans workers are willing to tolerate conditions of exorbitant wealth disparity for at least two reasons.
I think most likely most American workers feel pretty helpless, like pawns in a game they can't exercise control over. They likely do not think of themselves as tolerating wealth disparity. They most likely think of themselves as enduring something they simply can't fix.
Health care is crazy expensive in the US. If you effectively addressed that, an issue which can leave The Have Nots bankrupt for needing medical care for a serious condition, you would go a long ways towards redressing income inequality.
Last I checked, the US was on a very short list of countries without maternity leave and the only developed/wealthy country without it and also on a short list of developed countries without universal health coverage.
It's not about reducing expenses. It's about trying to establish a social safety net that most countries that see themselves as more civilized than the US already have and it's part of what reduces inequality between The Haves and Have Nots in most places.
Look, I'm only reporting what Pew Research purports to be on Amercans' minds. If you think Pew is wrong or that Americans have the wrong view, that's fine. I'm not arguing that.
The US has maternity/paternity leave via the FMLA, but its not very long (12 weeks) and its not paid. Small businesses (<50 employees) are also exempt.
Unpaid leave isn't what anyone means here. It's not helpful to tell a wage worker to take FMLA for three months and thereby lose a quarter of your yearly income.
Americans only care about moving up that ladder themselves and keeping others below them.
Here’s examples..
1. Universal healthcare was proposed as early as 1912, but overwhelmingly white Americans at the time didn’t want to cover minorities. This was backed by insurers and the work of Frederick Hoffman, the head of American Statistical Association at the time. Nothing has changed on the universal healthcare issue since that time.
3. Americans obviously don’t care about gun violence. Since the sunsetting of the Federal Assault Weapons Bans, mass shootings (overwhelmingly committed by AR15) have skyrocketed with no end in sight.
4. 58% of Republicans believe the US country should prioritize expanding exploration and production of oil, coal and natural gas. As an example, China outspent the US almost to 5:1 in transitioning to low carbon energy production in 2023.
5. Americans don’t care about income inequality. Improving public school education is the best way to improve outcomes for poor people but state legislatures have been defunding public schools since desegregation.
Let's say this is true. Logically then, one could make the point that people who are loudly proclaiming that the American worker is suffering because of income inequality (just look through this thread) are clearly out of touch with the average American worker and maybe even have a saviour complex with a touch self-righteousness.
Americans realize that any tool that can be used to remove wealth from the rich can be used wealth from themselves as well.
France recently had what, like a hundred days of continual protests over poor economic conditions? But yes, give politicians more power to fight the fat cats. That'll work!
---
EDIT: Check my profile, I founded a healthcare billing company. You know who supported the Affordable Care Act the most? Insurance companies. [1]
This! Every time I hear about a new tax for the rich, somehow the middle class end up actually paying it.
The rich have every tool in their arsenal to limit their tax liability. When you can afford the best accountants, lawyers, and politicians that money can buy, you can ensure that you will never have to pay your fair share. Oh, and even if a new law somehow does make their costs go up, they'll just pass it on to the middle class by raising the prices on whatever they sell to us to make money.
"The poor often say, “‘Why don’t the rich pay for it?’ or ‘The rich should pay more in taxes and give it to the poor.’” However, the real rich never pay taxes. The people who pay taxes are the educated, middle class."
Reduce government waste. You could leave entitlement programs completely intact and still cut billions of dollars in spending without anyone but heavy hitting political donors noticing. Then simplify the tax code with straight forward graduated income tax brackets without exceptions or deductions.
I looked around for lists of waste, fraud, and abuse and found this [0], and while I'm loathe to use The Heritage Foundation as a source let's just stipulate we could save something like $300B a year if we deleted all this stuff. That isn't even 5% of the budget, like you're not gonna tackle climate change with that kind of investment, especially if you're scouring all the new green policies to ensure they're minimally wasteful (also imagine the meta-bureaucracy needed for this).
I love the idea of simplifying the tax code, but I think if you're arguing it's useless to tax the rich I think you have to think that would be useless too, right? We'd need to actually have the rich pay their rate, but you're saying that doesn't work.
The problem is that your loophole is my socially-significant deduction. Why wouldn’t we want to allow deductions for insulation and energy efficiency, or electric cars?
Every deduction exists because someone wants it to exist. And “eliminate all deductions” lasts until someone goes “but what about the child credit” or any other issue.
This will continue as long as Americans continue to be averse to the idea of actual cash transfers and continue to use the tax code as a backdoor system for remitting subsidies and credits.
I understood your argument to be that the rich tend to find ways to pass on their taxes, which I might even agree with--maybe, so I was thinking that if taxes don't do it then what would. I agree with my sibling though: the tax code is where we've subsidized people for generations now because of our discomfort with giving benefits to Black Americans. I think it would be hard to convert all that into equivalent benefit programs, and I think that's why people on the right advocate so much for it, because the likely end result is that many of those programs are cut, and the rich still only pay capital gains.
It’s simpler than that - there is no economic system that both rewards and incentivizes productivity that wouldn’t result in stark wealth or income equality.
Consider a simple game of flipping coins would result in similar inequality despite it not being a game of skill at all.
Inequality is inevitable. Waste of time and energy discussing that. Rather we should discuss actual quality of life grievances (and to be fair these may be related to the inequalities).
There's also the weird and uncomfortable truth that substantial wealth accumulated all in one place is what keeps a significant amount of private business, arts, and craftsmen in business.
Who really needs a beautifully-curved hand-carved wood staircase? Nobody, but rich guys want them occasionally, so that's keeping a whole company and a couple of master craftsmen in business.
You could argue that this is just a misallocation of resources and that without said rich guys the same economic productivity would move elsewhere instead. But I'm not convinced that's equivalent, it's "mythical man-month" reasoning. $1 in 100 people's pockets is not the same as $100 in one pocket.
I'm not at all arguing that the money is deserved or that we shouldn't consider wealth redistribution policy. As you said, it comes back down to quality of life. When we talk about wealth inequality, it's important to realize that we're not trying to prevent people from making money and spending it on things other than groceries and gas, we're trying to prevent people from being materially poor, and mess with everything else as little as possible.
It's sort of true. Give money to the masses, and there will be more demand for mass-produced products. When money is concentrated in the hands of a few elite, they'll commission some exceptional things.
But on the other hand, the same masterwork staircases could be put towards public buildings, like grand libraries, museums, and train stations. Then everyone can enjoy those splendid things and the master craftsmen still stay in business, without their works needing to be confined to the mansions of the wealthy.
But these days we tend to disapprove of public works being anything beyond the bare minimum of utilitarian functionality. Anything else is seen as a wasteful extravagance.
I often hear your first point repeated, but have never met a poor person that thought they were going to be a millionaire.
However, I think it is a biased caricature of a real phenomenon.
I think Most Americans don't want the government to put limitations on how successful they can be and put barriers in their way if they start being coming successful.
This is essentially the same idea with different framing
Part of the issue is that regulation tends to hurt small businesses more than big ones. So I think resistance to adding new rules is somewhat justified by the experience that adding new rules makes it primarily harder to run small business. Whether this is a matter of naivety in policymaking or anticompetitive regulatory capture is another issue.
Even the shareholders of some of these firms think the executives are robbing them with the compensation they've set (see ongoing Tesla lawsuits, for example). But, they can't come up with a clear rule for what "appropriate" pay is.
I suspect we'll hit a point where the pay is so excessive that it's worth buying companies purely to cut the pay of management, as it'll be the easiest way to improve profitability.
Curious if anyone else thinks this. Maybe it’s just optimism but I’ve yet to be convinced that a company would want to maintain its productivity through trading engineers for AI if it had the same opportunity to grow its productivity through AI and maintaining headcount.