I manage a few rather large k8s clusters during the day and I don't understand what this is or why I'd need that without reading too much. If I read too much I switch off. This is a side effect of the emotional damage from managing a few rather large k8s clusters during the day.
Multi cluster management that separates the control plane from the worker plane. So it's like a RYO managed K8s service. Now you just need to get a full time SRE team to take care of what Red Hat, GCP, AWS, IBM Cloud SRE teams do for you today in any of the managed container platforms.
The article, and the project page[1] don't do a very good job of explaining WHY the project exists. It definitely explains what it is (an operator for a control plane,) but so what?
What can I do now that I could not do before? What makes it novel or innovative?
This just seems like Gardener? https://gardener.cloud/ No affiliation but they built what I would've built and I've worked on enough Kubernetes deployers/orchestrators that it could be considered a Human Rights violation.
Vcluster is virutalizing a single cluster, it looks like. The use case here is actually making many different clusters, each with their own dedicated hardware, I believe.
I've never used Kubernetes in anger, for the stuff I do, I manage just fine with bare-metal hardware running services with systemd.
The (IMO very funny) comments here about k8s causing emotional damage, human rights violations and need to take drugs have reassured me that I should stick to my bare-metal. It is the way.
That's going to leave a big hole in your CV/resume where there could be a CNCF word salad to make the recruitment and HR drones drool.
They need to know if you can write a custom k8s controller in Clojure to generate cronjobs that run k8s jobs that call AWS lambada to ingest data from elastic cakehole and inject it into substandard-IA random availability zone iceberg storage while a Annoying Notification is triggered via ANS to a Rickety Queue Storage sent to someone via Shaky Email Service that eventually ends up in someone's Junk Mail folder in O365 but contains a link to the insecure bucket with the entire company's payroll data in it.
Btw if anyone wants to hire me to wire stuff up like this and help you piss away money you don't need into Bezos' pockets, please get in contact.
Edit: incidentally this is why I'm breakdown level done with all this shit and am going to do something else with my life.
>They need to know if you can write a custom k8s controller in Clojure to generate cronjobs that run k8s jobs that call AWS lambada to ingest data from elastic cakehole and inject it into substandard-IA random availability zone iceberg storage while a Annoying Notification is triggered via ANS to a Rickety Queue Storage sent to someone via Shaky Email Service that eventually ends up in someone's Junk Mail folder in O365 but contains a link to the insecure bucket with the entire company's payroll data in it.
:D LOL
As someone getting paid for exactly that, 1000 internets to you!
Kubernetes is just fine. Don't use it for your personal projects, or for stuff that can easily be managed with systemd services. For those, it's probably overkill. But k8s is not some horrible, soul damaging software either. It's actually amazing at what it does.
Thank god I don’t need this, but the Lens folks are great and congrats to them. I suspect this is custom built for a tiny handful of supermassive clients, so hopefully they land the contracts!
As overly complex as all this is, in a few more years there will be very little difference between any given datacenter and the feature-set AWS datacenter software brings to an AWS datacenter.
I do wish kubernetes had headed down to the startup world instead of up into the enterprise world. I’d take a complex standard over 500 pounds of “simple” glue/bash/Jenkins any day of the week.
The idea of data centers and the providers that may or may not own or run atop any given data center is the rationale behind our entire architecture at my business!
We started by adopting a common OS, and selecting against the use of entirely-proprietary services or tools.
Now our server management control plane is code in git, executed by applications in Kubernetes. As long as we can get a connection to the app or server, we can now pretty much take our system to any environment that runs the expected OS. GCP, AWS, Azure, Linode, DigitalOcean, OVH bare metal, you name it.
This is great. It's pretty easy to standup a Kubernetes cluster on bare metal these days. Projects like k0s are for exactly that: running Kube (cluster or single nodes) on bare metal. But ideally if you're a mid or large size entity you want to be able to stand up multiple clusters, for different teams or projects, to have some isolation. Cattle not pets, for clusters. Moving from a one to many mode.
So k0smotron is a way to be able to stand-up/upgrade/manage many k0s bare-metal clusters, via Kubernetes itself. Clusters are - like everything in Kubernetes - just resources (defined by CRDs). K0smotron can go setup & manage Kubernetes clusters on the bare-metal nodes it's given, via typical kubectl operations.
Personally I think this is really great. It's making me consider switching off my own Ansible-based k3s installs. I already run a lot of small single-node clusters here & there, and a couple very modest bigger clusters, and it feels like I could consolidate my management footprint considerably by using this fleet-of-clusters operators.
I know a guy who runs a small company. He contracted a tech wiz to set up the initial infrastructure. AWS, EKS, all built with Terraform modules that wrapped other terraform modules, some of which wrapped helm charts. The "proper" way to do it, right! Russian dolls, each with their own names, pinned version and weird limitations!
Everything was fine for a year or two, but suddenly he noticed a warning message in the EKS console. The latest version available was 1.24. There's an upgrade button. "How hard could this be?"
Well, very hard, as it happens. Time to take apart the Russian dolls! This is where pinning versions and wrapping modules becomes a gigantic pain. To see how one surface level change is made you need to go backwards, one doll at a time, hoping that the important option is exposed as a variable in the smallest doll and every doll that subsequently encases it, and that the name of the variable names don't change anywhere (because sometimes infrastructure engineers wrap a whole bunch of changes as a "set" with one big on/off switch - again, a sensible approach sometimes!).
It looked like every single module and helm chart was badly out of date. If our unfortunate CEO tried to do a terraform apply from scratch, none of this code would work any more. So what do we do?
How about forgetting how the cluster got there, ditching all the decaying terraform and helm crap and simply pushing the "Upgrade" button!? As you might expect, the whole thing fell apart. Time to bring in help!
On inspecting the change logs there were a heap of breaking changes; from renamed apiVersions to removal of deprecated APIs to new requirements for CNI and EBS volumes to entire sections of "we don't do it that way any more". His Persistent Volumes wouldn't attach, Cert-Manager no longer worked, his ingresses disappeared because they didn't specifically name an ingressClass, one pod even had a direct dependence on docker being the container runtime (and that was no longer the case).
Fixing all of this directly, ie: ignoring the decayed terraform code that was used to build it, took considerable time. You need to get new helm repos and charts, inspect successive upgrade guides for breaking changes, extract current helm values for each installed chart and update them, then remove old charts (and manually delete CRDs) and try installing new ones. You can't use the monitoring solution because it died in the fire. Let's use OpenLens to check logs and events. Very slowly put the pieces together: fix Ingress, DNS, TLS, ENS/CNI addons (and the IRSA crap to make them work) and finally upgrade your product charts so they are compatible.
Doing that via Russian doll terraform and helm would have been absolute hell.
Q: Does any tool in existence make this type of problem go away? EKS, GKE, OKE already kinda manage the control plane, and I'm happy to let them do it. FluxCD and ArgoCD (and others) help you manage your deployments. What sews it all up and watches everything, making your Russian dolls go away?
I run my own cluster on bare metal. My control plane runs on 3 separate notes over a Postgres db. So that part is as solid as the managed solutions. Thank God no ETCD.
However, I have been burned many times by keeping critical infra things inside the cluster, so I only deploy stateless apps there. Everything I build does not need storage orchestration, it can just die and pop up elsewhere. Only storage I have is local RWO. No weird CSIs.
No TF. I manage my k3s cluster using k3sup with ansible. I use cherts directly (cert manager, ingress…and that’s it).
Upgrades are usually smooth (I did go through the whole ingressClass thing). I think all this would be much harder the more abstractions you have. I think k8s alone is peak abstraction so I try to make everything as raw as possible. Otherwise I don’t think any tool can save you besides hiring a full time infra engineer. No tool can guarantee a perfect upgrade path because it will not know how parts of your setup and apps are affected by the upgrade.
