So in Part 1, I was pretty enthusiastic about managed Kubernetes. I’d just spun up a DigitalOcean cluster and deployed Temporal with a handful of commands. Mission accomplished, right?

Well, fast forward about two years and I’ve moved everything off of managed K8s and onto a self-hosted setup on Hetzner. This post is about what changed and why.

What Happened?⌗

The short answer is: I ran it for a while and learned what I actually needed versus what I was paying for.

When I wrote Part 1, my main goal was to learn Kubernetes without dealing with cluster management overhead. That made sense. I needed to understand K8s as a developer tool and a DigitalOcean managed cluster let me focus on that. But after running production workloads on it for a couple years, I started noticing some things:

  1. The bill kept growing. What started as ~$25/mo for the cluster + $15/mo for the load balancer kept creeping up as I added nodes, storage, and bandwidth. Eventually I was paying a few times that with no end in sight.

  2. My workloads were pretty stable. I wasn’t doing the kind of dynamic scaling that K8s excels at. Most of my stuff just…runs. Temporal, some web services, background workers. Nothing that needs to scale from 10 to 1000 instances on demand. Also, the Temporal service I was running (which was my main motivation for using K8s in the first place) didn’t need to be highly available or fault tolerant.

  3. I’d learned K8s well enough. The original goal—understanding how to deploy and manage applications in Kubernetes—was accomplished. I didn’t need the training wheels anymore.

  4. I was paying for features I never used. Multi-zone redundancy? Automatic failover? Managed upgrades? All nice to have, but I wasn’t running anything that required that level of operational sophistication.

I ran the numbers and realized I was money on infrastructure that was essentially idle most of the time. That’s when I started looking at alternatives. Let’s be real, those dollars could be better spent on something else like sweet sweet LLM tokens.

The Hetzner Setup⌗

In the past couples years, Hetzner has popped up as a wonderful alternative to DigitalOcean (who ironically used to be the budget provider when I got started tinkering with cloud providers). Plus the hcloud CLI is a super easy to use, so why not. So I booted a small machine and got hacking.

For comparison, my DigitalOcean K8s cluster with 4 nodes (2 vCPU, 4GB RAM each) plus the load balancer was costing me about $100/mo at minimum, and that’s before adding any meaningful storage or bandwidth overages. The Hetzner box gives me way more compute for that price, or the same compute for about 25% of the cost. In other words, it’s a pretty good deal.

The Kubernetes Setup⌗

I’m using K3s for my Kubernetes cluster. It’s a lightweight version of Kubernetes that’s designed for edge computing and is a great fit for my use case. Coupled with hcloud and k3sup, you can go from zero to kubeconfig in literally a minute:

hcloud datacenter list
hcloud datacenter describe hil-dc1
hcloud image list
hcloud server create --name k8s-node --image ubuntu-22.04 --type cx22 --location hil-dc1 --ssh-key ~/.ssh/your-ssh-key.pub
k3sup install --ip XXX.XXX.XXX.XXX
export KUBECONFIG=$(pwd)/kubeconfig
kubectl get node -o wide

That’s it! I can now use kubectl to manage my cluster. This where the “fun” part starts, serializing my existing K8s resources into YAML and then applying them to the new cluster. I won’t bore you with this part; fortunately my K8s stuff is entirely stateless (all my databases are hosted elsewhere), so this wasn’t too bad.

Bonus: Tailscale Revisited⌗

In my first post, I used Tailscale as a proxy to access my Temporal web UI. This was ok, but it wasn’t ideal. The biggest issue was that Tailscale was only exposing that single service and it required running a pod in this sort of sidecar pattern.

Since I’m writing about this, I might as well mention how I like to use Tailscale now. Instead of using a proxy pod, I just use the Tailscale operator which only requires you to add the tailscale.com/expose: "true" annotation on the Service you want to expose. Cool! Read the Tailscale docs here; I’ll summarize the relevant parts below.

First, update your tailnet policy to have the following tags:

"tagOwners": { "tag:k8s-operator": [], "tag:k8s": ["tag:k8s-operator"] }

Create an OAuth client in the Tailscale console and get the CLIENT_ID and CLIENT_SECRET. Now you can install the Tailscale operator via Helm:

# 1. Install the operator
helm repo add tailscale https://pkgs.tailscale.com/helmcharts
helm repo update
helm install tailscale tailscale/tailscale-operator --namespace tailscale --create-namespace --set-string oauth.clientId=YOUR_CLIENT_ID --set-string oauth.clientSecret=YOUR_CLIENT_SECRET

Done! Now you can deploy a pod you can reach via Tailscale (the only relevant part is the tailscale.com/expose: "true" annotation on the Service):

apiVersion: v1
kind: Pod
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  containers:
    - name: httpbin
      image: kennethreitz/httpbin
      ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  annotations:
    tailscale.com/expose: "true"
spec:
  selector:
    app: httpbin
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

Finally, ping the service from your device:

curl http://default-httpbin

Cool! Now I can reach the service from anywhere in my Tailnet, and all it took was a single annotation!!!

Tradeoffs⌗

Let me be specific about the savings:

DigitalOcean (old setup): $110/mo Hetzner (current setup): $30/mo

So I’m spending about 1/4th of what I was before, with better performance and more resources available.

That’s the good; what’s the bad?

  1. More manual scaling. If I need to scale up my services, I need to manually provision more capacity and it probably won’t be as seamless as with DO’s managed service.

  2. More manual upgrades. I need to actually SSH in and update things myself. DigitalOcean was handling K8s version upgrades and security patches for me.

  3. Single point of failure. No multi-zone redundancy. If my Hetzner box dies, my services are down until I restore from backups.

For my use case, none of these tradeoffs matter much. If I was running a SaaS product with thousands of users depending on uptime, the calculation would be different. But for personal infrastructure and side projects? The simplicity and cost savings are worth it.

Final Thoughts⌗

The lesson here isn’t “managed Kubernetes is bad” or “everyone should self-host”. The lesson is that you should periodically re-evaluate your infrastructure choices as your understanding and needs evolve.

Two years ago, managed K8s was the right choice for me. It let me learn the tool without the operational burden. But now that I understand what I actually need, a simpler self-hosted setup makes more sense.

Cool, now jump into the comments and let me know if you’ve learned any painful lessons yourself doing one of these migrations, or if you have any questions I can help answer!