MicroK8s to K3s Migration: A Clean Hybrid Foundation

What distribution are you using for your homelab Kubernetes?

In our house, we had been relying on MicroK8s as our base. It is absolutely fantastic for quickly spinning up a cluster on an Ubuntu server, and it has served us very well.

However, we are currently planning to build a somewhat unique hybrid architecture that will combine an Ubuntu (Incus VM) control plane with a Mac Studio GPU virtual node via Virtual Kubelet. With an eye toward constructing and operating this highly programmable infrastructure in the future, we realized we needed to reevaluate our distribution and clean up our foundation now.

That’s why we made the leap to k3s. Here are the two main reasons:

1. The Purity Needed for Custom Virtual Kubelet Provider Development

This is the biggest motivator. Down the line, we plan to integrate native processes on a Mac Studio as a k8s node. To be completely honest, there is no wildly popular de facto standard tool for Mac Virtual Kubelets right now, leaving us with the strong possibility that we might just have to build our own provider from scratch.

While the API connection itself might not differ drastically from MicroK8s, we strongly expect that targeting the pure, open binary of k3s will make troubleshooting much easier when our custom tool inevitably hits weird errors. MicroK8s’s Snap package encapsulates the host OS’s network and certificates, which adds an opaque layer we’d rather avoid when debugging.

2. Footprint and Maintainability within Incus VMs

When running on our foundational Incus VMs, the “elegance” and “lightness” of the system really matter.

MicroK8s relies heavily on snapd, and its background auto-updates and complex loopback mounts quietly eat into the VM’s resources. On the other hand, k3s aggressively strips out unnecessary drivers and operates entirely on a single binary and a single systemd service. Its memory footprint as a control plane is incredibly small, keeping the inside of the VM surprisingly clean.

The verdict? Switching to k3s really streamlined things. We now have a clean, highly visible foundation ready for our future cross-environment, cross-architecture integrations.

However, behind what was supposed to be a “quick and easy” migration, we stepped on quite a few subtle landmines. I’ll summarize these pitfalls here as a memo—ranging from OS selection blunders to k3s-specific traps and tips for a safe migration.

The OS Selection Maze: Is Latest Always Greatest?

Since we were rebuilding the cluster anyway, I figured we might as well upgrade the OS.

Chanting “latest is greatest!”, I excitedly grabbed the freshly released Ubuntu 26.04 (Resolute). However, the moment I started the setup, apt update just froze entirely on Waiting for headers. Absolute despair.

No matter how many times I rebooted or changed mirrors, it was useless. Feeling somewhat enlightened, I muttered, “Ah, new releases are always full of traps… ‘More haste, less speed’ is truly a universal law of infrastructure,” and made the decision to downgrade back to the stable 24.04 LTS.

A Lifesaving apt Tip (And an Unexpected Plot Twist)

So, we rolled back to the trusty 24.04 LTS. But guess what? We ran into the exact same apt communication error.

“I downgraded the OS, and it’s STILL happening?!” I was ready to tear my hair out. But after a quick search, it turns out that since Ubuntu 24.04, the default http mirror connection can be unstable depending on your environment. Simply changing http to https in /etc/apt/sources.list.d/ubuntu.sources solved it instantly.

sudo sed -i 's/http:\/\/archive.ubuntu.com/https:\/\/archive.ubuntu.com/g' /etc/apt/sources.list.d/ubuntu.sources

And then, it hit me. “Wait… was that not a 26.04 bug at all? Was it just a mirror communication error the whole time…?”

I decided to try reinstalling 26.04 one more time and ran that HTTPS one-liner. Lo and behold, it worked perfectly. The time I wasted downgrading was completely for nothing! In the end, we are happily running on the latest Ubuntu 26.04.

These kinds of “detours caused by misunderstandings” really pile on the stress, so it’s best to bake this HTTPS fix into your initial setup scripts.

K3s Installation: The “Subtraction” Setup

Once the OS was ready, it was time to install K3s.

One of K3s’s strengths is that it comes bundled with Traefik and ServiceLB by default, making it usable out of the box. However, in our environment, we wanted to manage our Ingress and load balancer with our own custom Helm Chart (Traefik + MetalLB), so we didn’t need the built-in ones.

Therefore, we use a “subtraction” configuration for the installation command:

curl -sfL https://get.k3s.io | sh -s - --disable traefik --disable servicelb

With just this one line, a clean, stripped-down Kubernetes cluster spins up. Being able to easily decouple the parts you want to control yourself really highlights the excellent design of K3s.

”The Command Isn’t Working!” The Kubeconfig Trap

Installation complete! I excitedly typed kubectl get nodes to check the status… and was greeted with an error.

I had grown so accustomed to MicroK8s’s microk8s kubectl command structure that I completely forgot to configure where standard kubectl and helm look for their configs.

With K3s, the kubeconfig is generated at /etc/rancher/k3s/k3s.yaml. You need to perform a quick “ritual” to export it as an environment variable and adjust the permissions so your regular user can read it.

echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc
sudo chmod 644 /etc/rancher/k3s/k3s.yaml

You only have to set it up once, but this is a classic trap you only fall into during a migration.

The Resurrection of GitOps (Argo CD)

Once you get past that point, it’s smooth sailing. As long as you get Argo CD into the cluster, it will automatically pull your definitions from Git and restore all your applications to their proper state.

By just deploying a single root application, the entire infrastructure and all apps deploy themselves like a chain reaction. It was a great reminder of just how powerful the “App-of-Apps” pattern truly is.

The TLS Certificate Ritual: Verifying Survival in Staging

With the migration, Traefik and Cert-Manager are also rebuilt. The one thing you absolutely must NOT do here is immediately try to fetch a Production Let’s Encrypt certificate.

I’ve had bitter experiences in the past where repeated configuration errors caused me to hit the Let’s Encrypt rate limits, making it impossible to issue certificates for several days. It is simply hell.

So, right after rebuilding, you should always switch to a Let's Encrypt Staging Issuer to confirm that the DNS-01 challenge passes correctly and a dummy certificate is issued.

# Use Staging during verification
clusterIssuer: letsencrypt-staging-cloudflare

You’ll get a “Not Secure” warning in your browser, but if that dummy certificate is properly applied, you’ve essentially won. Once you’ve confirmed it works, you rewrite that single line and switch to Production.

# Switch to this for Production
clusterIssuer: letsencrypt-production-cloudflare

Just adding this one little buffer step makes an incredible difference for your peace of mind.

Conclusion

Despite the Ubuntu version selection maze, apt communication errors, and Kubeconfig traps, the actual migration to k3s really was a “quick and easy” process once we got past those hurdles.

While MicroK8s is fantastic when you just want to “spin up k8s as fast as possible,” if you are aiming for a highly programmable infrastructure—like our plan to integrate Mac Studio GPU resources via Virtual Kubelet on top of an Incus VM virtualization layer—k3s, with its lightweight, open nature and massive ecosystem benefits, feels like the most fitting foundation.

Sprinkling in a bit of “professionalism”—like safety measures to avoid rate limits—adds a nice touch to the speed of a “quick migration.” It wasn’t a bad way to spend a weekend tinkering with infrastructure.

If you’re planning a migration towards a slightly more complex architecture, I hope this serves as a helpful reference.