Infrastructure & Engineering

Auto-build K8s with Proxmox & cloud-init: Templating Method

Otama
2 min read
Progress 4 / 18
Table of Contents

Recently, I often rebuild Kubernetes environments myself, and getting tired of digging through logs going “Huh? How did I do it last time?”, I made something template-like to reproduce it more easily.

It is based on the flow written in the previous article (here), but this time I’m elaborating the config file for cloud-init while organizing and splitting it a bit. Doing it manually every time is fine, but automating it gives a bit of peace of mind for “reproducibility” and “myself when I forget”.

So, the content this time is a story about making a template to quickly create a Kubernetes environment with Proxmox + cloud-init. It’s strictly a memo for myself, so I haven’t included CNI settings etc. yet, but it still reduced effort considerably.

Procedure

The rough procedure is same as the previous article. I will customize only the cloud-init config file for this time.

UbuntuのCloud ImageでテンプレートVMを作ってみた話【Proxmox + cloud-init】

>-

blog.otama-playground.com

Create yaml

I will drop the contents of the following article into cloud-init config file. I’m still checking various things around CNI, so I haven’t included it in the template.

おうちKubernetesを構築してみた話【kubeadm + CRI-O + Cilium】

>-

blog.otama-playground.com

Make per phase

Enable IPv4 Forwarding

#cloud-config
write_files:
  - path: /usr/local/bin/enable-ipv4-forwarding.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      cat <<EOF | tee /etc/modules-load.d/k8s.conf
      overlay
      br_netfilter
      EOF


      modprobe overlay
      modprobe br_netfilter


      cat <<EOF | tee /etc/sysctl.d/k8s.conf
      net.bridge.bridge-nf-call-iptables  = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward                 = 1
      EOF


      sysctl --system


runcmd:
  - /usr/local/bin/enable-ipv4-forwarding.sh

Install Container Runtime. Since variable substitution doesn’t work well, specifying version directly for now.

#cloud-config
packages:
  - software-properties-common
  - curl


write_files:
  - path: /usr/local/bin/install-crio.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      curl -fsSL https://download.opensuse.org/repositories/isv:/cri-o:/stable:/v1.32/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg


      echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://download.opensuse.org/repositories/isv:/cri-o:/stable:/v1.32/deb/ /" | tee /etc/apt/sources.list.d/cri-o.list


      apt-get update
      apt-get install -y cri-o


      systemctl start crio


runcmd:
  - /usr/local/bin/install-crio.sh

Disable swap

#cloud-config
write_files:
  - path: /usr/local/bin/disable-swap.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      sed -i '/ swap / s/^$.*$$/#\1/g' /etc/fstab


runcmd:
  - /usr/local/bin/disable-swap.sh

Install kubeadm

#cloud-config
write_files:
  - path: /usr/local/bin/install-kubernetes.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg


      echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list


      apt-get update
      apt-get install -y kubelet kubeadm kubectl
      apt-mark hold kubelet kubeadm kubectl


runcmd:
  - /usr/local/bin/install-kubernetes.sh

Install helm

#cloud-config
packages:
  - apt-transport-https


write_files:
  - path: /usr/local/bin/install-helm.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | tee /usr/share/keyrings/helm.gpg > /dev/null
      echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | tee /etc/apt/sources.list.d/helm-stable-debian.list


      apt-get update
      apt-get install helm


runcmd:
  - /usr/local/bin/install-helm.sh

Organize into 1 yaml

#cloud-config
packages:
  - software-properties-common
  - curl
  - apt-transport-https


write_files:
  - path: /usr/local/bin/enable-ipv4-forwarding.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      cat <<EOF | tee /etc/modules-load.d/k8s.conf
      overlay
      br_netfilter
      EOF


      modprobe overlay
      modprobe br_netfilter


      cat <<EOF | tee /etc/sysctl.d/k8s.conf
      net.bridge.bridge-nf-call-iptables  = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward                 = 1
      EOF


      sysctl --system
  - path: /usr/local/bin/install-crio.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      curl -fsSL https://download.opensuse.org/repositories/isv:/cri-o:/stable:/v1.32/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg


      echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://download.opensuse.org/repositories/isv:/cri-o:/stable:/v1.32/deb/ /" | tee /etc/apt/sources.list.d/cri-o.list


      apt-get update
      apt-get install -y cri-o


      systemctl start crio
  - path: /usr/local/bin/disable-swap.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      sed -i '/ swap / s/^$.*$$/#\1/g' /etc/fstab
  - path: /usr/local/bin/install-kubernetes.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg


      echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list


      apt-get update
      apt-get install -y kubelet kubeadm kubectl
      apt-mark hold kubelet kubeadm kubectl
  - path: /usr/local/bin/install-helm.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | tee /usr/share/keyrings/helm.gpg > /dev/null
      echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | tee /etc/apt/sources.list.d/helm-stable-debian.list


      apt-get update
      apt-get install helm


runcmd:
  - /usr/local/bin/enable-ipv4-forwarding.sh
  - /usr/local/bin/install-crio.sh
  - /usr/local/bin/disable-swap.sh
  - /usr/local/bin/install-kubernetes.sh
  - /usr/local/bin/install-helm.sh

Embed

Copy to snippet on Proxmox host and embed it into VM.

Terminal window
# Copy to snippets
cat << EOF | sudo tee /var/lib/vz/snippets/install-kubeadm.yaml
...Copy paste created yaml
EOF


# Embed cloud-init config file
vmTemplateId = 9000
qm set ${vmTemplateId} --cicustom "vendor=local:snippets/install-kubeadm.yaml"

Startup

Start and check install status of kubeadm and cri-o.

Terminal window
# Success if Active
systemctl status crio


# Success if not command not found
kubeadm version


# Success if not command not found
helm version

Conclusion

So, with this templating, the effort of environment construction was significantly reduced. However, since all processing is currently executed serially in runcmd, there is still room for improvement in terms of performance and flexibility.

Actually, it would be smarter to allow parallel execution like this, but that’s for when I feel like it (lol). I made this config file prioritizing “get something working in shortest time” for now, so I hope to grow it while eyeing future expansions.

I hope this helps those who are “rebuilding Kubernetes environments again and again~”. See you.

Related Posts