Kubernetes 클러스터 업그레이드.

v1.29.15에서 v1.35.1로 올리는 작업.

처음에는 바로 점프하려고 했다.

sudo kubeadm upgrade apply v1.35.1 -y

에러.

error: [upgrade] FATAL: this version of kubeadm only supports deploying 
clusters with the control plane version >= 1.34.0. Current version: v1.29.15

Kubernetes는 마이너 버전 하나씩 올려야 함.

저장소 버전도 맞춰야 함.

실패했던 케이스.

sudo sed -i "s|v1.29|v1.35|g" /etc/apt/sources.list.d/kubernetes.list

sudo apt-get install kubeadm=1.30.*-1.1

에러.

E: Version '1.30.*-1.1' for 'kubeadm' was not found

hold 패키지도 문제.

sudo apt-get install kubeadm=1.30.*-1.1

에러.

E: Held packages were changed and -y was used without --allow-change-held-packages

수정.

sudo apt-mark unhold kubeadm kubelet kubectl cri-tools
sudo apt-get install -y --allow-change-held-packages kubeadm=1.30.*-1.1

마스터 업그레이드.

sudo apt-mark unhold kubeadm kubelet kubectl cri-tools

for VERSION in 30 31 32 33 34; do
  sudo sed -i "s|v1\.[0-9]*|v1.$VERSION|g" /etc/apt/sources.list.d/kubernetes.list
  sudo apt-get update -qq
  
  sudo apt-get install -y --allow-change-held-packages kubeadm=1.$VERSION.*-1.1
  sudo kubeadm upgrade apply v1.$VERSION.0 -y
  
  sudo apt-get install -y --allow-change-held-packages kubelet=1.$VERSION.*-1.1 kubectl=1.$VERSION.*-1.1
  
  sudo systemctl daemon-reload
  sudo systemctl restart kubelet
  sleep 15
done

v1.35.1 최종 적용.

sudo sed -i "s|v1\.[0-9]*|v1.35|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -qq
sudo apt-get install -y --allow-change-held-packages kubeadm=1.35.1-1.1
sudo kubeadm upgrade apply v1.35.1 -y
sudo apt-get install -y --allow-change-held-packages kubelet=1.35.1-1.1 kubectl=1.35.1-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

워커 업그레이드.

마스터에서 drain.

kubectl drain k8s-worker1 --ignore-daemonsets --delete-emptydir-data

워커에서 같은 업그레이드 과정 실행.

마스터에서 uncordon.

kubectl uncordon k8s-worker1

worker2도 동일.

업그레이드 후 노드가 NotReady.

증상.

Kubelet stopped posting node status.

kubelet 로그.

E0218 14:28:33.572037 1412329 run.go:72] 
"command failed" err="failed to parse kubelet flag: unknown flag: --pod-infra-container-image"

v1.35.1에서 --pod-infra-container-image 플래그가 제거됨.

각 노드에서 제거.

sudo sed -i "s/ --pod-infra-container-image=registry.k8s.io\/pause:[^ ]*//g" \
  /var/lib/kubelet/kubeadm-flags.env

변경 전.

KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"

변경 후.

KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock"

kubelet 재시작.

sudo systemctl restart kubelet

최종 확인.

kubectl get nodes

결과.

k8s-master
Ready
v1.35.1

k8s-worker1
Ready
v1.35.1

k8s-worker2
Ready
v1.35.1