オンプレミスな環境にkubernetesを構築

2024年2月14日

environment

  • ubuntu22.04
  • kubernetes
    • version 1.28.2
    • control node * 1
    • worker node * 2

VMなので1台作り、kubernetes環境が初期状態で複製しました。

install kubernetes

https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://dl.k8s.io/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
whereis kubectl

install docker

add-apt-repository   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
apt-get update && apt-get install -y containerd.io docker-ce docker-ce-cli
systemctl list-unit-files|grep -i docker
cat /etc/docker/daemon.json
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
   "storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker
docker ps

set iptables

sudo apt-get install -y iptables arptables ebtables
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy
systemctl daemon-reload
systemctl restart kubelet
kubectl get nodes
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
swapon -s
swapoff /swap.img 
vi /etc/fstab

control planeでinit

root@k8s-cont01:/home/ocarina# kubeadm init --control-plane-endpoint 172.31.1.1:6443 --pod-network-cidr=10.255.0.0/16 --upload-certs
I0212 22:10:29.072594    1973 version.go:256] remote version is much newer: v1.29.1; falling back to: stable-1.28
[init] Using Kubernetes version: v1.28.6
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-cont01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.31.1.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-cont01 localhost] and IPs [172.31.1.1 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-cont01 localhost] and IPs [172.31.1.1 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 74.022769 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
0febb335b6d9438814b8ae24673af74811c9dc7fa4ebae5d202689071594ea7f
[mark-control-plane] Marking the node k8s-cont01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-cont01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: exonyk.7rrbqyzc533da4ft
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 172.31.1.1:6443 --token exonyk.7rrbqyzc533da4ft \
    --discovery-token-ca-cert-hash sha256:4e8085fde3a129befbdbba029da2695aaf935203adcde36b5c708dc22266adee \
    --control-plane --certificate-key 0febb335b6d9438814b8ae24673af74811c9dc7fa4ebae5d202689071594ea7f

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.31.1.1:6443 --token exonyk.7rrbqyzc533da4ft \
    --discovery-token-ca-cert-hash sha256:4e8085fde3a129befbdbba029da2695aaf935203adcde36b5c708dc22266adee

errorの場合

reset

systemctl stop kubelet
 yes | kubeadm reset
  iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X && ipvsadm -C
systemctl restart containerd

error回避

https://github.com/kubernetes/kubeadm/issues/2699#issuecomment-1353250314

systemctl stop kubelet
\rm /etc/containerd/config.toml
containerd config default | sudo tee /etc/containerd/config.toml
sed -ie 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
PAUSE_IMAGE=$(kubeadm config images list | grep pause 2>/dev/null)
echo $PAUSE_IMAGE
sudo -E sed -i "s,sandbox_image = .*,sandbox_image = \"$PAUSE_IMAGE\",g" /etc/containerd/config.toml
systemctl restart containerd

もう一度init

workerのjoin

https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#join-nodes

root@k8s-worker02:/home/ocarina# kubeadm join 172.31.1.1:6443 --token exonyk.7rrbqyzc533da4ft \
    --discovery-token-ca-cert-hash sha256:4e8085fde3a129befbdbba029da2695aaf935203adcde36b5c708dc22266adee

configファイルコピー

control1からコピー

scp /etc/kubernetes/admin.conf 172.31.1.2:/etc/kubernetes/.
scp /etc/kubernetes/admin.conf 172.31.1.3:/etc/kubernetes/.

worker1,2 一般ユーザーで

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

CNI

https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/

download

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

install

root@k8s-cont01:/home/ocarina# cilium install
ℹ️  Using Cilium version 1.14.6
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected kube-proxy has been installed
root@k8s-cont01:/home/ocarina# cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             1 errors, 3 warnings
 \__/¯¯\__/    Operator:           1 errors, 1 warnings
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 1, Unavailable: 1/1
DaemonSet              cilium             Desired: 3, Unavailable: 3/3
Containers:            cilium             Pending: 3
                       cilium-operator    Pending: 1
Cluster Pods:          0/0 managed by Cilium
Helm chart version:    1.14.6
Image versions         cilium             quay.io/cilium/cilium:v1.14.6@sha256:37a49f1abb333279a9b802ee8a21c61cde9dd9138b5ac55f77bdfca733ba852a: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.6@sha256:2f0bf8fb8362c7379f3bf95036b90ad5b67378ed05cd8eb0410c1afc13423848: 1
Errors:                cilium             cilium                             3 pods of DaemonSet cilium are not ready
                       cilium-operator    cilium-operator                    1 pods of Deployment cilium-operator are not ready
Warnings:              cilium             cilium-4cfqh                       pod is pending
                       cilium             cilium-9vxxz                       pod is pending
                       cilium             cilium-hhq8z                       pod is pending
                       cilium-operator    cilium-operator-c8cfd6fcf-v5hdh    pod is pending
status check failed

しばらく待つ....

bashroot@k8s-cont01:/home/ocarina# cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Containers:            cilium             Running: 3
                       cilium-operator    Running: 1
Cluster Pods:          2/2 managed by Cilium
Helm chart version:    1.14.6
Image versions         cilium             quay.io/cilium/cilium:v1.14.6@sha256:37a49f1abb333279a9b802ee8a21c61cde9dd9138b5ac55f77bdfca733ba852a: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.6@sha256:2f0bf8fb8362c7379f3bf95036b90ad5b67378ed05cd8eb0410c1afc13423848: 1
root@k8s-cont01:/home/ocarina# 
root@k8s-cont01:/home/ocarina# kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-cont01     Ready    control-plane   77m   v1.28.2
k8s-worker02   Ready    <none>          23m   v1.28.2
k8s-worker03   Ready    <none>          23m   v1.28.2
root@k8s-cont01:/home/ocarina# kubectl get pods -n kube-system | grep cilium
cilium-cp6vw                         1/1     Running   0                 13m
cilium-fq9qx                         1/1     Running   0                 13m
cilium-nxt5h                         1/1     Running   0                 13m
cilium-operator-c8cfd6fcf-pl59q      1/1     Running   6 (3m31s ago)     13m
root@k8s-cont01:/home/ocarina#

test

create pod

ocarina@k8s-worker03:~$ kubectl create deployment nginx --image=nginx:latest
deployment.apps/nginx created
ocarina@k8s-worker03:~$ kubectl expose deployment nginx --type=NodePort --port=80
service/nginx exposed
ocarina@k8s-worker03:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        25h
nginx        NodePort    10.104.210.75   <none>        80:30288/TCP   8s

network

node

root@k8s-cont01:/home/ocarina# iptables -nvL -t nat | grep 302
    0     0 KUBE-EXT-2CMXP7HKUVJN7L6M  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp dpt:30288

コントロールプレーンノード含め全台で同一のnat設定が作られたことを確認した。

ip_forwardは自動で有効になっていた。

root@k8s-cont01:/home/ocarina# grep ip_forward /etc/sysctl.conf 
#net.ipv4.ip_forward=1
root@k8s-cont01:/home/ocarina# sysctl -a | grep ip_forward\ 
net.ipv4.ip_forward = 1

grepした限り、kube-proxyかciliumで自動的に有効にしてるっぽい。

router側でnat

iptables -t nat -A PREROUTING -d 172.31.0.200 -p tcp -m tcp --dport 30288 -j DNAT --to-destination 172.31.1.254

vip

worker nodeにてkeepalivedでvrrp 172.31.1.254 をlistenするように設定

ネットワークの外からの接続確認

http://172.31.0.200:30288/

ocarina@ab350-pro4:~$ curl --head http://172.31.0.200:30288/
HTTP/1.1 200 OK
Server: nginx/1.25.3
Date: Tue, 13 Feb 2024 14:28:24 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 24 Oct 2023 13:46:47 GMT
Connection: keep-alive
ETag: "6537cac7-267"
Accept-Ranges: bytes

Posted by ocarina