1. 개요
쿠버네티스 노드의 클러스터 조인에 대한 내용이다.
2. 사전 정보
- 설치해야 할 것 : https://sarc.io/index.php/cloud/1315-kubeadm-toolkit 을 참고하라.
- 쿠버네티스 클러스터는 : 하나의 마스터와 여러 노드(=worker, minion)로 이루어져 있음
3. 내용
사전에 설치되어야 할 것이 다 설치되었다면, 마스터에 접속을 하겠다.
참고로 "kubeadm join 10.0.10.220:6443 --token 09si41.rjipcw83obk3vna4 --discovery-token-ca-cert-hash sha256:fc629584400772a5f0b61f4579a317399b1b430793e28129206ed02ea1882134"는 마스터 노드에서 kubeadm init 를 실행하였을 때 나온 커맨드 라인이다.
# kubeadm join 10.0.10.220:6443 --token 09si41.rjipcw83obk3vna4 --discovery-token-ca-cert-hash sha256:fc629584400772a5f0b61f4579a317399b1b430793e28129206ed02ea1882134
[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
I0807 04:27:53.370349 14840 kernel_validator.go:81] Validating kernel version
I0807 04:27:53.370517 14840 kernel_validator.go:96] Validating kernel config
[discovery] Trying to connect to API Server "10.0.10.220:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.10.220:6443"
실패했다. 결론적으로는 /etc/hosts 에 호스트 자신에 대한 정보가 없었다. 해당 내용을 추가하고 다시 실행한다.
# kubeadm join 10.0.10.220:6443 --token 09si41.rjipcw83obk3vna4 --discovery-token-ca-cert-hash sha256:fc629584400772a5f0b61f4579a317399b1b430793e28129206ed02ea1882134
[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
I0807 04:27:53.370349 14840 kernel_validator.go:81] Validating kernel version
I0807 04:27:53.370517 14840 kernel_validator.go:96] Validating kernel config
[discovery] Trying to connect to API Server "10.0.10.220:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.10.220:6443"
이번에는 이 상태로 계속 멈춰있었는데. 결론적으로는 Security Group에서 6443 포트가 막혀있었기 때문이다. 6443을 열고 다시 실행한다.
# kubeadm join 10.0.10.220:6443 --token 09si41.rjipcw83obk3vna4 --discovery-token-ca-cert-hash sha256:fc629584400772a5f0b61f4579a317399b1b430793e28129206ed02ea1882134
[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr] or no builtin kernel ipvs support: map[ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
I0807 04:28:53.156400 14915 kernel_validator.go:81] Validating kernel version
I0807 04:28:53.156614 14915 kernel_validator.go:96] Validating kernel config
[discovery] Trying to connect to API Server "10.0.10.220:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.10.220:6443"
[discovery] Requesting info from "https://10.0.10.220:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.0.10.220:6443"
[discovery] Successfully established connection with API Server "10.0.10.220:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ip-10-0-10-196" as an annotation
This node has joined the cluster:
* Certificate signing request was sent to master and a response
was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
4. 마스터 노드에서 확인
4-1. 연결되기 전
$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-10-220 NotReady master 1h v1.11.1
4-2. 연결된 후
$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-10-196 NotReady 45s v1.11.1 ip-10-0-10-220 NotReady master 2h v1.11.1
참고로 NotReady 상태인 이유는 아직 CNI(Container Network Interface)가 배포되지 않았기 때문.
5. CNI 배포
그러면 CNI를 배포한다. calico 기준이다.
$ kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml configmap/calico-config created daemonset.extensions/calico-etcd created service/calico-etcd created daemonset.extensions/calico-node created deployment.extensions/calico-kube-controllers created deployment.extensions/calico-policy-controller created clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin created clusterrole.rbac.authorization.k8s.io/calico-cni-plugin created serviceaccount/calico-cni-plugin created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created serviceaccount/calico-kube-controllers created
node 상태를 본다.
$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-10-196 NotReady 48m v1.11.1 ip-10-0-10-220 NotReady master 2h v1.11.1
아직 NotReady 상태에 머물러있다. systemctl restart kubelet 하여 재기동한 후.. 다시 상태를 본다.
$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-10-196 Ready 49m v1.11.1 ip-10-0-10-220 NotReady master 2h v1.11.1
마스터가 Ready로 올라왔다. 그리고 조금 더 기다리면 노드도 Ready로 올라온다.
$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-10-196 Ready 51m v1.11.1 ip-10-0-10-220 Ready master 3h v1.11.1
확실치 않은 부분 : CNI 배포 후 kubelet restart를 꼭 해줘야 하는지, 아니면 기다리면 자동으로 마스터가 Ready가 되는지 여부.