安装高可用集群(自动上传证书)

主机准备

准备三台主机，和一个VIP

• master1：192.168.2.104

• master2：192.168.2.105

• master3：192.168.2.106

• vip：192.168.2.110

在每台主机的/etc/hosts中添加记录

192.168.2.110    apiserver.dcos.com

安装docker、kubeadm、kubelet、keepalived

安装master

下载镜像

下载镜像到各master节点

k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/kube-apiserver:v1.17.0
k8s.gcr.io/kube-scheduler:v1.17.0
k8s.gcr.io/kube-controller-manager:v1.17.0
k8s.gcr.io/kube-proxy:v1.17.0
k8s.gcr.io/coredns:1.6.5
k8s.gcr.io/pause:3.1 

calico/node:v3.8.5
calico/cni:v3.8.5
calico/pod2daemon-flexvol:v3.8.5
calico/kube-controllers:v3.8.5

安装master1

$ kubeadm init --control-plane-endpoint apiserver.dcos.com:6443 --upload-certs --pod-network-cidr 172.26.0.0/16 --kubernetes-version 1.17.0

安装成功后，会有类似下面的输出

...
You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join apiserver.dcos.com:6443 --token 61eu2f.f4dd6lucgaf13w9i \
    --discovery-token-ca-cert-hash sha256:e117e39b455a8dbab863d76f2b3b4a74051901a989721cf86f1cf96b12fe6b44 \
    --control-plane --certificate-key a301c9c55596c54c5d4c7173aa1e3b6fd304130b0c703bb23149c0c69f94b8e0

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join apiserver.dcos.com:6443 --token 61eu2f.f4dd6lucgaf13w9i \
    --discovery-token-ca-cert-hash sha256:e117e39b455a8dbab863d76f2b3b4a74051901a989721cf86f1cf96b12fe6b44

然后，安装网络插件，首选下载下来

$ wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml

然后编辑里面的192.168.0.0/16为上面的172.26.0.0/16

安装其他master

根据上面的输出，可以执行下面的命令安装其他的master

$ kubeadm join apiserver.dcos.com:6443 --token 61eu2f.f4dd6lucgaf13w9i --discovery-token-ca-cert-hash sha256:e117e39b455a8dbab863d76f2b3b4a74051901a989721cf86f1cf96b12fe6b44 --control-plane --certificate-key a301c9c55596c54c5d4c7173aa1e3b6fd304130b0c703bb23149c0c69f94b8e0

但是，如果过了两个小时，--certificate-key会失效，此时需要重新upload，执行以下命令：

$ kubeadm init phase upload-certs --upload-certs
...
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
e1cad9c1c339100e1946c19b930e18c2809fcc59e5f6d44cb0a1b7d7d7862079

然后使用上面新的值e1cad9c1c339100e1946c19b930e18c2809fcc59e5f6d44cb0a1b7d7d7862079

$ kubeadm join apiserver.dcos.com:6443 --token 61eu2f.f4dd6lucgaf13w9i --discovery-token-ca-cert-hash sha256:e117e39b455a8dbab863d76f2b3b4a74051901a989721cf86f1cf96b12fe6b44 --control-plane --certificate-key e1cad9c1c339100e1946c19b930e18c2809fcc59e5f6d44cb0a1b7d7d7862079

执行成功后，查看节点信息，添加成功

kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
peng04   Ready    master   17h   v1.17.0
peng05   Ready    master   35s   v1.17.0

错误排查

Q1: 主机上网络插件calico安装不成功，卡在了Init容器？

首先查看容器的日志

$ kubectl logs calico-node-xxxxx -c <container> -n kube-system

然后清理掉主机的上的目录，再重启网络插件试试

/var/lib/cni/networks
/host/opt/cni
/host/etc/cni
/host/driver
/run/xtables.lock
/var/run/calico
/var/lib/calico
/var/run/nodeagent
/etc/cni
/opt/cni