CentOS7 单节点 k8s 部署拾遗

Posted on

本文讲使用 kubeadm 来安装 k8s，环境为 CentOS 7.5。

kubeadm 安装

由于无法访问 google，需要配置阿里的镜像仓库来安装 kubeadm：

1
2
3
4
5
6
7
8
9
10
11
12
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

按照文档 https://kubernetes.io/docs/setup/independent/install-kubeadm/ 安装 kubeadm 后，执行 systemctl start kubelet，查看状态，报如下错误：

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Sun 2018-11-18 11:04:07 CST; 54ms ago
     Docs: https://kubernetes.io/docs/
  Process: 2003 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 2003 (code=exited, status=255)

Nov 18 11:04:07 k8s systemd[1]: Unit kubelet.service entered failed state.
Nov 18 11:04:07 k8s systemd[1]: kubelet.service failed.

这里的坑在于文档里不会告诉你在 init 之前是铁定报错的，在 init 之后查看状态一切 OK。汗

kubeadm init 踩坑

按照文档 https://kubernetes.io/docs/setup/independent/install-kubeadm/ 安装 kubeadm，执行初始化，报错。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
[root@k8s ~]# kubeadm init
[init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
	[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 3.10.0-862.14.4.el7.x86_64
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
CONFIG_INET: enabled
CONFIG_EXT4_FS: enabled (as module)
CONFIG_PROC_FS: enabled
CONFIG_NETFILTER_XT_TARGET_REDIRECT: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
CONFIG_OVERLAY_FS: enabled (as module)
CONFIG_AUFS_FS: not set - Required for aufs.
CONFIG_BLK_DEV_DM: enabled (as module)
DOCKER_VERSION: 18.09.0
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
	[WARNING Hostname]: hostname "k8s" could not be reached
	[WARNING Hostname]: hostname "k8s" lookup k8s on 116.228.111.118:53: no such host
[preflight] Some fatal errors occurred:
	[ERROR Swap]: running with swap on is not supported. Please disable swap
	[ERROR SystemVerification]: unsupported docker version: 18.09.0
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

巨大的 ERROR 错误，kubeadm 不支持 18.09 的 Docker 版本，只得降级。

1
yum downgrade docker-ce-18.06.1.ce

用 yum 直接这样降级会报一堆文件冲突错误：

1
2
3
4
Transaction check error:
  file /usr/bin/docker from install of docker-ce-18.06.1.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:18.09.0-3.el7.x86_64
  file /usr/share/bash-completion/completions/docker from install of docker-ce-18.06.1.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:18.09.0-3.el7.x86_64
  ...

好吧，只得卸载重新安装指定版本。

1
2
rpm -qa|grep docker # 查找并逐个卸载
rm -rf /var/lib/docker # 删除旧的镜像信息等

重新安装低版本：

1
yum update && yum install docker-ce-18.06.1.ce

建议还是按照 CRI installation 的步骤来进行安装。

提示要禁用 swap 分区，查找资料说是为了性能考虑，所以 k8s 默认是不允许使用交换分区的。可以添加kubelet 参数 --fail-swap-on=false 来启用 swap，这里同样进行关闭。

1
swapoff -a

将 /etc/fstab 中的 swap 挂载删除以永久禁用 swap。

还有一个报错是更改主机名忘记加 hosts 中，把 hostname 加入 hosts 文件中问题解决。

无法下载镜像

kubeadm 初始化时会从 https://k8s.gcr.io/v2/ 拉取镜像，国内无法访问。

先将 docker 的镜像改为 daocloud 的镜像，官网提供了一键配置镜像的脚本。

1
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io

执行之后重启 docker 失败，查看该脚本排查原因，做了一件事情，将 /etc/docker/daemon.json 备份，生成新的 daemon.json，打开新生成的内容：

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s docker]# cat daemon.json
{"registry-mirrors": ["http://f1361db2.m.daocloud.io"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {"registry-mirrors": ["http://f1361db2.m.daocloud.io"],
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}

很明显这里替换错误，将 log-opts 下面也进行了替换，删掉多余的配置之后启动成功，在此不得不吐槽下，脚本写好得测试呀，何况还是 daocloud 这种企业。

在 github 有这样的仓库 https://github.com/mritd/gcr 提供了官方镜像的同步，我们要从 hub 上面拉取改仓库的同步镜像 https://hub.docker.com/r/gcrxio/ ，将所需的镜像 pull 下来。

查看当前 k8s 版本需要哪些镜像：

1
2
3
4
5
6
7
8
[root@k8s ~]# kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.12.2
k8s.gcr.io/kube-controller-manager:v1.12.2
k8s.gcr.io/kube-scheduler:v1.12.2
k8s.gcr.io/kube-proxy:v1.12.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.24
k8s.gcr.io/coredns:1.2.2

拉取：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
docker pull gcrxio/kube-apiserver:v1.12.2
docker pull gcrxio/kube-controller-manager:v1.12.2
docker pull gcrxio/kube-scheduler:v1.12.2
docker pull gcrxio/kube-proxy:v1.12.2
docker pull gcrxio/pause:3.1
docker pull gcrxio/etcd:3.2.24
docker pull gcrxio/coredns:1.2.2

docker tag gcrxio/kube-apiserver:v1.12.2 k8s.gcr.io/kube-apiserver:v1.12.2
docker tag gcrxio/kube-controller-manager:v1.12.2 k8s.gcr.io/kube-controller-manager:v1.12.2
docker tag gcrxio/kube-scheduler:v1.12.2 k8s.gcr.io/kube-scheduler:v1.12.2
docker tag gcrxio/kube-proxy:v1.12.2 k8s.gcr.io/kube-proxy:v1.12.2
docker tag gcrxio/pause:3.1 k8s.gcr.io/pause:3.1
docker tag gcrxio/etcd:3.2.24 k8s.gcr.io/etcd:3.2.24
docker tag gcrxio/coredns:1.2.2 k8s.gcr.io/coredns:1.2.2

再次执行 kubeadm init 初始化成功。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.1.102:6443 --token xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxx

安装网络插件

网络插件有多种选择，看教程一众的 weave，在此使用 weave。

1
2
3
4
5
6
7
[root@k8s ~]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created

安装 weave scope:

1
2
3
4
5
6
7
8
[root@k8s ~]# kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')"
namespace/weave created
serviceaccount/weave-scope created
clusterrole.rbac.authorization.k8s.io/weave-scope created
clusterrolebinding.rbac.authorization.k8s.io/weave-scope created
deployment.apps/weave-scope-app created
service/weave-scope-app created
daemonset.extensions/weave-scope-agent created

执行命令验证是否安装成功：

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-576cbf47c7-vlfnl          1/1     Running   1          9m35s
kube-system   coredns-576cbf47c7-xhmnz          1/1     Running   1          9m35s
kube-system   etcd-k8s                          1/1     Running   0          8m40s
kube-system   kube-apiserver-k8s                1/1     Running   0          8m43s
kube-system   kube-controller-manager-k8s       1/1     Running   0          8m37s
kube-system   kube-proxy-8xwvn                  1/1     Running   0          9m35s
kube-system   kube-scheduler-k8s                1/1     Running   0          8m52s
kube-system   weave-net-vw7ww                   2/2     Running   0          8m6s
weave         weave-scope-agent-n6cq6           1/1     Running   0          3m50s
weave         weave-scope-app-dccf9fd7c-9tdls   1/1     Running   0          3m50s

看到 coredns 处于运行状态，证明网络环境 OK。

创建 Pod

默认情况下集群容器不会运行于 master 节点，要在 master 节点创建 Pod，需要将 master 节点的 taint 去掉，否则普通的 Pod 默认不会调度。

1
kubectl taint nodes --all node-role.kubernetes.io/master-

至此便完成了单个节点的 k8s 安装。

执行 kubeadm join 以加入其他节点，此时需要 kubeadm init 时的输出：

1
kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

如果忘记记录，则可以在 master 节点执行以下命令找回 token:

1
kubeadm token list

默认 token 的有效期为 24h，超过时间可以创建新的 token:

1
kubeadm token create

由于目前只有一台主机，后续测试多节点的加入移除。

参考

-EOF-

本博客所有文章除特别声明外，均采用 CC BY-NC-ND 4.0 许可协议。转载请注明出处！ © 雨落
Index Bio
  1. 1. kubeadm 安装
  2. 2. kubeadm init 踩坑
  3. 3. 无法下载镜像
  4. 4. 安装网络插件
  5. 5. 创建 Pod
  6. 6. 参考