ultrajiのblog

搭建一套kubernetes集群环境

 
  • 2021年 07月21日

在本地通过vmware虚拟机搭建一套供学习使用的kubernetes集群。

  • 实验环境
    • 用vmware创建的两台2u4g的centos8虚机（可以先创建一台虚机，中途克隆）

系统配置

  1. 修改主机名（PS：我这里用master做主机名主要用于区分k8s中master和node）；

     vim /etc/hostname
 # 输入主机名
 master

  2. 在hosts中将要加入集群的主机进行域名映射

     vim /etc/hosts
 # 增加以下内容：
 192.168.199.222 master
 192.168.199.223 node1

  3. 永久关闭swap（主要出于性能考虑？）

     swapoff -a
 vim /etc/fstab
 # 注释掉SWAP分区项，即可
 #   /dev/mapper/cl_master-root /                       xfs     defaults        0 0
 #   UUID=28bf76b1-e134-4f8a-b44b-02395a3b888b /boot                   ext4    defaults        1 2
 #   # /dev/mapper/cl_master-swap swap                    swap    defaults        0 0  # 这行

 #刷新swap使之生效
 sysctl -p

  4. 关闭selinux （出错了不容易排查？）

     setenforce 0

  5. 永久关闭防火墙，或者不关闭防火墙但需要打开一些端口（二选一）

     systemctl disable firewalld
 systemctl stop firewalld

    或

     firewall-cmd --zone=public --add-port=80/tcp --permanent
 firewall-cmd --zone=public --add-port=6443/tcp --permanent
 firewall-cmd --zone=public --add-port=2379-2380/tcp --permanent
 firewall-cmd --zone=public --add-port=10250-10255/tcp --permanent
 firewall-cmd --zone=public --add-port=30000-32767/tcp --permanent
 firewall-cmd --reload
 firewall-cmd --zone=public --list-ports

  6. 配置流量转发

     cat <<EOF >  /etc/sysctl.d/k8s.conf
 net.bridge.bridge-nf-call-ip6tables = 1
 net.bridge.bridge-nf-call-iptables = 1
 vm.swappiness = 0
 EOF

 modprobe br_netfilter
 sysctl -p /etc/sysctl.d/k8s.conf
 sysctl --system
 systemctl daemon-reload
 systemctl restart kubelet

安装软件

  1. 更新docker和k8s的软件源信息（PS：这里我选择了阿里源）

     yum -y install yum-utils

 # docker-ce 源
 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

 # kubernetes 源
 cat <<EOF > /etc/yum.repos.d/kubernetes.repo
 [kubernetes]
 name=Kubernetes
 baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
 enabled=1
 gpgcheck=0
 repo_gpgcheck=0
 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
     http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
 EOF

 # 更新yum包的索引
 yum clean all && yum makecache fast
 yum -y update

  2. 安装docker，以及配置镜像加速

     # 卸载原有docker
 systemctl stop docker && systemctl disable docker
 yum remove docker-ce docker-ce-cli

 # 安装k8s支持的docker版本，我这里选择了`18.09.0`
 yum list docker-ce --showduplicates|sort -r
 yum install -y docker-ce-18.09.0-3.el7 docker-ce-cli-18.09.0-3.el7
 systemctl start docker && systemctl enable docker

 # 配置docker
 touch /etc/docker/daemon.json
 vim /etc/docker/daemon.json
 {
     "registry-mirrors": ["https://xxxxx.mirror.aliyuncs.com"] # 可以选用各大云厂商的容器镜像加速服务
 }
 systemctl daemon-reload
 systemctl restart docker

  3. 安装kubeadm，安装kubeadm的命令同时会安装下面的3个二进制

     # kubeadm k8s的配置工具
 # kubelet k8s的核心服务
 # kubectl kubelet的client
 # kubernetes-cni 容器网络接口标准协议
 yum install -y kubeadm
 systemctl daemon-reload
 systemctl start kubelet && systemctl enable kubelet

  4. 自此，可以克隆多份虚拟机做工作节点了，记得改对应的hostname和静态IP。

搭建master节点

  1. 生成默认的配置文件

     kubeadm config print init-defaults > kubeadm.conf
 # 修改localAPIEndpoint
 # vim kubeadm.conf
 localAPIEndpoint:
     advertiseAddress: 192.168.199.222 # 改成Master IP

  2. 由于国内网络环境的缘故，事先把kubeadm init需要的镜像拉下来

     kubeadm config images list --config kubeadm.conf
 # 根据输出修改脚本
 vim pullkubeimages.sh

    内容如下：

     #!/bin/bash

 # 国内地址
 # registry.cn-hangzhou.aliyuncs.com/google_containers
 # registry.cn-beijing.aliyuncs.com/escience/escience-beijing

 KUBE_VERSION=v1.19.0        # 根据上条命令的输出修改版本
 KUBE_PAUSE_VERSION=3.2
 ETCD_VERSION=3.4.13-0
 CORE_DNS_VERSION=1.7.0

 GCR_URL=k8s.gcr.io
 ALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/google_containers

 images=(
         kube-apiserver:${KUBE_VERSION}
         kube-controller-manager:${KUBE_VERSION}
         kube-scheduler:${KUBE_VERSION}
         kube-proxy:${KUBE_VERSION}
         pause:${KUBE_PAUSE_VERSION}
         etcd:${ETCD_VERSION}
         coredns:${CORE_DNS_VERSION}
 )

 for imageName in ${images[@]} ; do
 docker pull $ALIYUN_URL/$imageName
 docker tag  $ALIYUN_URL/$imageName $GCR_URL/$imageName
 docker rmi  $ALIYUN_URL/$imageName
 done

  3. 初始化master节点

     kubeadm init --config ./kubeadm.conf

    记录输出

     W1118 22:44:02.998868   10181 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
 [init] Using Kubernetes version: v1.19.0
 [preflight] Running pre-flight checks
         [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
         [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
         [WARNING FileExisting-tc]: tc not found in system path
 [preflight] Pulling images required for setting up a Kubernetes cluster
 [preflight] This might take a minute or two, depending on the speed of your internet connection
 [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
 [certs] Using certificateDir folder "/etc/kubernetes/pki"
 [certs] Generating "ca" certificate and key
 [certs] Generating "apiserver" certificate and key
 [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.199.222]
 [certs] Generating "apiserver-kubelet-client" certificate and key
 [certs] Generating "front-proxy-ca" certificate and key
 [certs] Generating "front-proxy-client" certificate and key
 [certs] Generating "etcd/ca" certificate and key
 [certs] Generating "etcd/server" certificate and key
 [certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.199.222 127.0.0.1 ::1]
 [certs] Generating "etcd/peer" certificate and key
 [certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.199.222 127.0.0.1 ::1]
 [certs] Generating "etcd/healthcheck-client" certificate and key
 [certs] Generating "apiserver-etcd-client" certificate and key
 [certs] Generating "sa" key and public key
 [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
 [kubeconfig] Writing "admin.conf" kubeconfig file
 [kubeconfig] Writing "kubelet.conf" kubeconfig file
 [kubeconfig] Writing "controller-manager.conf" kubeconfig file
 [kubeconfig] Writing "scheduler.conf" kubeconfig file
 [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
 [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
 [kubelet-start] Starting the kubelet
 [control-plane] Using manifest folder "/etc/kubernetes/manifests"
 [control-plane] Creating static Pod manifest for "kube-apiserver"
 [control-plane] Creating static Pod manifest for "kube-controller-manager"
 [control-plane] Creating static Pod manifest for "kube-scheduler"
 [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
 [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
 [apiclient] All control plane components are healthy after 18.506231 seconds
 [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
 [kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
 [upload-certs] Skipping phase. Please see --upload-certs
 [mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
 [mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
 [bootstrap-token] Using token: abcdef.0123456789abcdef
 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
 [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
 [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
 [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
 [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
 [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
 [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
 [addons] Applied essential addon: CoreDNS
 [addons] Applied essential addon: kube-proxy

 Your Kubernetes control-plane has initialized successfully!

 To start using your cluster, you need to run the following as a regular user:

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

 You should now deploy a pod network to the cluster.
 Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
 https://kubernetes.io/docs/concepts/cluster-administration/addons/

 Then you can join any number of worker nodes by running the following on each as root:

 kubeadm join 192.168.199.222:6443 --token abcdef.0123456789abcdef \
     --discovery-token-ca-cert-hash sha256:105f7cec3de32bd6b5061d5205b1b439d5a96f9db205e4db001c3d0136f57181

    在工作节点执行上面这条命令就可以加入集群。

  4. 执行集群的基本配置

     mkdir -p $HOME/.kube
 cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 chown $(id -u):$(id -g) $HOME/.kube/config

  5. 验证集群安装情况

kubectl get nodes
# NAME     STATUS   ROLES    AGE   VERSION
# master   Ready    master   19m   v1.19.4

kubectl get cs
# Warning: v1 ComponentStatus is deprecated in v1.19+
# NAME                 STATUS      MESSAGE ERROR
# controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused
# scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused
# etcd-0               Healthy     {"health":"true"}

配置工作节点

  1. 利用上面配置master输出结果中的内容，将node节点加入集群

     kubeadm join 192.168.199.222:6443 --token abcdef.0123456789abcdef \
         --discovery-token-ca-cert-hash sha256:105f7cec3de32bd6b5061d5205b1b439d5a96f9db205e4db001c3d0136f57181

排错

上面kubectl get cs中controller-manager、scheduler的状态均为Unhealthy

出现这种情况，是/etc/kubernetes/manifests/下的kube-controller-manager.yaml和kube-scheduler.yaml设置的默认端口是0导致的，解决方式是注释掉对应的port即可（- --port=0改为#- --port=0），操作如下：

cd /etc/kubernetes/manifests
vim kube-controller-manager.yaml
vim kube-scheduler.yaml
systemctl restart kubelet
kubectl get cs
搜索