抱歉，您的浏览器无法访问本站
本页面需要浏览器支持（启用）JavaScript
了解详情 >

k8s-1.32集群部署指南

云原生

发布于：2025年2月13日

字数：2.3k 字

时长：12 分钟

配置规划

系统版本

Debian11

节点规划

3master高可用+1worker

1
2
3
4
192.168.2.10    k8s-master-192.168.2.10
192.168.2.11    k8s-master-192.168.2.11
192.168.2.12    k8s-master-192.168.2.12
192.168.2.21    k8s-worker-192.168.2.21
1
2
3
apiserver vip：192.168.2.100
podCIRD：172.168.0.0/16
svcCIRD：100.168.0.0/16

软件版本

1
2
3
kubelet=1.32.2-1.1
kubeadm=1.32.2-1.1
kubectl=1.32.2-1.1

基础环境初始化

权限不足自行加sudo

初始化系统

安装必要的工具

1
2
apt update
apt install -y gpg apt-transport-https ca-certificates curl gnupg2 software-properties-common gpg

禁用交换分区

在Kubernetes 1.22版本之前，默认要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。
从Kubernetes 1.22开始引入了NodeSwap的Alpha支持，改功能在Kubernetes 1.32进入了Beta。

修改hosts

1
2
3
4
5
# nano /etc/hosts
192.168.2.10    k8s-master-192.168.2.10
192.168.2.11    k8s-master-192.168.2.11
192.168.2.12    k8s-master-192.168.2.12
192.168.2.21    k8s-worker-192.168.2.21

修改hostname

以k8s-master-192.168.2.10节点为例

1
2
3
4
# nano /etc/hostname
k8s-master-192.168.2.10

hostname k8s-master-192.168.2.10

配置内核

配置内核模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack

内核参数配置，允许iptables管理二层流量

1
2
3
4
5
6
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system

添加GPG公钥

从阿里云源添加

1
2
curl -fsSL https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/deb/Release.key |
    gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

添加软件源

从阿里云源添加

1
2
3
4
# kubernetes
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/ /" |
    tee /etc/apt/sources.list.d/kubernetes.list
apt-get update

安装containerd

方式1：通过 nerdctl 打包安装（推荐）

nerdctl 完整安装包包含了 containerd/runc/buildkit/CNI plugins/RootlessKit/slirp4netns等组件，并且指令和docker一样，若从docker过来的使用体验更好。

Name Community API Target Web site
ctr Native (None, see ctr --help to learn the usage)
nerdctl containerd (non-core) Native General-purpose https://github.com/containerd/nerdctl
crictl Kubernetes SIG-node CRI For debugging only https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md 
1
2
3
wget https://github.com/containerd/nerdctl/releases/download/v2.0.3/nerdctl-full-2.0.3-linux-amd64.tar.gz

tar Cxzvf /usr/local containerd-2.0.3-linux-amd64.tar.gz

方式2：二进制安装

若想要极简或自定义安装，可以自行下载安装包进行安装。

使用root用户

1
wget https://github.com/containerd/containerd/releases/download/v2.0.3/containerd-2.0.3-linux-amd64.tar.gz

将其解压缩到/usr/local下:

1
tar Cxzvf /usr/local containerd-2.0.3-linux-amd64.tar.gz

方式3：apt安装

1
sudo apt-get install containerd.io

设置用systemd管理

在 /usr/local/lib/systemd/system/containerd.service 写入

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target dbus.service

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

执行：

1
2
3
systemctl daemon-reload
systemctl enable containerd --now 
systemctl status containerd

配置containerd

生成containerd的配置文件:

1
2
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

根据文档Container runtimes中的内容，设置 cgroup 驱动为 systemd

修改前面生成的配置文件/etc/containerd/config.toml：

1
2
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

再修改/etc/containerd/config.toml中的镜像源：

1
2
3
[plugins.'io.containerd.cri.v1.images'.pinned_images]
  # sandbox = "registry.k8s.io/pause:3.8"
  sandbox = "registry.aliyun.com/gcr/google_containers/pause:3.10"

安装runc

若通过 nerdctl 一键安装则跳过，否则要单独安装runc

接下来从runc的github上单独下载安装runc，该二进制文件是静态构建的，并且应该适用于任何Linux发行版。

1
2
wget https://github.com/opencontainers/runc/releases/download/v1.2.5/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc

安装Kubernetes

1
2
3
4
5
# 安装指定版本k8s
apt update
apt install -y kubelet=1.32.2-1.1 kubeadm=1.32.2-1.1 kubectl=1.32.2-1.1
# 锁定版本
apt-mark hold kubelet kubeadm kubectl

删除Kubernetes

1
apt --purge remove kubelet kubeadm kubectl kubernetes-cni

清除exited 容器

1
nerdctl  rm `nerdctl ps -a|grep Exited|awk ' {print $1}'`

[可选]apiserver高可用

使用 haproxy + keepalived 对apiserver 做高可用

仅在master节点配置，配置后可使用 vip:haproxy （192.168.2.100:8443）端口访问 apiserver。

两种方式：

1、通过静态pod管理：在 /etc/kubernetes/manifests 静态pod目录下，创建 haproxy.yaml 和 keepalived.yaml

2、通过systemd管理：apt install -y keepalived haproxy

本文介绍方式2，方式1虽然管理方便，但依赖 kubelet ，若kubelet异常将不可用。

配置 haproxy：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
cat << EOF > /etc/haproxy/haproxy.cfg
global
    log /dev/log  local0 warning
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
   
   stats socket /var/lib/haproxy/stats
   
defaults
  log global
  option  httplog
  option  dontlognull
        timeout connect 5000
        timeout client 50000
        timeout server 50000

frontend kube-apiserver
    bind *:8443
    mode tcp
    option tcplog
    default_backend kube-apiserver
   
backend kube-apiserver
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    stick-table type ip  size 1m  expire 30m
    stick on src
    server kube-apiserver-1 192.168.2.10:6443 check # Replace the IP address with your own.
    server kube-apiserver-2 192.168.2.11:6443 check
    server kube-apiserver-3 192.168.2.12:6443 check
 EOF

配置keekalived

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
cat << EOF > /etc/keepalived/keepalived.conf
global_defs {
  notification_email {
  }
  router_id LVS_DEVEL
  vrrp_skip_check_adv_addr
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}
   
vrrp_script chk_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}
   
vrrp_instance haproxy-vip {
  state BACKUP
  priority 100
  interface eth0                     # network adapter
  virtual_router_id 60
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass keEpAlived             # password
  }
  unicast_src_ip 192.160.2.10  # The IP address of this machine
  unicast_peer {
    192.160.2.10                     # The IP address of peer machines
    192.160.2.11
    192.160.2.12
  }
   
  virtual_ipaddress {
   192.160.2.100                     # The VIP address
  }
   
  track_script {
    chk_haproxy
  }
}
EOF

重启生效并设置开机自启

1
2
3
4
systemctl restart haproxy
systemctl restart keepalived
systemctl enable haproxy --now
systemctl enable keepalived --now

Kubernetes 配置

初始化集群

创建基础配置，以master为例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 0.0.0.0
  bindPort: 6443
nodeRegistration:
  nodeNmae: k8s-master-192.168.2.10
  criSocket: unix:///run/containerd/containerd.sock
  taints:
  - effect: PreferNoSchedule
    key: node-role.kubernetes.io/master
  ignorePreflightErrors:
  - DirAvailable--etc-kubernetes-manifests
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: 1.32.2
imageRepository: registry.aliyun.com/gcr/google_containers
controlPlaneEndpoint: "192.168.2.200:8443"
networking:
  podSubnet:  172.168.0.0/16
  serviceSubnet: 100.168.0.0/16
  dnsDomain: "cluster.local"
apiServer:
  extraArgs:
    - name:  service-node-port-range
      vaule: 30000-32767
controllerManager:
  extraArgs:
    bind-address: "0.0.0.0"
scheduler:
  extraArgs:
    bind-address: "0.0.0.0"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
containerRuntimeEndpoint:  unix:///run/containerd/containerd.sock
failSwapOn: false
---
apiVersion: kubeproxy.config.k8s.io/v1beta1
kind: KubeProxyConfiguration
mode: "ipvs"

查看涉及的镜像

1
2
3
4
5
6
7
8
9
kubeadm config images list --config kubeadm.yaml
 
registry.aliyun.com/google_containers/kube-apiserver:v1.32.2
registry.aliyun.com/gcr/google_containers/kube-controller-manager:v1.32.2
registry.aliyun.com/gcr/google_containers/kube-scheduler:v1.32.2
registry.aliyun.com/gcr/google_containers/kube-proxy:v1.32.2
registry.aliyun.com/gcr/google_containers/pause:3.10
registry.aliyun.com/gcr/google_containers/etcd:3.5.16-0
registry.aliyun.com/gcr/google_containers/coredns:v1.11.3

拉取镜像

1
kubeadm config images pull --config kubeadm.yaml

初始化集群

1
kubeadm init --config kubeadm.yaml

初始化完成

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.2.100:8443 --token 4raim49.vjcptmrka1jb2bpi \
    --discovery-token-ca-cert-hash sha256:bc6c70r231e8fa19bd9a0a8b17371afbfcb1e4ed4e5cae259688bdb3f3e12411 \
    --control-plane --certificate-key 96yu876i632ac8885881a964ddd0fb47eb3144a9b0a568250d3941f4019a6653

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.2.100:8443 --token 5fim49.vjcptmrka1jb2bpi \
    --discovery-token-ca-cert-hash sha256:bc6c70r231e8fa19bd9a0a8b17371afbfcb1e4ed4e5cae259688bdb3f3e12411

保存配置

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

加入master

执行上面环境初始化，包括apiserver高可用安装keepalived和haproxy步骤

1
2
3
kubeadm join 192.168.2.100:8443 --token 4ram42.vjcptmree1jb2bpf \
    --discovery-token-ca-cert-hash sha256:bc6c70r231e8fa19bd9a0a8b17371afbfcb1e4ed4e5cae259688bdb3f3e12411 \
    --control-plane --certificate-key 96yu876i632ac8885881a964ddd0fb47eb3144a9b0a568250d3941f4019a6653

加入worker

执行上面环境初始化

1
2
sudo kubeadm join 192.168.2.100:8443 --token 4ram42.vjcptmree1jb2bpf \
    --discovery-token-ca-cert-hash sha256:bc6c70e463e8fa19bd9a0a8b17371afbfcb1e4ed4e5cae259688bdb3f3e12411

检查集群状态

注意： 在未安装网络插件的情况下，所有节点均为NotReady状态

1
kubectl get node

安装网络插件

可选的网络插件有很多，具体见

这里简单选择calico。

1
2
3
4
5
# 安装tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/tigera-operator.yaml

# 安装calico crd
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/custom-resources.yaml

测试验证

部署一个pods

通过 deployment 命令来部署基于 Nginx 的应用程序，来验证 Kubernetes 集群的安装是否正确

1
2
3
kubectl create deployment nginx-app --image=nginx
kubectl expose deployment nginx-app --name=nginx-web-svc --type NodePort --port 80 --target-port 80
kubectl describe svc nginx-web-svc

安装其他依赖

1、Dashboard：https://github.com/kubernetes/dashboard

1
2
3
helm install kubernetes-dashboard/kubernetes-dashboard \
    --create-namespace --namespace kubernetes-dashboard \
    --set 'api.containers.args={--disable-csrf-protection=true}' --set kong.proxy.http.enabled=true

2、监控组件

3、集群代理：https://doc.traefik.io/traefik/user-guides/crd-acme/

本博客内容遵循 署名-非商业性使用-相同方式共享 4.0 国际 (CC BY-NC-SA 4.0) 协议

本文章著作权归作者所有，任何形式的转载都请注明出处

更新于：2024年2月26日

kubernetes

评论