高可用组件安装（可选）

采用 KeepAlived 和 HAProxy 实现高可用，所以需要安装 KeepAlived 和 HAProxy。

KeepAlived 和HAProxy 的节点可以和 Master 在同一个节点，也可以在不同的节点。

如果是在公有云搭建高可用集群，可以采用公有云的负载均衡替代 KeepAlived 和 HAProxy。

如果想要搭建只有一个 Master 节点的集群，可以不安装高可用组件。

所有 Master 节点安装 HAProxy 和 KeepAlived：

yum install -y keepalived haproxy

所有 Master 节点配置 HAProxy

所有 Master 节点的 HAProxy 配置相同。

vim /etc/haproxy/haproxy.cfg 

haproxy.cfg

global
    maxconn 2000 
    ulimit-n 16384
    log 127.0.0.1 local0 err 
    stats timeout 30s
defaults 
    log global 
    mode http 
    option httplog 
    timeout connect 5000 
    timeout client 50000 
    timeout server 50000 
    timeout http-request 15s 
    timeout http-keep-alive 15s 

frontend monitor-in
    bind *:33305 
    mode http 
    option httplog 
    monitor-uri /monitor 

frontend k8s-master
    bind 0.0.0.0:16443 # 监听的端口
    bind 127.0.0.1:16443
    mode tcp 
    option tcplog
    tcp-request inspect-delay 5s 
    default_backend k8s-master 

backend k8s-master
    mode tcp 
    option tcplog 
    option tcp-check 
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server k8s-master01 192.168.10.121:6443 check # 配置后端服务器地址
    server k8s-master02 192.168.10.124:6443 check 
    server k8s-master03 192.168.10.125:6443 check 

scp /etc/haproxy/haproxy.cfg root@k8s-master02:/etc/haproxy/haproxy.cfg
scp /etc/haproxy/haproxy.cfg root@k8s-master03:/etc/haproxy/haproxy.cfg

所有 Master 节点配置 KeepAlived

由于 KeepAlived 需要配置自身的 IP 地址和网卡名称，因此每个 KeepAlived 节点的配置不一样。

vim /etc/keepalived/keepalived.conf 

Master01节点的配置

! Configuration File for keepalived 
global_defs {
    router_id LVS_DEVEL 
    script_user root
    enable_script_security 
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh" 
    interval 5
    weight -5 
    fall 2 
    rise 1 
}
vrrp_instance VI_1 { 
    state MASTER                       # 
    interface ens33                    # 本机网卡名称
    mcast_src_ip 192.168.10.121        # 本机IP地址 
    virtual_router_id 51
    priority 101                       # 
    advert_int 2 
    authentication { 
        auth_type PASS
        auth_pass K8SHA_KA_AUTH 
    }
    virtual_ipaddress {
        192.168.10.129                 # VIP地址，需要是宿主机同网段且不存在的IP地址
    }
    track_script { 
        chk_apiserver 
    }
}

Master02节点的配置

! Configuration File for keepalived 
global_defs {
    router_id LVS_DEVEL 
    script_user root
    enable_script_security 
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh" 
    interval 5
    weight -5 
    fall 2 
    rise 1 
}
vrrp_instance VI_1 { 
    state BACKUP                       # 后备机
    interface ens33                    # 本机网卡名称
    mcast_src_ip 192.168.10.124        # 本机IP地址 
    virtual_router_id 51
    priority 100                       # 降低一点优先级
    advert_int 2 
    authentication { 
        auth_type PASS
        auth_pass K8SHA_KA_AUTH 
    }
    virtual_ipaddress {
        192.168.10.129                 # VIP地址，需要是宿主机同网段且不存在的IP地址
    }
    track_script { 
        chk_apiserver 
    }
}

Master03节点的配置

! Configuration File for keepalived 
global_defs {
    router_id LVS_DEVEL 
    script_user root
    enable_script_security 
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh" 
    interval 5
    weight -5 
    fall 2 
    rise 1 
}
vrrp_instance VI_1 { 
    state BACKUP                   # 后备机
    interface ens33                # 本机网卡名称
    mcast_src_ip 192.168.10.125    # 本机IP地址 
    virtual_router_id 51
    priority 100                   # 降低一点优先级
    advert_int 2 
    authentication { 
        auth_type PASS
        auth_pass K8SHA_KA_AUTH 
    }
    virtual_ipaddress {
        192.168.10.129             # 虚拟IP(VIP)，需要是宿主机同网段且不存在的IP地址
    }
    track_script { 
        chk_apiserver 
    }
}

所有 Master 节点配置 KeepAlived 健康检查文件

vim /etc/keepalived/check_apiserver.sh 

check_apiserver.sh

#!/bin/bash
err = 0
for k in $(seq 1 3) 
do
    check_code=$(pgrep haproxy) 
    if [[ $check_code == ""]]; then 
        err = $(expr $err + 1)
        sleep 1 
        continue 
    else
        err = 0 
        break 
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived" 
    /usr/bin/systemctl stop keepalived 
    exit 1
else 
    exit 0 
fi

chmod +x /etc/keepalived/check_apiserver.sh 
scp /etc/keepalived/check_apiserver.sh root@k8s-master02:/etc/keepalived/check_apiserver.sh
scp /etc/keepalived/check_apiserver.sh root@k8s-master03:/etc/keepalived/check_apiserver.sh

启动 HAProxy 和 KeepAlived

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

如果是用 HAProxy 和 KeepAlived 实现的高可用，则需要测试 VIP 是否是正常的：

# 所有节点进行ping测试
# ping 192.168.10.129 -c 4
PING 192.168.10.129 (192.168.10.129) 56(84) bytes of data.
64 bytes from 192.168.10.129: icmp_seq=1 ttl=64 time=0.888 ms
64 bytes from 192.168.10.129: icmp_seq=2 ttl=64 time=0.567 ms
64 bytes from 192.168.10.129: icmp_seq=3 ttl=64 time=15.1 ms
64 bytes from 192.168.10.129: icmp_seq=4 ttl=64 time=0.403 ms

--- 192.168.10.129 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3043ms
rtt min/avg/max/mdev = 0.403/4.260/15.183/6.308 ms

# 所有节点进行telnet测试
# telnet 192.168.10.129 16443
Trying 192.168.10.129...
Connected to 192.168.10.129.
Escape character is '^]'.
^CConnection closed by foreign host.

如果 ping 不通且 telnet 没有出现，则认为 VIP 不可用。如果 VIP 不可用，不可再继续往下执行，需要排查 VIP 的问题，比如防火墙和 SELinux、HAProxy 和 Keepalived 的状态，监听端口是否正常等。