@yanglt7 2018-10-21T15:58:02.000000Z 字数 11305 阅读 810

【Web 集群实战】20_Keepalived 高可用集群

Web集群实战

【Web 集群实战】20_Keepalived 高可用集群

1. Keepalived 高可用故障切换原理

Keepalived 高可用服务队之间的故障切换转移，是通过 VRRP（Virtual Router Redundancy Protocol，虚拟路由器冗余协议）来实现的。
VRRP 通过竞选机制来实现虚拟路由器的功能，所有的协议报文都是通过 IP 多播（Muiticast)包（默认的多播地址 224.0.0.18）形式发送的。虚拟路由器由 VRID（范围 0-255）和一组 IP 地址组成，对外表现为一个周知的 MAC 地址：00-00-5E-00-01-{VRID}。所以，在一个虚拟路由器中，不管谁是 Master，对外都是相同的 MAC 和 IP（称之为 VIP）。客户端主机并不需要因 Master 的改变而修改自己的路由配置。对它们来说，这种切换是透明的。

2. Keepalived 高可用服务搭建准备

硬件准备

HOSTNAME	IP	说明
lb001	192.168.2.129	Keepalived 主服务器（Nginx 主负载均衡器）
lb002	192.168.2.130	Keepalived 辅服务器（Nginx 辅负载均衡器）
web001	192.168.2.146	web001 服务器
web002	192.168.2.131	web002 服务器

- 开始安装 keepalived 软件

[root@lb001 ~]# yum install keepalived -y
[root@lb002 ~]# yum install keepalived -y

3. 配置 Keepalived 实现单实例单 IP 自动漂移接管

3.1 配置 Keepalived 主服务器 lb001 MASTER

关闭防火墙

[root@lb001 ~]# systemctl stop firewalld

配置 lb001 MASTER 的 keepalived.conf 配置文件

[root@lb001 ~]# cd /etc/keepalived/
[root@lb001 keepalived]# vim keepalived.conf
[root@lb001 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     1622320046@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb001 # <-- 局域网内应唯一
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 55 # <-- 虚拟路由 ID 标识，在一个 keepalived.conf 中是唯一的。MASTER 和 BACKUP 配置中相同实例又必须是一致的。
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.188/24 dev ens33 label ens33:1
    }
}

启动 keepalived 服务

[root@lb001 keepalived]# systemctl start keepalived.service
[root@lb001 keepalived]# ps -ef|grep keepalived
root       1258      1  0 16:30 ?        00:00:00 /usr/sbin/keepalived -D
root       1259   1258  0 16:30 ?        00:00:00 /usr/sbin/keepalived -D
root       1260   1258  0 16:30 ?        00:00:00 /usr/sbin/keepalived -D
root       1298   1165  0 16:33 pts/0    00:00:00 grep --color=auto keepalived
[root@lb001 keepalived]# ip addr|grep 192.168.2.188
    inet 192.168.2.188/24 scope global secondary ens33:1

3.1 配置 Keepalived 主服务器 lb0012 BACKUP

关闭防火墙

[root@lb002 ~]# systemctl stop firewalld

配置 lb002 BACKUP 的 keepalived.conf 配置文件

[root@lb002 ~]# cd /etc/keepalived/
[root@lb002 keepalived]# vim keepalived.conf
[root@lb002 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     1622320046@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb002 # <-- 局域网内应唯一
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 55 # <-- 虚拟路由 ID 标识，在一个 keepalived.conf 中是唯一的。MASTER 和 BACKUP 配置中相同实例又必须是一致的。
    priority 100 # <-- 优先级需低于主节点至少50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.188/24 dev ens33 label ens33:1
    }
}

启动 keepalived 服务

[root@lb002 keepalived]# systemctl start keepalived.service
[root@lb002 keepalived]# ps -ef|grep keepalived
root       1298      1  0 16:43 ?        00:00:00 /usr/sbin/keepalived -D
root       1299   1298  0 16:43 ?        00:00:00 /usr/sbin/keepalived -D
root       1300   1298  0 16:43 ?        00:00:00 /usr/sbin/keepalived -D
root       1306   1192  0 16:43 pts/0    00:00:00 grep --color=auto keepalived
[root@lb002 keepalived]# ip addr|grep 192.168.2.188
[root@lb002 keepalived]# 
# 此时应无返回

3.3 高可用主备服务器切换实验

[root@lb001 ~]# systemctl stop keepalived.service
[root@lb002 keepalived]# ip addr|grep 192.168.2.188
    inet 192.168.2.188/24 scope global ens33:1
[root@lb001 ~]# systemctl start keepalived.service
[root@lb001 ~]# ip addr|grep 192.168.2.188
    inet 192.168.2.188/24 scope global eth33:1
[root@lb002 keepalived]# ip addr|grep 192.168.2.188
[root@lb002 keepalived]#

4. Keepalived 双实例双主模式配置

Keepalived 双实例双主模式的 IP 及 VIP 规划表

HOSTNAME	IP	说明
lb001	192.168.2.129	VIP：192.168.2.188（用于绑定 A 服务 www.yangyangyang.org 域名）
lb002	192.168.2.130	VIP：192.168.2.189（用于绑定 B 服务 bbs.yangyangyang.org 域名）

4.1 配置服务器 lb001

配置 lb001 的 keepalived.conf

[root@lb001 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     1622320046@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb001
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 55
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.188/24 dev ens33 label ens33:1
    }
}
vrrp_instance VI_2 {
    state BACKUP
    interface ens33
    virtual_router_id 56
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.189/24 dev ens33 label ens33:2
    }
}

4.2 配置服务器 lb002

配置 lb002 的 keepalived.conf

[root@lb002 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     1622320046@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb002
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.188/24 dev ens33 label ens33:1
    }
}
vrrp_instance VI_2 {
    state MASTER
    interface ens33
    virtual_router_id 56
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.189/24 dev ens33 label ens33:2
    }
}

4.3 高可用切换测试

在 lb001 和 lb002 分别启动 Keepalived 服务

[root@lb001 keepalived]# systemctl restart keepalived.service
[root@lb001 keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
    inet 192.168.2.188/24 scope global ens33:1
[root@lb002 keepalived]# systemctl restart keepalived.service
[root@lb002 keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
    inet 192.168.2.189/24 scope global ens33:2

停掉 lb002 Keepalived 服务：

[root@lb002 keepalived]# systemctl stop keepalived.service
[root@lb001 keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
    inet 192.168.2.188/24 scope global ens33:1
    inet 192.168.2.189/24 scope global secondary ens33:2
[root@lb002 keepalived]# systemctl start keepalived.service
[root@lb002 keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
    inet 192.168.2.189/24 scope global ens33:2
[root@lb001 keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
    inet 192.168.2.188/24 scope global ens33:1

停掉 lb001 Keepalived 服务：

[root@lb001 keepalived]# systemctl stop keepalived.service    
[root@lb002 keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
    inet 192.168.2.189/24 scope global ens33:2
    inet 192.168.2.188/24 scope global secondary ens33:1
[root@lb001 keepalived]# systemctl start keepalived.service
[root@lb001 keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
    inet 192.168.2.188/24 scope global ens33:1
[root@lb002 keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
    inet 192.168.2.189/24 scope global ens33:2

5. Nginx 负载均衡配合 Keepalived 服务

5.1 在 lb001 和 lb002 上配置 Nginx 负载均衡

[root@lb001 keepalived]# cat /application/nginx/conf/nginx.conf
worker_processes  1;
error_log logs/error.log;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    upstream www_server_pools {
        server 192.168.2.131:80 weight=1;
        server 192.168.2.146:80 weight=1;
    }
    server {
        listen 192.168.2.188:80;
        server_name www.yangyangyang.org;
        location / {
                proxy_pass http://www_server_pools;
                include proxy.conf;
        }
    }
}

[root@lb002 keepalived]# cat /application/nginx/conf/nginx.conf
worker_processes  1;
error_log logs/error.log;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    upstream www_server_pools {
        server 192.168.2.131:80 weight=1;
        server 192.168.2.146:80 weight=1;
    }
    server {
        listen 192.168.2.188:80;
        server_name www.yangyangyang.org;
        location / {
                proxy_pass http://www_server_pools;
                include proxy.conf;
        }
    }
}

5.2 在 lb001 和 lb002 上配置 Keepalived 服务

[root@lb001 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     1622320046@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb001
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 55
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.188/24 dev ens33 label ens33:1
    }
}

[root@lb002 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     1622320046@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb002
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.188/24 dev ens33 label ens33:1
    }
}

5.3 解决服务监听的网卡上不存在 IP 地址问题

[root@lb001 keepalived]# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf
[root@lb001 keepalived]# tail -1 /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
[root@lb001 keepalived]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1

[root@lb002 keepalived]# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf
[root@lb002 keepalived]# tail -1 /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
[root@lb002 keepalived]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1

p.s. 上述部分均由桥接模式完成，由于用户模拟访问时桥接模式无法访问外网，于是改成NAT模式的DHCP方式上网，以下测试均为NAT模式。篇首的IP地址为NAT模式的地址。

5.4 用户访问准备及模拟实际访问

（1）在客户端 hosts 文件里把 www.yangyangyang.org 域名解析到 VIP 192.168.2.188 上，正式场景需要通过 DNS 解析。

192.168.2.188 www.yangyangyang.org

（2）两台 web 服务器开启 Nginx 服务，并配置首页文件

[root@web001 ~]# /application/nginx/sbin/nginx 
[root@web001 ~]# cat /application/nginx/html/www/index.html
192.168.2.146 www.yangyangyang.org
[root@web002 ~]# /application/nginx/sbin/nginx 
[root@web002 ~]# cat /application/nginx/html/www/index.html
192.168.2.131 www.yangyangyang.org

（3）两台负载均衡服务器配好 Nginx 服务，并确保后面代理的 Web 节点可以测试访问

[root@lb001 keepalived]# /application/nginx/sbin/nginx
[root@lb001 keepalived]# lsof -i:80
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nginx   1151  root    6u  IPv4  20341      0t0  TCP www.yangyangyang.org:http (LISTEN)
nginx   1215 nginx    6u  IPv4  20341      0t0  TCP www.yangyangyang.org:http (LISTEN)
[root@lb001 keepalived]# ip addr|grep 192.168.2.188
    inet 192.168.2.188/24 scope global secondary ens33:1

（4）模拟访问

在客户端浏览器输入 www.yangyangyang.org 测试访问，刷新几次

192.168.2.131
192.168.2.146

此时停止 lb001 服务器或者停掉 Keepalived 服务，观察业务是否正常

[root@lb001 keepalived]# systemctl stop keepalived.service

观察 lb002 备节点是否接管 VIP 192.168.2.188

[root@lb002 keepalived]# ip addr|grep 192.168.2.188
    inet 192.168.2.188/24 scope global secondary ens33:1

再次在客户端浏览器输入 www.yangyangyang.org 测试访问，刷新几次，出现和切换 lb002 前相同的访问结果
开启 lb001 的 Keepalived 服务，VIP 又接管回来了。

6. 配置指定文件接收 Keepalived 服务日志

（1）编辑配置文件 /etc/sysconfig/keepalived ，将 14 行的 KEEPALIVED_OPTIONS="-D" 修改为 KEEPALIVED_OPTIONS="-D -d -S 0"

[root@lb001 ~]# sed -i '14 s#KEEPALIVED_OPTIONS="-D"#KEEPALIVED_OPTIONS="-D -d -S 0"#g' /etc/sysconfig/keepalived
[root@lb001 ~]# sed -n '14p' /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -d -S 0"

（2）修改 rsyslog 的配置文件 vim /etc/rsyslog.conf ，在结尾处加上如下两行内容

[root@lb001 ~]# tail -2 /etc/rsyslog.conf
#keepalived
local0.*    /var/log/keepalived.log

上述配置表示来自 local0 设备的所有日志信息都记录到 /var/log/keepalived.log 文件。

（3）修改 rsyslog 的配置文件 vim /etc/rsyslog.conf，将 54 行改成

*.info;mail.none;authpriv.none;cron.none;local0.none                /var/log/messages

（4）配置完成后，重启 rsyslog 服务

[root@lb001 ~]# systemctl restart rsyslog.service

（5）关闭 keepalived 服务，测试 Keepalived 日志记录结果。

[root@lb001 ~]# tail /var/log/keepalived.log
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:33 localhost Keepalived[1250]: Stopping
Oct 10 20:07:33 localhost Keepalived_vrrp[1252]: VRRP_Instance(VI_1) sent 0 priority
Oct 10 20:07:33 localhost Keepalived_vrrp[1252]: VRRP_Instance(VI_1) removing protocol VIPs.
Oct 10 20:07:33 localhost Keepalived_healthcheckers[1251]: Stopped