@nalan90 2019-10-27T10:29:47.000000Z 字数 12456 阅读 963

etcd集群部署(二)

自动化运维

简介

etcd集群的组建有三种方式

static 方式
etcd discovery 方式
DNS discovery 方式

本文以static的方式进行演示,通常集群的节点为奇数(3、5、7),一般部署3个即可。

环境准备

服务器
- dev-161 (172.16.1.161)
- dev-162 (172.16.1.162)
- dev-163 (172.16.1.163)
- dev-164 (172.16.1.164) 后续添加的节点
- dev-165 (172.16.1.165) 后续添加的节点
操作系统
- CentOS 7.3
etcd版本
- v3.0以上
环境目录
- /opt/data/etcd/data (data-dir)
- /opt/data/etcd/wal (wal-dir)
- /opt/data/etcd/logs (log-dir)

## 安装etcd
work:~ ys$ for item in 161 162 163 164 165; do ssh root@dev-$item 'yum install -y etcd';done
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * epel: mirrors.tuna.tsinghua.edu.cn
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package etcd.x86_64 0:3.2.15-1.el7 will be installed
--> Finished Dependency Resolution
Installed:
  etcd.x86_64 0:3.2.15-1.el7
Complete!
......
## 创建etcd目录
work:~ ys$ for item in 161 162 163 164 165; do ssh root@dev-$item 'mkdir -p /opt/data/etcd/data && mkdir -p /opt/data/etcd/wal && mkdir -p /opt/data/etcd/logs && chown -R etcd:etcd /opt/data/etcd';done

node及etcd对应关系

Node Name	IP	HostName	Etcd Name
node1	172.16.1.161	dev-161	infra0
node2	172.16.1.162	dev-162	infra1
node3	172.16.1.163	dev-163	infra2
node4	172.16.1.164	dev-164	infra3
node5	172.16.1.165	dev-165	infra4

目标

以静态方式组建三个节点的etcd集群
- infra0 (dev-161)
- infra1 (dev-162)
- infra2 (dev-163)
添加两个节点
- infra3 (dev-164)
- infra4 (dev-165)
删除两个节点
- infra3 (dev-164)
- infra4 (dev-165)

修改配置文件(/etc/etcd/etcd.conf)

node1 (dev-161)

ETCD_DATA_DIR="/opt/data/etcd/data"
ETCD_WAL_DIR="/opt/data/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://172.16.1.161:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.1.161:2379,http://127.0.0.1:2379"
ETCD_NAME="infra0"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.1.161:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.1.161:2379"
ETCD_INITIAL_CLUSTER="infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node2 (dev-162)

ETCD_DATA_DIR="/opt/data/etcd/data"
ETCD_WAL_DIR="/opt/data/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://172.16.1.162:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.1.162:2379,http://127.0.0.1:2379"
ETCD_NAME="infra1"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.1.162:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.1.162:2379"
ETCD_INITIAL_CLUSTER="infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node3 (dev-163)

ETCD_DATA_DIR="/opt/data/etcd/data"
ETCD_WAL_DIR="/opt/data/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://172.16.1.163:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.1.163:2379,http://127.0.0.1:2379"
ETCD_NAME="infra2"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.1.163:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.1.163:2379"
ETCD_INITIAL_CLUSTER="infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

在三个节点上依次启动etcd服务

[root@dev-161 ~]# systemctl start etcd
[root@dev-161 ~]# systemctl enable etcd
[root@dev-162 ~]# systemctl start etcd
[root@dev-162 ~]# systemctl enable etcd
[root@dev-163 ~]# systemctl start etcd
[root@dev-163 ~]# systemctl enable etcd

查看集群状态

etcd API有v2、v3两个版本，v2提供更多的操作命令，因此我们通过v2进行集群的管理

[root@dev-161 ~]# etcdctl -v
etcdctl version: 3.2.15
API version: 2
## 设置ETCDCTL_API环境变量
[root@dev-162 ~]$ echo "export ETCDCTL_API=2" >> ~/.bash_profile
[root@dev-162 ~]$ source ~/.bash_profile
[root@dev-162 ~]$ echo $ETCDCTL_API
2
[root@dev-161 ~]# etcdctl member list
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=true
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=false
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[root@dev-161 ~]# etcdctl cluster-health
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy

也可以通过页面访问

image_1c8h5phaeno2smvrvle701joc9.png-74.8kB

image_1c8hal18pfc01kfhsn61go4qkm.png-19.7kB

到现在为止三个节点的etcd已经可以正常访问

添加新的节点

在dev-161(node1)执行添加infra3命令

[root@dev-161 ~]# etcdctl member add infra3 http://172.16.1.164:2379
Added member named infra3 with ID 12ceea0b7048b928 to cluster
## 执行完成生成如下环境变量
ETCD_NAME="infra3"
ETCD_INITIAL_CLUSTER="infra3=http://172.16.1.164:2379,infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
[root@dev-161 ~]# etcdctl member list
12ceea0b7048b928[unstarted]: peerURLs=http://172.16.1.164:2379
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=true
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=false
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[root@dev-161 ~]# etcdctl cluster-health
member 12ceea0b7048b928 is unreachable: no available published client urls
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy

在dev-164(node4)上启动新的etcd服务

## 修改配置如下
ETCD_DATA_DIR="/opt/data/etcd/data"
ETCD_WAL_DIR="/opt/data/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://172.16.1.164:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.1.164:2379,http://127.0.0.1:2379"
ETCD_NAME="infra3"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.1.164:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.1.164:2379"
## 上面生成的环境变量
ETCD_INITIAL_CLUSTER="infra3=http://172.16.1.164:2379,infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"
## 启动etcd
[root@dev-164 ~]# systemctl start etcd
[root@dev-164 ~]# systemctl enable etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
[root@dev-164 ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-03-21 09:53:01 CST; 52s ago
 Main PID: 3047 (etcd)
   CGroup: /system.slice/etcd.service
           └─3047 /usr/bin/etcd --name=infra3 --data-dir=/opt/data/etcd/data --listen-client-urls=http://172.16.1.164:2379,http://127.0.0.1:2379

在dev-161(node1)执行添加infra4命令

[root@dev-161 ~]# etcdctl member add infra4 http://172.16.1.165:2379
Added member named infra4 with ID 7ca9bd7d5389fec7 to cluster
ETCD_NAME="infra4"
ETCD_INITIAL_CLUSTER="infra3=http://172.16.1.164:2379,infra4=http://172.16.1.165:2379,infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
[root@dev-161 ~]# etcdctl member list
12ceea0b7048b928: name=infra3 peerURLs=http://172.16.1.164:2379 clientURLs=http://172.16.1.164:2379 isLeader=false
7ca9bd7d5389fec7[unstarted]: peerURLs=http://172.16.1.165:2379
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=true
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=false
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[root@dev-161 ~]# etcdctl cluster-health
member 12ceea0b7048b928 is healthy: got healthy result from http://172.16.1.164:2379
member 7ca9bd7d5389fec7 is unreachable: no available published client urls
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy

在dev-165(node5)上启动新的etcd服务

## 修改配置如下
ETCD_DATA_DIR="/opt/data/etcd/data"
ETCD_WAL_DIR="/opt/data/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://172.16.1.165:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.1.165:2379,http://127.0.0.1:2379"
ETCD_NAME="infra4"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.1.165:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.1.165:2379"
ETCD_INITIAL_CLUSTER="infra3=http://172.16.1.164:2379,infra4=http://172.16.1.165:2379,infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"
## 启动etcd
[root@dev-165 ~]# systemctl start etcd
[root@dev-165 ~]# systemctl enable etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
[root@dev-165 ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-03-21 10:01:17 CST; 7s ago
 Main PID: 3073 (etcd)
   CGroup: /system.slice/etcd.service
           └─3073 /usr/bin/etcd --name=infra4 --data-dir=/opt/data/etcd/data --listen-client-urls=http://172.16.1.165:2379,http://127.0.0.1:2379

在dev-161(node1)查看集群状态

[root@dev-161 ~]# etcdctl member list
12ceea0b7048b928: name=infra3 peerURLs=http://172.16.1.164:2379 clientURLs=http://172.16.1.164:2379 isLeader=false
7ca9bd7d5389fec7: name=infra4 peerURLs=http://172.16.1.165:2379 clientURLs=http://172.16.1.165:2379 isLeader=false
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=true
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=false
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[root@dev-161 ~]# etcdctl cluster-health
member 12ceea0b7048b928 is healthy: got healthy result from http://172.16.1.164:2379
member 7ca9bd7d5389fec7 is healthy: got healthy result from http://172.16.1.165:2379
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy

node4、node5两个节点正常添加至etcd集群，五个节点正常访问

删除节点

[test@dev-162 ~]$ etcdctl member --help
NAME:
   etcdctl member - member add, remove and list subcommands
USAGE:
   etcdctl member command [command options] [arguments...]
COMMANDS:
     list    enumerate existing cluster members
     add     add a new member to the etcd cluster
     remove  remove an existing member from the etcd cluster
     update  update an existing member in the etcd cluster
OPTIONS:
   --help, -h  show help
[root@dev-161 ~]# etcdctl member list
12ceea0b7048b928: name=infra3 peerURLs=http://172.16.1.164:2379 clientURLs=http://172.16.1.164:2379 isLeader=false
7ca9bd7d5389fec7: name=infra4 peerURLs=http://172.16.1.165:2379 clientURLs=http://172.16.1.165:2379 isLeader=false
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=true
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=false
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[root@dev-161 ~]# etcdctl cluster-health
member 12ceea0b7048b928 is healthy: got healthy result from http://172.16.1.164:2379
member 7ca9bd7d5389fec7 is healthy: got healthy result from http://172.16.1.165:2379
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy
## 删除infra3、infra4两个节点
[root@dev-161 ~]# etcdctl member remove 7ca9bd7d5389fec7
Removed member 7ca9bd7d5389fec7 from cluster
[root@dev-161 ~]# etcdctl member remove 12ceea0b7048b928
Removed member 12ceea0b7048b928 from cluster
[root@dev-161 ~]# etcdctl member list
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=true
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=false
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
## 查看infra3、infra4两个节点的状态
[root@dev-164 ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2018-03-22 09:20:36 CST; 16min ago
 Main PID: 3047 (code=exited, status=0/SUCCESS)
[root@dev-165 ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2018-03-22 09:20:28 CST; 14min ago
 Main PID: 3073 (code=exited, status=0/SUCCESS)

从集群中删除指定的etcd节点后，相应服务器上的etcd进程将会停止运行