etcd集群部署

自动化运维

简介

etcd集群的组建有三种方式

• static 方式
• etcd discovery 方式
• DNS discovery 方式

本文以static的方式进行演示,通常集群的节点为奇数(3、5、7),一般部署3个即可。

环境准备

• 服务器

  • dev-161 (172.16.1.161)
  • dev-162 (172.16.1.162)
  • dev-163 (172.16.1.163)
  • dev-164 (172.16.1.164) 后续添加的节点
  • dev-165 (172.16.1.165) 后续添加的节点

• 操作系统

  • CentOS 7.3
• etcd版本
  
  • v3.0以上
• 环境目录
  
  • /opt/data/etcd/data (data-dir)
  • /opt/data/etcd/wal (wal-dir)
  • /opt/data/etcd/logs (log-dir)

node及etcd对应关系

目标

• 以静态方式组建三个节点的etcd集群
  
  • infra0 (dev-161)
  • infra1 (dev-162)
  • infra2 (dev-163)
• 添加两个节点
  
  • infra3 (dev-164)
  • infra4 (dev-165)
• 删除两个节点
  
  • infra3 (dev-164)
  • infra4 (dev-165)

安装脚本

• node1 (dev-161)

etcd --name infra0 --initial-advertise-peer-urls http://172.16.1.161:2380 \
       --data-dir /opt/data/etcd/data \
       --wal-dir /opt/data/etcd/wal \
       --listen-peer-urls http://172.16.1.161:2380 \
       --listen-client-urls http://172.16.1.161:2379,http://127.0.0.1:2379 \
       --advertise-client-urls http://172.16.1.161:2379 \
       --initial-cluster-token etcd-cluster \
       --initial-cluster infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380 \
       --initial-cluster-state new >> /opt/data/etcd/logs/infra0.log 2>&1 &

• node2 (dev-162)

etcd --name infra1 --initial-advertise-peer-urls http://172.16.1.162:2380 \
       --data-dir /opt/data/etcd/data \
       --wal-dir /opt/data/etcd/wal \
       --listen-peer-urls http://172.16.1.162:2380 \
       --listen-client-urls http://172.16.1.162:2379,http://127.0.0.1:2379 \
       --advertise-client-urls http://172.16.1.162:2379 \
       --initial-cluster-token etcd-cluster \
       --initial-cluster infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380 \
       --initial-cluster-state new >> /opt/data/etcd/logs/etcd.log 2>&1 &

• node3 (dev-163)

etcd --name infra2 --initial-advertise-peer-urls http://172.16.1.163:2380 \
       --data-dir /opt/data/etcd/data \
       --wal-dir /opt/data/etcd/wal \
       --listen-peer-urls http://172.16.1.163:2380 \
       --listen-client-urls http://172.16.1.163:2379,http://127.0.0.1:2379 \
       --advertise-client-urls http://172.16.1.163:2379 \
       --initial-cluster-token etcd-cluster \
       --initial-cluster infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380 \
       --initial-cluster-state new >> /opt/data/etcd/logs/etcd.log 2>&1 &

• 查看集群状态

etcd API有v2、v3两个版本，v2提供更多的操作命令，因此我们通过v2进行集群的管理

[test@dev-162 ~]$ etcdctl -v
etcdctl version: 3.2.15
API version: 2
## 设置ETCDCTL_API环境变量
[test@dev-162 ~]$ echo "export ETCDCTL_API=2" >> ~/.bash_profile
[test@dev-162 ~]$ source ~/.bash_profile
[test@dev-162 ~]$ echo $ETCDCTL_API
2
[test@dev-162 ~]$ etcdctl member list
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=false
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=true
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[test@dev-162 ~]$ etcdctl cluster-health
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy

也可以通过页面访问

到现在为止三个节点的etcd已经可以正常访问

添加新的节点

• 在dev-162(node2)执行添加infra3命令

[test@dev-162 ~]$ etcdctl member add infra3 http://172.16.1.164:2379
Added member named infra3 with ID 206aaee1711610f7 to cluster
## 执行完成生成如下环境变量
ETCD_NAME="infra3"
ETCD_INITIAL_CLUSTER="infra3=http://172.16.1.164:2379,infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
[test@dev-162 ~]$ etcdctl member list
## 新添加的节点还处于未启动状态
5922e75caa279deb[unstarted]: peerURLs=http://172.16.1.164:2379  
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=false
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=true
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[test@dev-162 ~]$ etcdctl cluster-health
member 5922e75caa279deb is unreachable: no available published client urls
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy

• 在dev-164(node4)上启动新的etcd服务

## 导入以上生成的环境变量
[test@dev-164 ~]$ export ETCD_NAME="infra3"
[test@dev-164 ~]$ export ETCD_INITIAL_CLUSTER="infra3=http://172.16.1.164:2379,infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
[test@dev-164 ~]$ export ETCD_INITIAL_CLUSTER_STATE="existing"
[test@dev-164 ~]$ etcd --data-dir /opt/data/etcd/data \
--wal-dir /opt/data/etcd/wal \
--listen-client-urls http://172.16.1.164:2379,http://127.0.0.1:2379 \
--advertise-client-urls http://172.16.1.164:2379 >> /opt/data/etcd/logs/etcd.log 2>&1 &
[test@dev-164 ~]$ ps -ef | grep etcd
test  5243  4999  7 13:17 pts/0    00:00:00 etcd --data-dir /opt/data/etcd/data --wal-dir /opt/data/etcd/wal --listen-client-urls http://172.16.1.164:2379,http://127.0.0.1:2379 --advertise-client-urls http://172.16.1.164:2379

• 在dev-162(node2)执行添加infra4命令

[test@dev-162 ~]$ etcdctl member add infra4 http://172.16.1.165:2379
Added member named infra4 with ID 4598a311935f7513 to cluster
## 执行完成生成如下环境变量
ETCD_NAME="infra4"
ETCD_INITIAL_CLUSTER="infra4=http://172.16.1.165:2379,infra3=http://172.16.1.164:2379,infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
[test@dev-162 ~]$ etcdctl member list
## 新添加的节点还处于未启动状态
4598a311935f7513[unstarted]: peerURLs=http://172.16.1.165:2379
5922e75caa279deb: name=infra3 peerURLs=http://172.16.1.164:2379 clientURLs=http://172.16.1.164:2379 isLeader=false
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=false
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=true
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[test@dev-162 ~]$ etcdctl cluster-health
member 4598a311935f7513 is unreachable: no available published client urls
member 5922e75caa279deb is healthy: got healthy result from http://172.16.1.164:2379
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy

• 在dev-165(node5)上启动新的etcd服务

## 导入以上生成的环境变量
[test@dev-165 ~]$ export ETCD_NAME="infra4"
[test@dev-165 ~]$ export ETCD_INITIAL_CLUSTER="infra4=http://172.16.1.165:2379,infra3=http://172.16.1.164:2379,infra0=http://172.16.1.161:2380,infra1=http://172.16.1.162:2380,infra2=http://172.16.1.163:2380"
[test@dev-165 ~]$ export ETCD_INITIAL_CLUSTER_STATE="existing"
[test@dev-165 ~]$ etcd --data-dir /opt/data/etcd/data \
--wal-dir /opt/data/etcd/wal \
--listen-client-urls http://172.16.1.165:2379,http://127.0.0.1:2379 \
--advertise-client-urls http://172.16.1.165:2379 >> /opt/data/etcd/logs/etcd.log 2>&1 &
[test@dev-165 ~]$ ps -ef | grep etcd
test  2973  2810 14 13:58 pts/0    00:00:00 etcd --data-dir /opt/data/etcd/data --wal-dir /opt/data/etcd/wal --listen-client-urls http://172.16.1.165:2379,http://127.0.0.1:2379 --advertise-client-urls http://172.16.1.165:2379

• 在dev-162(node2)查看集群状态

[test@dev-162 ~]$ etcdctl member list
4598a311935f7513: name=infra4 peerURLs=http://172.16.1.165:2379 clientURLs=http://172.16.1.165:2379 isLeader=false
5922e75caa279deb: name=infra3 peerURLs=http://172.16.1.164:2379 clientURLs=http://172.16.1.164:2379 isLeader=false
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=false
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=true
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[test@dev-162 ~]$ etcdctl cluster-health
member 4598a311935f7513 is healthy: got healthy result from http://172.16.1.165:2379
member 5922e75caa279deb is healthy: got healthy result from http://172.16.1.164:2379
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy

node4、node5两个节点正常添加至etcd集群，五个节点正常访问

删除节点

[test@dev-162 ~]$ etcdctl member --help
NAME:
   etcdctl member - member add, remove and list subcommands
USAGE:
   etcdctl member command [command options] [arguments...]
COMMANDS:
     list    enumerate existing cluster members
     add     add a new member to the etcd cluster
     remove  remove an existing member from the etcd cluster
     update  update an existing member in the etcd cluster
OPTIONS:
   --help, -h  show help
## 当前有五个节点
[test@dev-162 ~]$ etcdctl member list
4598a311935f7513: name=infra4 peerURLs=http://172.16.1.165:2379 clientURLs=http://172.16.1.165:2379 isLeader=false
5922e75caa279deb: name=infra3 peerURLs=http://172.16.1.164:2379 clientURLs=http://172.16.1.164:2379 isLeader=false
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=false
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=true
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
## 删除infra4
[test@dev-162 ~]$ etcdctl member remove 4598a311935f7513
Removed member 4598a311935f7513 from cluster
## 再次查看仅剩四个节点
[test@dev-162 ~]$ etcdctl member list
5922e75caa279deb: name=infra3 peerURLs=http://172.16.1.164:2379 clientURLs=http://172.16.1.164:2379 isLeader=false
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=false
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=true
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[test@dev-162 ~]$ etcdctl cluster-health
member 5922e75caa279deb is healthy: got healthy result from http://172.16.1.164:2379
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy
## 删除infra3之前，etcd进程正在运行
[test@dev-164 ~]$ ps -ef | grep etcd
test  5243  4999  6 13:17 pts/0    00:03:02 etcd --data-dir /opt/data/etcd/data --wal-dir /opt/data/etcd/wal --listen-client-urls http://172.16.1.164:2379,http://127.0.0.1:2379 --advertise-client-urls http://172.16.1.164:2379
## 删除节点infra3
[test@dev-162 ~]$ etcdctl member remove 5922e75caa279deb
Removed member 5922e75caa279deb from cluster
[test@dev-162 ~]$ etcdctl member list
88e89ef57491e822: name=infra0 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=false
905994ad8f9fbbcd: name=infra1 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=true
cf5f03d58c1b363b: name=infra2 peerURLs=http://172.16.1.163:2380 clientURLs=http://172.16.1.163:2379 isLeader=false
[test@dev-162 ~]$ etcdctl cluster-health
member 88e89ef57491e822 is healthy: got healthy result from http://172.16.1.161:2379
member 905994ad8f9fbbcd is healthy: got healthy result from http://172.16.1.162:2379
member cf5f03d58c1b363b is healthy: got healthy result from http://172.16.1.163:2379
cluster is healthy
## 删除infra3之后，node4上的etcd进程停止运行
[test@dev-164 ~]$ ps -ef | grep etcd
test  5351  5327  0 14:06 pts/1    00:00:00 grep --color=auto etcd

从集群中删除指定的etcd节点后，相应服务器上的etcd进程将会停止运行