[关闭]
@zwei 2018-07-30T10:26:32.000000Z 字数 8817 阅读 1274

kubenernet 集群制动化安装介绍

kubernetes

服务简介

1，Appengine（DataCenter）

管理和存储线上云平台中的所有k8s 集群的服务

2，App-agent（Demon）

安装和检查，更新 k8s 基础服务和k8s 内部服务

3，Rpmserver（nginx）

存储 k8s 集群的插件yaml 文件 和 app-agent 的插件demon

服务介绍图

1，Appengine（DataCenter）

功能列表：

安装编译手册:

You have a working [Go environment].
  1. $ git clone http://newgit.op.ksyun.com/openstack/appengine.git
  2. $ cd $GOPATH/src/appengine
  3. $ make all
You have a working [Docker environment].
  1. $ git clone http://newgit.op.ksyun.com/openstack/appengine.git
  2. $ cd appengine
  3. $ make all
  4. $ docker build -t hub.kce.ksyun.com/ksyun/appengine:v1.0.1 .

服务配置文件 (/etc/appengine/appengine.toml)：

  1. title = "Appengine configurations"
  3. [server]
  4. port = ":80" #web服务开启端口
  6. [database]
  7. type = "mysql"
  8. connection = "appengine:UPLGAu4p6r9NVKzH@tcp(172.31.21.7:3306)/appengine?charset=utf8&parseTime=True&loc=Local" # mysql db 连接地址
  10. [ca]
  11. cadir = "/opt/ca" # k8s 集群的config file 保存地址
  13. [neutron]
  14. proxy = true # 是否开启neutron 代理
  15. remotehost = "tjwqone.neutron.sdns.ksyun.com" # 代理目标地址
  16. remoteport = "9696" # 代理目标端口
  17. localport = "9696" # 代理本地端口

2，App-agent（Demon）

插件服务列表：

安装编译手册:

You have a working [Go environment And linux].
  1. $ mkdir -p /opt/src/github.com/ksyun
  2. $ cd /opt/src/github.com/ksyun
  3. $ git clone http://newgit.op.ksyun.com/openstack/app-agent-go.git
  4. $ mv app-agent-go app-agent
  5. $ cd app-agent
  6. $ export GOPATH=/opt/
  7. $ go build  --ldflags '-extldflags "-static"' -o  /opt/src/github.com/ksyun/app-agent/cmd/appctl/appctl /opt/src/github.com/ksyun/app-agent/cmd/appctl/appctl.go
  8. $ go build  --ldflags '-extldflags "-static"' -o  /opt/src/github.com/ksyun/app-agent/cmd/app-plugin/demon/demon /opt/src/github.com/ksyun/app-agent/cmd/app-plugin/demon/demon.go
  9. $ go build  --ldflags '-extldflags "-static"' -o /opt/src/github.com/ksyun/app-agent/cmd/app-plugin/plugins/plugins /opt/src/github.com/ksyun/app-agent/cmd/app-plugin/plugins/plugins.go
在linux环境中编译并且发布到线上.
  1. $ mkdir -p /opt/src/github.com/ksyun
  2. $ cd /opt/src/github.com/ksyun
  3. $ git clone http://newgit.op.ksyun.com/openstack/app-agent-go.git
  4. $ mv app-agent-go app-agent
  5. $ export GOPATH=/opt/
  6. $ cd app-agent
  7. $ ./build.sh

服务配置选项：

  1. $ appctl --help 
  2. The appctl is appengine service client and update node info to server.
  4. Usage:
  5.   appctl [flags]
  7. Flags:
  8.       --address string                   The IP address for the app-agent to serve on (set to 0.0.0.0 for all interfaces) (default "127.0.0.1")
  9.       --alsologtostderr                  log to standard error as well as files
  10.       --appctlconfig string              app-agent config file.  (default "/etc/appctl/appctl.conf")
  11.       --arrangement string               openstack node arrangement file default /opt/app-agent/arrangement/clusterinfo (default "/opt/app-agent/arrangement/clusterinfo")
  12.       --enable-server                    Enable the app-agent's server 
  13.       --endpoint string                  appengine server url. (default set http://appengine.sdns.ksyun.com:80).  (default "http://appengine.sdns.ksyun.com:80")
  14.       --healthz-bind-address string      The IP address for the healthz server to serve on. (set to 0.0.0.0 for all interfaces) (default "127.0.0.1")
  15.       --healthz-port int32               The port of the localhost healthz endpoint (set to 0 to disable) (default 1321)
  16.   -h, --help                             help for appctl
  17.       --hostname-override string         If non-empty, will use this string as identification instead of the actual hostname.
  18.       --iface iface                      interface to use (IP or name) for inter-host communication. Can be specified multiple times to check each option in order. Returns the first match found. (default [])
  19.       --iface-regex iface                regex expression to match the first interface to use (IP or name) for inter-host communication. Can be specified multiple times to check each regex in order. Returns the first match found. Regexes are checked after specific interfaces specified by the iface option have already been checked. (default [])
  20.       --log-backtrace-at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
  21.       --log-dir string                   If non-empty, write log files in this directory
  22.       --log-flush-frequency duration     Maximum number of seconds between log flushes (default 5s)
  23.       --logtostderr                      log to standard error instead of files (default true)
  24.       --node-ip string                   IP address of the node. If set, app-agent will use this IP address for the node
  25.       --plugin-config-file string        The app plugin bin config file (set /etc/app-agent/plugin/config) (default "/etc/appctl/plugin/config")
  26.       --plugin-dir string                The app plugin bin dir. (set /opt/app-agent/plugin/bin) (default "/opt/app-agent/plugin/bin")
  27.       --port int32                       The port for the app-agent to serve on. (default 1322)
  28.       --rootdir string                   app-agent server root dir.  (default "/opt/app-agent")
  29.       --rpm-endpoint string              The IP address rpm server (set default http://rpm.kce.sdns.ksyun.com:80) (default "http://rpm.kce.sdns.ksyun.com:80")
  30.       --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  31.       --templatedir string               app-agent cluster ca file default /opt/app-agent/tempalte. (default "/opt/app-agent/template")
  32.       --tls-cert-file string             File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir.
  33.       --tls-private-key-file string      File containing x509 private key matching --tls-cert-file.
  34.   -v, --v Level                          log level for V logs
  35.       --version version[=true]           Print version information and quit
  36.       --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
  38. $ 主要选选项 appctl --endpoint http://192.168.122.1:80   --rpm-endpoint http://192.168.122.2:80
  39. $  --endpoint 为 appengine 服务地址
  40. $ --rpm-endpoint  为rpm nginx服务地址

3，Rpmserver（nginx）

功能列表：

安装手册:

You have a working [Docker environment].
  1. $ docker  run -d --name nginx  --restart=always --net=host -v /data/app-agent-go/template:/usr/share/nginx/html:ro -v /data/nginx/nginx.conf:/etc/nginx/nginx.conf:ro hub.kce.ksyun.com/ksyun/nginx

服务目录架构：

  1. binary/                                          # 存放demon，plugins 插件地址
  2. centos/                                          # 存放centos 7.3 的docker， kubelet rpm包 yum 源
  3. rpm/                                             # 存放docker， kubelet 服务的配置文件和启动文件
  4. test/                                            # 测试环境相关的文件和配置文件，插件
  5. yaml/                                            # 存放k8s 服务中yaml地址
  6. version                                          # rpmservice 服务版本信息
  8. ├── binary
  9. │   ├── appctl
  10. │   ├── demon
  11. │   └── plugins
  12. ├── centos
  13. │   └── 7
  14. │       ├── RPM-GPG-KEY-CENTOS-7
  15. │       └── x86_64
  16. ├── rpm
  17. │   ├── 7
  18. │   │   ├── docker
  19. │   │   ├── k8s.repo
  20. │   │   └── kubelet
  21. │   ├── appctl-v1.8.3-ksyun.x86_64.rpm
  22. │   ├── docker-ce-17.09.0-ksyun.x86_64.rpm
  23. │   ├── etcdctl-3.2.5-01.ksyun.x86_64.rpm
  24. │   └── kubelet-v1.8.3-ksyun.x86_64.rpm
  25. ├── test
  26. │   ├── appengine.toml
  27. │   ├── app.sh
  28. │   ├── demon
  29. │   ├── nginx.conf
  30. │   ├── nginx.yaml
  31. │   └── service.yaml
  32. ├── version
  33. └── yaml
  34.     ├── dns
  35.     │   └── kube-dns.yaml
  36.     ├── heapster
  37.     │   └── heapster.yaml
  38.     ├── ingress
  39.     │   └── traefik.yaml
  40.     ├── kube
  41.     │   ├── canal-flanneld.yaml
  42.     │   ├── etcd.yaml
  43.     │   ├── kube-apiserver.yaml
  44.     │   ├── kube-controller-manager.yaml
  45.     │   ├── kube-proxy.yaml
  46.     │   └── kube-scheduler.yaml
  47.     ├── system-monitor
  48.     │   └── system-monitor.yaml
  49.     └── volume
  50.         └── flexvolume.yaml

修改yaml文件方法：

1, 修改app-agent 项目中template.
  1. $ git clone http://newgit.op.ksyun.com/openstack/app-agent-go.git
  2. $ cd app-agent-go/template/yaml
2, 登入nginx 服务器中修改
  1. $ ssh root@10.69.59.238
  2. $ cd /data/app-agent-go/template/yaml

控制流程图

安装控制流程图

rd使用手册

添加k8s 集群服务或资源

  1. curl -X POST \
  2.   http://10.97.2.74/d9b78b541c2c4878af4041b1a90b9ee3/plugins \
  3.   -H 'cache-control: no-cache' \
  4.   -H 'content-type: application/json' \
  5.   -H 'postman-token: 694756ca-855a-8d70-c7d9-86d466f91eff' \
  6.   -H 'x-auth-token: d9b78b541c2c4878af4041b1a90b9ee2:d9b78b541c2c4878af4041b1a90b9ee3' \
  7.   -d '{"name": "demon", "version": "v1.0.1", "location": "http://1.1.1.1/demon"}'

更新k8s 集群服务或资源

调用appengine api 接口实现 组件更新
operate 可用参数介绍

  • update: 设置为true， 表示滚动更新服务的image （kubectl set image， 适合于k8s集群非断服务不需要修改yaml文件）
  • reset： 设置为true， 表示删除服务后，在执行创建 （kubectl delete -f xx.yaml && kubectl create -f xx.yaml）
  • delete: 设置为true， 表示创建为yaml 后， 在把yaml 文件删除掉（适合用于image 中只是copy文件和测试，如flexvolume）
  1. # 更新k8s 集群服务或资源
  2. curl -X PUT \
  3.   http://10.97.2.74/d9b78b541c2c4878af4041b1a90b9ee3/plugins/c10e70ae-4aef-420b-81c7-52b181a6ddda \
  4.   -H 'cache-control: no-cache' \
  5.   -H 'content-type: application/json' \
  6.   -H 'postman-token: 4851a99a-4669-db49-4c89-838f1af7680b' \
  7.   -H 'x-auth-token: d9b78b541c2c4878af4041b1a90b9ee2:d9b78b541c2c4878af4041b1a90b9ee3' \
  8.   -d '{"version": "v1.0.2", "location": "http://1.1.1.1/demon", "operate": "{\"update\":true}"}'
  1. # 重置 k8s 集群中的服务或资源
  2. curl -X PUT \
  3.   http://10.97.2.74/d9b78b541c2c4878af4041b1a90b9ee3/plugins/c10e70ae-4aef-420b-81c7-52b181a6ddda \
  4.   -H 'cache-control: no-cache' \
  5.   -H 'content-type: application/json' \
  6.   -H 'postman-token: 4851a99a-4669-db49-4c89-838f1af7680b' \
  7.   -H 'x-auth-token: d9b78b541c2c4878af4041b1a90b9ee2:d9b78b541c2c4878af4041b1a90b9ee3' \
  8.   -d '{"version": "v1.0.2", "location": "http://1.1.1.1/demon", "operate": "{\"reset\":true}"}'
  1. # 创建完并删除 k8s 集群中的服务或资源
  2. curl -X PUT \
  3.   http://10.97.2.74/d9b78b541c2c4878af4041b1a90b9ee3/plugins/c10e70ae-4aef-420b-81c7-52b181a6ddda \
  4.   -H 'cache-control: no-cache' \
  5.   -H 'content-type: application/json' \
  6.   -H 'postman-token: 4851a99a-4669-db49-4c89-838f1af7680b' \
  7.   -H 'x-auth-token: d9b78b541c2c4878af4041b1a90b9ee2:d9b78b541c2c4878af4041b1a90b9ee3' \
  8.   -d '{"version": "v1.0.2", "location": "http://1.1.1.1/demon", "operate": "{\"delete\":true}"}'

删除 k8s 集群中的服务或资源

去掉现有集群中的内部服务

  1. curl -X DELETE \
  2.   http://10.97.2.74/d9b78b541c2c4878af4041b1a90b9ee3/plugins/48f98098-31ac-485e-a1ef-1323962522e9 \
  3.   -H 'cache-control: no-cache' \
  4.   -H 'content-type: application/json' \
  5.   -H 'postman-token: 27e541fc-c41d-434d-7d86-60c1be141a7d' \
  6.   -H 'x-auth-token: d9b78b541c2c4878af4041b1a90b9ee2:d9b78b541c2c4878af4041b1a90b9ee3'
添加新批注
在作者公开此批注前，只有你和作者可见。
回复批注