Dolphinscheduler集群部署

大数据湖仓技术

前置依赖

• JDK：下载JDK (1.8+)，安装并配置 JAVA_HOME 环境变量，并将其下的 bin 目录追加到 PATH 环境变量中。如果你的环境中已存在，可以跳过这步。
• 二进制包：在下载页面下载 DolphinScheduler 二进制包
• 数据库：PostgreSQL (8.2.15+) 或者 MySQL (5.7+)，两者任选其一即可，如 MySQL 则需要 JDBC Driver 8.0.16
• 注册中心：ZooKeeper (3.4.6+)，下载地址
• 进程树分析
  
  • macOS安装pstree
  • Fedora/Red/Hat/CentOS/Ubuntu/Debian安装psmisc
    
    • yum install -y psmisc

前置准备

1. 集群机器规划
   vim /etc/hosts

   192.168.132.64  st.master.node2 sr.fe.node2 ds.master.node2 zk.service.node1
   192.168.132.211 st.master.node1 sr.fe.node1 ds.worker.node1 ss.query.server zk.service.node2 ds.api.server flink.master.node1
   192.168.132.212 st.worker.node1 sr.be.node1 ds.worker.node2
   flink.worker.node1
   192.168.132.213 st.worker.node2 sr.be.node2 ds.worker.node3 pgsql.db.node flink.worker.node2 flink.master.node2
   192.168.132.214 st.worker.node3 sr.be.node3 ds.master.node1
   zk.service.node3 ds.alert.server flink.worker.node3 flink.master.node3

   1）. 安装PostgreSQL

   sudo yum update
   sudo yum install postgresql-server postgresql-contrib
   sudo postgresql-setup initdb
   sudo systemctl start postgresql
   #sudo systemctl restart postgresql
   sudo systemctl enable postgresql
   #验证
   sudo -i -u postgres
   psql
   SELECT rolname,rolpassword FROM pg_authid;
   ALTER USER postgres WITH PASSWORD 'BT@024';
   CREATE DATABASE dolphinscheduler;
   # 修改 {user} 和 {password} 为你希望的用户名和密码
   #CREATE USER {user} PASSWORD {password};
   ALTER DATABASE dolphinscheduler OWNER TO postgres;
   #远程访问
   find / -name postgresql.conf
   vim postgresql.conf
   listen_addresses = '*'
   find / -name pg_hba.conf
   vim pg_hba.conf
   host    all             all             0.0.0.0/0            md5
   sudo systemctl restart postgresql

   2). 安装zookeeper
   a. 下载安装包：https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz
   b. 解压并安装配置

   tar -zxvf apache-zookeeper-3.8.4-bin.tar.gz
   mv apache-zookeeper-3.8.4-bin /opt/zookeeper-3.8.4
   cd zookeeper-3.8.4/conf/
   mv zoo_sample.cfg zoo.cfg
   vim zoo.cfg
   server.1=zk.service.node1:2888:3888
   server.2=zk.service.node2:2888:3888
   server.3=zk.service.node3:2888:3888

   c. 创建zk数据目录并myid文件，内容与zoo.cfg中server.x项中的x对应。

   sudo mkdir -p /data/zookeeper
   touch /data/zookeeper/myid
   sudo chmod -R 775 /data/zookeeper
   echo "1" >> /data/zookeeper/myid
   cd /opt/zookeeper-3.8.4/bin
   sh ./zkServer.sh start

   3) Flink安装（可选, 需要配置SSH免登录）
   a. Fink: https://nightlies.apache.org/flink/flink-docs-release-1.20/zh/docs/deployment/resource-providers/standalone/overview/
   b. Flink-CDC: https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/zh/docs/deployment/standalone/
   c. Yarn集群：集群资源调度
   https://www.cnblogs.com/caoweixiong/p/13451038.html
   https://nightlies.apache.org/flink/flink-docs-release-1.20/zh/docs/deployment/resource-providers/yarn/

   NFS配置参考

   • 服务端：

     sudo mkdir -p /data/flink/ha
     sudo yum install nfs-utils rpcbind
     sudo vi /etc/exports
     /data/flink/ha *(rw,sync,no_root_squash)
     /opt/flink *(rw,sync,no_root_squash)
     sudo systemctl start rpcbind
     sudo systemctl start nfs-server
     sudo systemctl enable rpcbind
     sudo systemctl enable nfs-server

   • 客户端：

     sudo yum install nfs-utils
     sudo mkdir -p /data/flink/ha 
     sudo mount 192.168.132.211:/data/flink/ha /data/flink/ha
     sudo mount 192.168.132.211:/data/flink/c
     onf /data/flink/conf
     sudo mount 192.168.132.211:/data/flink/lib /data/flink/lib
     #取消挂载
     sudo umount -lf /opt/flink

2. 下载安装包

   • https://www.apache.org/dyn/closer.lua/dolphinscheduler/3.1.9/apache-dolphinscheduler-3.1.9-bin.tar.gz

3. 上传包到服务器各节点解压
   tar -zxvf apache-dolphinscheduler-3.1.9-bin.tar.gz

4. 配置用户免密及权限

   useradd dsuser
   echo "dsuser" | passwd --stdin dsuser
   sed -i '$adsuser  ALL=(ALL)  NOPASSWD: ALL' /etc/sudoers
   sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
   chown -R dsuser:dsuser dolphinscheduler-3.1.9
   #配置机器SSH免密登陆
   su dsuser
   ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
   chmod 600 ~/.ssh/authorized_keys
   ssh-copy-id dsuser@ip
   #在数据库节点初始化数据库
   bash tools/bin/upgrade-schema.sh

   • *因为任务执行服务是以 sudo -u {linux-user} 切换不同 linux 用户的方式来实现多租户运行作业，所以部署用户需要有 sudo 权限，而且是免密的。初学习者不理解的话，完全可以暂时忽略这一点
   • 如果发现 /etc/sudoers 文件中有 "Defaults requirett" 这行，也请注释掉*

5. 启动集群

    # 一键停止集群所有服务
   bash ./bin/stop-all.sh
   # 一键开启集群所有服务
   bash ./bin/start-all.sh
   # 启停 Master
   bash ./bin/dolphinscheduler-daemon.sh stop master-server
   bash ./bin/dolphinscheduler-daemon.sh start master-server
   # 启停 Worker
   bash ./bin/dolphinscheduler-daemon.sh start worker-server
   bash ./bin/dolphinscheduler-daemon.sh stop worker-server
   # 启停 Api
   bash ./bin/dolphinscheduler-daemon.sh start api-server
   bash ./bin/dolphinscheduler-daemon.sh stop api-server
   #启停 Alert
   bash ./bin/dolphinscheduler-daemon.sh start alert-server
   bash ./bin/dolphinscheduler-daemon.sh stop alert-server

   浏览器访问地址 http://localhost:12345/dolphinscheduler/ui 即可登录系统 UI。默认的用户名和密码是 admin/dolphinscheduler123