Seatunnel集群部署

大数据湖仓技术

一、seatunnel安装

1. 安装包上传

   tar -zxvf apache-seatunnel-2.3.6-bin.tar.gz apache-seatunnel-2.3.6
   tar -zxvf apache-seatunnel-web-1.0.2-bin.tar.gz apache-seatunnel-web-1.0.2

2. 下载链接器相关包

   cd apache-seatunnel-2.3.6/bin
   sh install-plugin.sh 2.3.6

3. 配置环境变量

   • vim /etc/profile

     export PATH=/data/mysql/bin:/data/mysql/lib:$PATH
         #set java environment
         export JAVA_HOME=/opt/java/jdk1.8.0_211
         export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
     export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
     # SEATUNNEL_HOME=${seatunnel install path}
         export SEATUNNEL_HOME=/opt/apache-seatunnel-2.3.6
         export PATH=$PATH:$SEATUNNEL_HOME/bin

   • vim /etc/bashrc
     source /etc/profile

4. 配置域名

   ### 开发集群
   #ip master-st-1 woker-st-1
   192.168.132.89 master-st-2 woker-st-2
   192.168.132.90 master-st-1 woker-st-1
   192.168.132.91 woker-st-3
   ......
   ### 测试集群
   192.168.132.64  st.master.node2 sr.fe.node2 ds.master.node2 ss.query.server 
   192.168.132.211 st.master.node1 sr.fe.node1 ds.worker.node1 
   192.168.132.212 st.worker.node1 sr.be.node1 ds.worker.node2 
   192.168.132.213 st.worker.node2 sr.be.node2 ds.worker.node3
   192.168.132.214 st.worker.node3 sr.be.node3 ds.master.node1
   bt\BTYY.com@123

   • 服务器免登录
   • st.master.node1
   • st.master.node2
   • st.worker.node1
   • st.worker.node2
     

     ssh-keygen -t rsa
     ssh-copy-id bt@192.168.132.211

5. 根据情况规划修改如下配置文件，复制配置文件到各节点

   • seatunnel.yaml
   • hazelcast-master.yaml
   • hazelcast-worker.yaml
   • jvm_master_options
   • jvm_worker_options

6. 启动集群

   • Master节点

     mkdir -p $SEATUNNEL_HOME/logs
     ./bin/seatunnel-cluster.sh -d -r master

   • Worker节点

     mkdir -p $SEATUNNEL_HOME/logs
     ./bin/seatunnel-cluster.sh -d -r worker

   • Client节点(提交任务)

     $SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template

二、seatunnel web安装

1. 初始化数据库并修改app相关配置

   sh $SEATUNNEL_WEB_HOME/script/init_sql.sh

2. 复制server配置文件到web

   cp $SEATUNNEL_HOME/config/hazelcast-client.yaml $SEATUNNEL_WEB_HOME/conf/
   cp $SEATUNNEL_HOME/connectors/plugin-mapping.properties  $SEATUNNEL_WEB_HOME/conf/

3. 启动服务

   cd $SEATUNNEL_WEB_HOME
   sh bin/seatunnel-backend-daemon.sh start