spark streaming集成flume、kafka

spark

集成flume

maven工程中添加依赖

    <!-- flume -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-flume_2.10</artifactId>
      <version>${spark.version}</version>
      <scope>compile</scope>
    </dependency>

可以在IDEA中运行

在命令行中运行需添加如下包到spark lib目录
spark源码包下 external/flume/target/spark-streaming-flume_2.10-1.6.1.jar
flume目录lib下 flume-avro-source-1.5.0-cdh5.3.6.jar 和 flume-ng-sdk-1.5.0-cdh5.3.6.jar

并添加到spark目录下自定义关联jar包目录 inputJars 就可以在shell命令行运行

[beifeng@hadoop-senior spark-1.6.1-bin-2.5.0-cdh5.3.6]$ bin/spark-shell \
> --master local[3] \
> --jars /opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/mysql-connector-java-5.1.27-bin.jar ,\
> /opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/flume-avro-source-1.5.0-cdh5.3.6.jar ,\
> /opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/flume-ng-sdk-1.5.0-cdh5.3.6.jar ,\
> /opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/spark-streaming-flume_2.10-1.6.1.jar 

集成kafka

    <!-- kafka -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka_2.10</artifactId>
      <version>${spark.version}</version>
      <scope>compile</scope>
    </dependency>

在命令行中运行需添加如下包
到spark目录下自定义关联jar包目录 inputJars 就可以在shell命令行运行

[beifeng@hadoop-senior spark-1.6.1-bin-2.5.0-cdh5.3.6]$ bin/spark-shell \
--master local[3] \
--jars /opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/mysql-connector-java-5.1.27-bin.jar ,\
/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/kafka_2.10-0.8.2.1.jar ,\
/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/kafka-clients-0.8.2.1.jar ,\
/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/metrics-core-2.2.0.jar ,\
/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/spark-streaming-kafka_2.10-1.6.1.jar ,\
/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/zkclient-0.3.jar

或是

export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/mysql-connector-java-5.1.27-bin.jar 
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/kafka_2.10-0.8.2.1.jar
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/kafka-clients-0.8.2.1.jar
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/metrics-core-2.2.0.jar
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/spark-streaming-kafka_2.10-1.6.1.jar
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/modules/spark-1.6.1-bin-2.5.0-cdh5.3.6/inputJars/zkclient-0.3.jar