spark安装部署

spark

想学习spark，hadoop，kafka等大数据框架，请加群459898801，满了之后请加2群224209501。后续文章会陆续公开

1，spark环境的安装

创建四个目录

sudo mkdir /opt/modules
sudo mkdir /opt/softwares
sudo mkdir /opt/tools
sudo mkdir /opt/datas
sudo chmod 777 -R /opt/

1，安装jdk1.7

先卸载自带的jdk

rpm –qa | grep java
sudo rpm -e --nodeps (自带java包)

安装jdk1.7

export JAVA_HOME=/opt/modules/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin

2，spark编译

安装mvn

export MAVEN_HOME=/usr/local/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin

3，安装scala

export SCALA_HOME=/opt/modules/scala-2.10.4
export PATH=$PATH:$SCALA_HOME/bin

4，修改mvn镜像源

编译之前先配置镜像及域名服务器，来提高下载速度，进而提高编译速度，用nodepad++打开/opt/compileHadoop/apache-maven-3.0.5/conf/setting.xml。（nodepad已经通过sftp链接到了机器）

  <mirror>
  <id>nexus-spring</id>
  <mirrorOf>cdh.repo</mirrorOf>
  <name>spring</name>
  <url>http://repo.spring.io/repo/</url>
  </mirror>
  <mirror>
  <id>nexus-spring2</id>
  <mirrorOf>cdh.releases.repo</mirrorOf>
  <name>spring2</name>
  <url>http://repo.spring.io/repo/</url>
  </mirror>

5，配置域名解析服务器

sudo vi /etc/resolv.conf
添加内容：
    nameserver 8.8.8.8
    nameserver 8.8.4.4

6，编译spark

为了提高编译速度，修改如下内容

VERSION=1.3.0
SPARK_HADOOP_VERSION=2.6.0-cdh5.4.0
SPARK_HIVE=1
#VERSION=$("$MVN" help:evaluate -Dexpression=project.version 2>/dev/null | grep -v "INFO" | tail -n 1)
#SPARK_HADOOP_VERSION=$("$MVN" help:evaluate -Dexpression=hadoop.version $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | tail -n 1)
#SPARK_HIVE=$("$MVN" help:evaluate -Dexpression=project.activeProfiles -pl sql/hive $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | fgrep --count "<id>hive</id>";\
#    # Reset exit status to 0, otherwise the script stops here if the last grep finds nothing\
#    # because we use "set -o pipefail"
#    echo -n)

执行编译指令：

./make-distribution.sh --tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive-0.13.1 -Phive-thriftserver
去掉下面编译会很快，即使编译失败也不会每次都清除
-DskipTests clean package

4 安装hadoop2.6

1，添加java主目录位置

hadoop-env.sh
mapred-env.sh
yarn-env.sh
添加如下：
export JAVA_HOME=/opt/modules/jdk1.7.0_67

2，core-site.xml配置

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/modules/hadoop-2.5.0/data/tmp</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://spark.learn.com:8020</value>
    </property>

3，hdfs-site.xml配置

    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>

4，mapred-site.xml配置

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

5，yarn-site.xml配置

    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>miaodonghua.host</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

6，slaves配置

miaodonghua.host//主机名：nodemanager和datanode地址

7，格式化namenode

bin/hdfs namenode -format

3，spark几种模式的安装部署

1 spark本地模式的安装

本地模式基础语法测试
1，直接运行

bin/spark-shell

2，spark的webAPP

http://spark.learn.com:4040

3，读取readme.cm

val textFile = sc.textFile("README.md")

4，count

testFile.count()

5，first

testFile.first()

6，filter

val linesWithSpark = testFile.filter(line => line.contains("Spark"))

testFile.filter(line => line.contains("Spark")).count()

相对复杂计算 complex computations

testFile.map(line => line.split(" ").size).reduce((a, b) => if (a > b) a else b)

import java.lang.Math
testFile.map(line => line.split(" ").size).reduce((a, b) => Math.max(a, b))

val wordCounts = testFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)

 wordCounts.collect()

cache

scala> linesWithSpark.cache()

scala> linesWithSpark.count()

2 spark standalone模式的安装

1，配置spark-env.sh

HADOOP_CONF_DIR=/opt/modules/hadoop-2.6.0-cdh5.4.0/etc/hadoop
JAVA_HOME=/opt/modules/jdk1.7.0_67
SCALA_HOME=/opt/modules/scala-2.10.4
SPARK_MASTER_IP=spark.learn.com
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=1
SPARK_WORKER_MEMORY=1000m
SPARK_WORKER_PORT=7078
SPARK_WORKER_WEBUI_PORT=8081
SPARK_WORKER_INSTANCES=1

2，配置spark-defaults.conf

spark.master spark://hadoop-spark.com:7077

3，配置slaves

spark.learn.com

4，启动spark

sbin/start-master.sh
sbin/start-slaves.sh

5，命令测试
读取本地的需要注释掉

HADOOP_CONF_DIR=/opt/modules/hadoop-2.6.0-cdh5.4.0/etc/hadoop

val textFile = sc.textFile("hdfs://spark.learn.com:8020/user/hadoop/spark")

val wordcount = textFile.flatMap(x=>x.split(" ")).map(x=>(x,1)).reduceByKey((a,b)=>a+b).collect()

val wordcount = textFile.flatMap(x=>x.split(" ")).map(x=>(x,1)).reduceByKey((a,b)=>a+b).sortByKey(true).collect()

val wordcount = textFile.flatMap(x=>x.split(" ")).map(x=>(x,1)).reduceByKey((a,b)=>a+b).map(x=>(x._2,x._1)).sortByKey(false).collect()

sc.textFile("hdfs://spark.learn.com:8020/user/cyhp/spark/wc.input").flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _).collect

3 spark on yarn

查看spark-submit参数

[hadoop@spark spark-1.3.0-bin-2.6.0-cdh5.4.0]$ bin/spark-submit --help

Spark assembly has been built with Hive, including Datanucleus jars on classpath
Usage: spark-submit [options] <app jar | python file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Options:
  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                              on one of the worker machines inside the cluster ("cluster")
                              (Default: client).
  --class CLASS_NAME          Your application's main class (for Java / Scala apps).
  --name NAME                 A name of your application.
  --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths.
  --packages                  Comma-separated list of maven coordinates of jars to include
                              on the driver and executor classpaths. Will search the local
                              maven repo, then maven central and any additional remote
                              repositories given by --repositories. The format for the
                              coordinates should be groupId:artifactId:version.
  --repositories              Comma-separated list of additional remote repositories to
                              search for the maven coordinates given with --packages.
  --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
                              on the PYTHONPATH for Python apps.
  --files FILES               Comma-separated list of files to be placed in the working
                              directory of each executor.
  --conf PROP=VALUE           Arbitrary Spark configuration property.
  --properties-file FILE      Path to a file from which to load extra properties. If not
                              specified, this will look for conf/spark-defaults.conf.
  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 512M).
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.
  --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).
  --proxy-user NAME           User to impersonate when submitting the application.
  --help, -h                  Show this help message and exit
  --verbose, -v               Print additional debug output
  --version,                  Print the version of current Spark
 Spark standalone with cluster deploy mode only:
  --driver-cores NUM          Cores for driver (Default: 1).
  --supervise                 If given, restarts the driver on failure.
  --kill SUBMISSION_ID        If given, kills the driver specified.
  --status SUBMISSION_ID      If given, requests the status of the driver specified.
 Spark standalone and Mesos only:
  --total-executor-cores NUM  Total cores for all executors.
 YARN-only:
  --driver-cores NUM          Number of cores used by the driver, only in cluster mode
                              (Default: 1).
  --executor-cores NUM        Number of cores per executor (Default: 1).
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --num-executors NUM         Number of executors to launch (Default: 2).
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.

1，submit本地模式

bin/spark-submit \
    --class org.apache.spark.examples.SparkPi \
    lib/spark-examples-1.3.0-hadoop2.6.0-cdh5.4.0.jar \
    10  

2，submit运行于standalone模式

bin/spark-submit \
    --deploy-mode cluster \
    --class org.apache.spark.examples.SparkPi \
    lib/spark-examples-1.3.0-hadoop2.6.0-cdh5.4.0.jar \
    10

3，submit提交到yarn上
yarn Cluster mode

yarn Client mode

bin/spark-submit \
    --master yarn \
    --class org.apache.spark.examples.SparkPi \
    lib/spark-examples-1.3.0-hadoop2.6.0-cdh5.4.0.jar \
    10  

4，spark监控界面

1，启动spark监控服务

执行下面指令启动spark historyserver

./sbin/start-history-server.sh

查看方式

http://<server-url>:18080

2，spark监控的配置项介绍

1，整体配置项

我们可以对historyserver进行如下配置：

2，SPARK_HISTORY_OPTS配置

3，标记一个任务是否完成

要注意的是historyserver仅仅显示完成了的spark任务。标记任务完成的一种方式是直接调用sc.stop()。

4，spark-defaults.conf中的配置

2，具体配置

1，在spark-env.sh

SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://spark.learn.com:8020/user/hadoop/spark/history"

2，在spark-defaults.conf

spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://spark.learn.com:8020/user/hadoop/spark/history
spark.eventLog.compress         true

3，测试

启动相关服务

sbin/start-master.sh
sbin/start-slaves.sh 
sbin/start-history-server.sh
bin/spark-shell

执行spark应用

val textFile = sc.textFile("hdfs://spark.learn.com:8020/user/hadoop/spark/input/")
textFile.count
sc.stop