[关闭]
@sasaki 2016-05-12T07:56:12.000000Z 字数 11056 阅读 3505

MAC环境编译Hadoop 2.6

BigData Hadoop

版本控制

  1.     @Title          MAC环境编译Hadoop 2.6
  2.     @Version        v1.0
  3.     @Timestamp      2016-05-12 19:44
  4.     @Author         Nicholas
  5.     @Mail           redskirt@outlook.com

硬件环境：MacBook Pro Retina
软件环境：

  1. 解压hadoop-2.6.0-src.tar.gz至需要的路径，执行mvn编译

    1. Nicholas-MacBook-Pro:hadoop-2.6.0-src sasaki$ mvn install -DskipTests

    在进行到hadoop-common子项目时遇到如下异常：

    1. [INFO] ------------------------------------------------------------------------
    2. [INFO] BUILD FAILURE
    3. [INFO] ------------------------------------------------------------------------
    4. [INFO] Total time: 19.192 s
    5. [INFO] Finished at: 2016-05-12T09:51:44+08:00
    6. [INFO] Final Memory: 58M/284M
    7. [INFO] ------------------------------------------------------------------------
    8. [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.6.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1]
    9. [ERROR] 
    10. [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    11. [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    12. [ERROR] 
    13. [ERROR] For more information about the errors and possible solutions, please read the following articles:
    14. [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
    15. [ERROR] 
    16. [ERROR] After correcting the problems, you can resume the build with the command
    17. [ERROR]   mvn <goals> -rf :hadoop-common

  2. 异常信息说描述得很清楚，当前环境缺少protocbuf组件，protobuf是GitHub一个很火的开源项目。
    解决
    下载protocol buffer工具：
    https://developers.google.com/protocol-buffers/docs/downloads#latest-version
    （链接地址需要梯子，你懂的）

    1. Nicholas-MacBook-Pro:Downloads sasaki$ tar -zxvf protobuf-2.5.0.tar.gz 
    2.     # 解压并进入protobuf目录
    3.     Nicholas-MacBook-Pro:Downloads sasaki$ cd protobuf-2.5.0
    4. Nicholas-MacBook-Pro:~ sasaki$ ls /opt/protobuf-2.5.0/
    5. CHANGES.txt                  config.log                   ltmain.sh
    6. CONTRIBUTORS.txt             config.status                m4
    7. COPYING.txt                  config.sub                   missing
    8. INSTALL.txt                  configure                    protobuf-lite.pc
    9. Makefile                     configure.ac                 protobuf-lite.pc.in
    10. Makefile.am                  depcomp                      protobuf.pc
    11. Makefile.in                  editors                      protobuf.pc.in
    12. README.txt                   examples                     python
    13. aclocal.m4                   generate_descriptor_proto.sh src
    14. autogen.sh                   gtest                        stamp-h1
    15. config.guess                 install-sh                   vsprojects
    16. config.h                     java
    17. config.h.in                  libtool
    20. # 终端切换管理员
    22. Nicholas-MacBook-Pro:protobuf-2.6.1 sasaki$ sudo su
    23. Password:
    24. sh-3.2# ./configure --prefix=/usr/local/protobuf
    25. ...
    26. sh-3.2# make check
    27. ...过程比较漫长
    29. ...
    31. make[6]: Nothing to be done for `all-am`.
    32. ============================================================================
    35. Testsuite summary for Protocol Buffers 2.6.1
    36. ============================================================================
    39. # TOTAL: 5
    42. # PASS:  5
    45. # SKIP:  0
    48. # XFAIL: 0
    51. # FAIL:  0
    54. # XPASS: 0
    58. # ERROR: 0
    60. ============================================================================
    63. # 执行编译
    65. sh-3.2# make install
    67. # 检票是否安装成功
    69. sh-3.2# protoc --version
    70. libprotoc 2.5.0
    72. # 添加到路径
    74. sh-3.2# ./configure --prefix=/usr

  3. 安装完protoc后再次执行编译，遇到如下异常

    1. Exception in thread "main" java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()
    2.     at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:395)
    3.     at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:683)
    4.     at org.codehaus.mojo.jspc.CompilationMojoSupport.findToolsJar(CompilationMojoSupport.groovy:371)
    5.     at org.codehaus.mojo.jspc.CompilationMojoSupport.this$4$findToolsJar(CompilationMojoSupport.groovy)
    6.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    7.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    8.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    9.     at java.lang.reflect.Method.invoke(Method.java:606)
    10.     at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
    11.     at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
    12.     at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:912)
    13.     at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:78)
    14.     at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrent0(ScriptBytecodeAdapter.java:112)
    15.     at org.codehaus.mojo.jspc.CompilationMojoSupport.execute(CompilationMojoSupport.groovy:318)
    16.     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
    17.     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
    18.     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    19.     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    20.     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
    21.     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
    22.     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
    23.     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
    24.     at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
    25.     at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
    26.     at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
    27.     at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
    28.     at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
    29.     at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
    30.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    31.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    32.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    33.     at java.lang.reflect.Method.invoke(Method.java:606)
    34.     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    35.     at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    36.     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    37.     at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

    报错说在/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/Classes目录下没有classes.jar文件，但是我尝试进入该目录发现并没有../Classes/目录，结合上下文说缺失tools.jar文件，tools.jar本身是jdk安装目录自带的一个jar，但这里却说该目录没有，无奈Google之，发现解决方式很简单。

    解决
    找到$JAVA_HOME$/lib目录下的tools.jar，复制到报错的目录并重命名为classes.jar文件即可。

    1. Nicholas-MacBook-Pro:Classes sasaki$ pwd
    2. /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/Classes
    4. Nicholas-MacBook-Pro:Classes sasaki$ sudo ln -s ../lib/tools.jar classes.jar
    6. Nicholas-MacBook-Pro:Classes sasaki$ ls
    7. classes.jar

  4. 再次执行编译，一路顺畅，编译完成。

    1. [INFO] ------------------------------------------------------------------------
    2. [INFO] Reactor Summary:
    3. [INFO] 
    4. [INFO] Apache Hadoop Main ................................. SUCCESS [  2.021 s]
    5. [INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.984 s]
    6. [INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.269 s]
    7. [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.898 s]
    8. [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.352 s]
    9. [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.099 s]
    10. [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  0.976 s]
    11. [INFO] Apache Hadoop Auth ................................. SUCCESS [  1.403 s]
    12. [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  1.706 s]
    13. [INFO] Apache Hadoop Common ............................... SUCCESS [ 17.642 s]
    14. [INFO] Apache Hadoop NFS .................................. SUCCESS [  0.344 s]
    15. [INFO] Apache Hadoop KMS .................................. SUCCESS [  2.618 s]
    16. [INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.041 s]
    17. [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 40.273 s]
    18. [INFO] Apache Hadoop HttpFS ............................... SUCCESS [  3.332 s]
    19. [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  1.027 s]
    20. [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  0.803 s]
    21. [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.056 s]
    22. [INFO] hadoop-yarn ........................................ SUCCESS [  0.082 s]
    23. [INFO] hadoop-yarn-api .................................... SUCCESS [  3.896 s]
    24. [INFO] hadoop-yarn-common ................................. SUCCESS [  4.698 s]
    25. [INFO] hadoop-yarn-server ................................. SUCCESS [  0.038 s]
    26. [INFO] hadoop-yarn-server-common .......................... SUCCESS [  1.289 s]
    27. [INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [  2.358 s]
    28. [INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  0.730 s]
    29. [INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  1.390 s]
    30. [INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [  3.952 s]
    31. [INFO] hadoop-yarn-server-tests ........................... SUCCESS [  0.893 s]
    32. [INFO] hadoop-yarn-client ................................. SUCCESS [  1.084 s]
    33. [INFO] hadoop-yarn-applications ........................... SUCCESS [  0.055 s]
    34. [INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  0.749 s]
    35. [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  0.494 s]
    36. [INFO] hadoop-yarn-site ................................... SUCCESS [  0.054 s]
    37. [INFO] hadoop-yarn-registry ............................... SUCCESS [  1.142 s]
    38. [INFO] hadoop-yarn-project ................................ SUCCESS [  0.230 s]
    39. [INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.151 s]
    40. [INFO] hadoop-mapreduce-client-core ....................... SUCCESS [  3.719 s]
    41. [INFO] hadoop-mapreduce-client-common ..................... SUCCESS [  1.926 s]
    42. [INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  0.686 s]
    43. [INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  2.048 s]
    44. [INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  1.281 s]
    45. [INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  2.564 s]
    46. [INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  0.440 s]
    47. [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  0.902 s]
    48. [INFO] hadoop-mapreduce ................................... SUCCESS [  0.144 s]
    49. [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  0.916 s]
    50. [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 23.791 s]
    51. [INFO] Apache Hadoop Archives ............................. SUCCESS [  0.668 s]
    52. [INFO] Apache Hadoop Rumen ................................ SUCCESS [  1.217 s]
    53. [INFO] Apache Hadoop Gridmix .............................. SUCCESS [  1.033 s]
    54. [INFO] Apache Hadoop Data Join ............................ SUCCESS [  0.565 s]
    55. [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  0.334 s]
    56. [INFO] Apache Hadoop Extras ............................... SUCCESS [  0.714 s]
    57. [INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.055 s]
    58. [INFO] Apache Hadoop OpenStack support .................... SUCCESS [  0.869 s]
    59. [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  3.328 s]
    60. [INFO] Apache Hadoop Client ............................... SUCCESS [  0.629 s]
    61. [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.235 s]
    62. [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  0.883 s]
    63. [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  0.621 s]
    64. [INFO] Apache Hadoop Tools ................................ SUCCESS [  0.058 s]
    65. [INFO] Apache Hadoop Distribution ......................... SUCCESS [  0.209 s]
    66. [INFO] ------------------------------------------------------------------------
    67. [INFO] BUILD SUCCESS
    68. [INFO] ------------------------------------------------------------------------
    69. [INFO] Total time: 02:36 min
    70. [INFO] Finished at: 2016-05-12T14:54:57+08:00
    71. [INFO] Final Memory: 182M/546M
    72. [INFO] ------------------------------------------------------------------------
添加新批注
在作者公开此批注前，只有你和作者可见。
回复批注