Bryan

:: 管理

37 Posts :: 3 Stories :: 24 Comments :: 0 Trackbacks

Please refer to hadoop-2.5.0-src\BUILDING.txt and install all the required components.

1. install jdk1.6.0_45 and configure the JAVA_HOME and PATH, CLASSPATH, install apache-maven-3.2.3 and config MAVEN_HOME and also the PATH
use java -version, javac -version, mvn -version to validate the installation.

2. install windows sdk with the following URL : http://www.microsoft.com/en-us/download/details.aspx?id=8279

Windows SDK 7.1 Command Prompt will appear

3. download and install cygwin64 setup-x86_64.exe,and add C:\cygwin64\bin to the path so that shell command(shell command in pom.xml) can be used.
shell command , eg sh command can be used

4. download and install protoc-2.5.0-win32.zip, place in system32 folder or added to the path.
use command protoc --version to test

5. download and install zlib(firstly,I installed and then uninstall, but it also can run, I am not sure whether It is required.)

6. get hadoop source hadoop-2.5.0-src.tar.gz and

open Windows SDK 7.1 Command Prompt and build hadoop with the following command

set Platform=x64
cd C:\hadoop\hadoop-2.5.0-src
mvn package -Pdist,native-win -DskipTests -Dtar

The following is the path for my win7 system and I manually add the jdk ,maven and cygwin64 to User PATH variable , others are automatically generated in System PATH variable.

C:\hadoop\hadoop-2.5.0-src>path
PATH=C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Windows\Microsoft.NET\Fr
amework\v4.0.30319;C:\Windows\Microsoft.NET\Framework64\v3.5;C:\Windows\Microsof
t.NET\Framework\v3.5;;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common
7\IDE;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools;;C:\Prog
ram Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64;C:\Program Files (x86)
\Microsoft Visual Studio 10.0\VC\Bin\VCPackages;;C:\Program Files\Microsoft SDKs
\Windows\v7.1\Bin\NETFX 4.0 Tools\x64;C:\Program Files\Microsoft SDKs\Windows\v7
.1\Bin\x64;C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin;;C:\Windows\system32
;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\Java\jdk1.7.0_67\bin
;C:\hadoop\apache-maven-3.2.3\bin;C:\cygwin64\bin;

I did not install findbugs ,but It also can buid successfully.

The following are errors I encounter while compiling hadoop 2.5.0.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:

run (pre-dist) on project hadoop-project-dist: An Ant BuildException has occured

: Execute failed: java.io.IOException: Cannot run program "sh" (in directory "C:

\hadoop\hadoop-2.5.0-src\hadoop-project-dist\target"): CreateProcess error=2, ??

???????

[ERROR] around Ant part ...<exec dir="C:\hadoop\hadoop-2.5.0-src\hadoop-project-

dist\target" executable="sh" failonerror="true">... @ 31:104 in C:\hadoop\hadoop

-2.5.0-src\hadoop-project-dist\target\antrun\build-main.xml

[ERROR] -> [Help 1]

cygwin64 should be installed and add C:\cygwin64\bin to the path

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (com

pile-ms-winutils) on project hadoop-common: Command execution failed. Process ex

ited with an error: 1(Exit value: 1) -> [Help 1]

Windows SDK 7.1 Command Prompt run the following cmd first before mvn build.
set Platform=x64

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.5.0:prot

oc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecut

ionException: 'protoc --version' did not return a version -> [Help 1]

add protoc.exe to the PATH or just place it in WINDOWS/system32 folder

I have uploaded different versions of hadoop build and they can be found in the location. they are compiled with the following requirements

jdk 1.6.0_45
apache-maven-3.2.3

Cygwin64
Windows SDK 7.1
protoc 2.5.0

hadoop eclipse plugins build

I have also compiled the hadoop eclipse plugins and you can find them with URL hadoop eclipse plugin.It includes versions of 2.2.0 2.3.0 2.4.0 2.4.1 and 2.5.0.
I tested and sometimes the following errors happen and I extract the eclipse again from eclipse build and they works, so I think the jar has no problem. as I have no idea of eclipse plugin development, so I am not sure about actual reason.

java.io.IOException: Unable to resolve plug-in "platform:/plugin/org.apache.hadoop.eclipse/resources/hadoop-logo-16x16.png".
    at org.eclipse.core.internal.runtime.PlatformURLPluginConnection.parse(PlatformURLPluginConnection.java:64)
    at org.eclipse.core.internal.runtime.PlatformURLPluginConnection.resolve(PlatformURLPluginConnection.java:75)
    at org.eclipse.core.internal.boot.PlatformURLHandler.openConnection(PlatformURLHandler.java:67)
    at org.eclipse.osgi.internal.url.URLStreamHandlerProxy.openConnection(URLStreamHandlerProxy.java:114)
    at java.net.URL.openConnection(URL.java:971)
    at org.eclipse.core.internal.runtime.PlatformURLConverter.toFileURL(PlatformURLConverter.java:33)
    at org.eclipse.core.runtime.FileLocator.toFileURL(FileLocator.java:206)
    at org.eclipse.jface.resource.URLImageDescriptor.getFilePath(URLImageDescriptor.java:141)
    at org.eclipse.jface.resource.URLImageDescriptor.createImage(URLImageDescriptor.java:162)
    at org.eclipse.jface.resource.ImageDescriptor.createImage(ImageDescriptor.java:230)
    at org.eclipse.jface.resource.ImageDescriptor.createImage(ImageDescriptor.java:208)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.getImageFromURI(SWTPartRenderer.java:221)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.getImage(SWTPartRenderer.java:233)
    at org.eclipse.e4.ui.workbench.renderers.swt.StackRenderer.createTab(StackRenderer.java:829)
    at org.eclipse.e4.ui.workbench.renderers.swt.LazyStackRenderer.processContents(LazyStackRenderer.java:134)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.SashRenderer.processContents(SashRenderer.java:185)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.SashRenderer.processContents(SashRenderer.java:185)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.PerspectiveRenderer.processContents(PerspectiveRenderer.java:49)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.PerspectiveStackRenderer.showTab(PerspectiveStackRenderer.java:103)
    at org.eclipse.e4.ui.workbench.renderers.swt.LazyStackRenderer.postProcess(LazyStackRenderer.java:98)
    at org.eclipse.e4.ui.workbench.renderers.swt.PerspectiveStackRenderer.postProcess(PerspectiveStackRenderer.java:77)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:678)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.SashRenderer.processContents(SashRenderer.java:185)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.WBWRenderer.processContents(WBWRenderer.java:664)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1078)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
    at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
    at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
    at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:135)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:382)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:236)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1438)

Ref http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os?utm_source=tuicool
http://www.codeproject.com/Articles/757934/Apache-Hadoop-for-Windows-Platform

posted on 2014-08-22 21:00 Life is no respector of any genius. 閱讀(8961) 評論(6) 編輯收藏所屬分類: hadoop

Feedback

# re: Hadoop2.5.0 build on win7 64 2014-11-01 01:23 lakshmi

I am getting this error and I dont know how to resolve it. I have cygwin64's path in my environment variables.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:
run (pre-dist) on project hadoop-project-dist: An Ant BuildException has occured
: exec returned: 2 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal o
rg.apache.maven.plugins:maven-antrun-plugin:1.6:run (pre-dist) on project hadoop
-project-dist: An Ant BuildException has occured: exec returned: 2
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:216)......

.......

Caused by: C:\Users\xxxx\Documents\hdfs\hadoop-project-dist\target\antrun\build-main.xml:31: exec returned: 2

Can you help me with this?

I am trying to build hadoop 2.2.0 but even 2.5.0 is giving the same error. Os: Windows 7 64 bit 回復更多評論

# re: Hadoop2.5.0 build on win7 64 2014-11-01 12:00 Life is no respector of any genius.

You have to test the sh command first and see whether It works on windows. From the error output(exec returned: 2), It seems the exec command can not be completed successfully, please check why It resulted in the error.

you can copy the shell script to one file and replace the variables and test to see what are the problems.

<exec executable="sh" dir="${project.build.directory}" failonerror="true">
<arg line="./dist-copynativelibs.sh"/>
</exec> 回復更多評論

# re: Hadoop2.5.0 build on win7 64 2014-11-22 15:35 bryan

All the hadoop windows build and eclipse plugin can be found
http://yun.baidu.com/share/home?uk=3976278079&view=share#category/type=0 回復更多評論

# re: Hadoop2.5.0 build on win7 64 2014-12-21 08:45 Life is no respector of any genius.

In order to use hadoop eclipse plugins2.6.0(not include previous versions) in windows eclipse for hadoop 2.6.0, we need to do the following things.

1. use hadoop2.6.0 and config the hadoop_home and path and ensure It points to hadoop2.6.0 env.If this step is ignored, NativeCrc32 will has Unsatisfied Linkage error.

2. the plugin jar file needs to include one more jar file htrace-core-3.0.4.jar compared with previous versions.

回復更多評論

# re: Hadoop2.5.0 build on win7 64 2015-05-09 19:54 Life is no respector of any genius.

hadoop-eclipse-plugin-2.7.0.jar

1. update the following jar filename(htrace-core-3.1.0-incubating.jar) and add servlet-api-2.5.jar file to classpath. 2.Tested with eclipse-jee-luna-SR2-win32-x86_64.

http://download.csdn.net/detail/tondayong1981/8680589

hadoop-eclipse-plugin-2.6.0.jar

It takes me two days to find the root cause for the exceptions and finally generate this jar file for hadoop 2.6.0.

1.use hadoop 2.6.0.
2.change the hadoop_home and path to point to hadoop 2.6.0
3.include one more jar file htrace-core-3.0.4.jar compared with previous versions.

Tested with eclipse-jee-luna-R-win32-x86_64.
Any problem,please let me know.
http://download.csdn.net/detail/tondayong1981/8280995 回復更多評論

# re: Hadoop2.5.0 build on win7 64 2015-07-12 07:46 Life is no respector of any genius.

hadoop 2.7.1 eclipse plugin
http://download.csdn.net/detail/tondayong1981/8889075

hadoop 2.7.1 win7
http://pan.baidu.com/s/1dDwF4op 回復更多評論

新用戶注冊刷新評論列表


只有注冊用戶登錄后才能發表評論。




網站導航: 博客園 IT新聞 Chat2DB C++博客博問
相關文章: Hadoop2.7.0 build on win7 64 Hadoop2.5.0 build on win7 64

Bryan

常用鏈接

留言簿(4)

我參與的團隊

隨筆分類

隨筆檔案

文章分類

我的博客

搜索

最新評論

閱讀排行榜

評論排行榜

Feedback