转载请注明出处,谢谢:
http://jackiee-cn.iteye.com/blog/1192526
问题:
hadoop安装完毕后,拿着hadoop的示例程序,尝试在单机环境下运行,先修改/etc/profile文件,增加如下内容:
export JAVA_HOME=/opt/jdk1.6.0_29
export HADOOP_HOME=/opt/hadoop-0.20.203.0
export HADOOP_CLASSPATH="$HADOOP_HOME/javademo"
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH
然后执行如下命令,让配置生效:
$. /etc/profile
再执行示例程序(或执行hadoop自带的wordcount示例):
hadoop demo.MaxTemperature input/sample.txt output
发现日志总停在如下状态:
mapred.JobClient: map 0% reduce 0%
通过查看hadoop的启动进程:
$ps -ef|grep hadoop
发现有一个进程异常终止,hadoop正常启动应该启动5个进程(namenode, datanode, secondary namenode, jobtracker, tasktracker),其中tasktracker异常终止了,通过查看 hadoop-root-tasktracker-***.log 日志,发现抛出了一个异常:
java.io.IOException: *Task process exit with nonzero status of 134.*
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
解决办法:
网上搜索了一番,最终终于把此问题解决掉了,把思路说明一下:
1、通过反编译查看org.apache.hadoop.mapred.TaskRunner类的258行附近的代码:
static
{
LOG.debug("Trying to load the custom-built native-hadoop library...");
try {
System.loadLibrary("hadoop");
LOG.info("Loaded the native-hadoop library");
nativeCodeLoaded = true;
}
catch (Throwable t) {
LOG.debug("Failed to load native-hadoop with error: " + t);
LOG.debug("java.library.path=" + System.getProperty("java.library.path"));
}
if (!nativeCodeLoaded)
LOG.warn("Unable to load native-hadoop library for your platform... using builtin-java classes where applicable");
}
感觉是加载本地库出了问题。
2、发现在$HADOOP_HOME目录下生成了如下日志文件:
hs_err_pid***.log
这个异常日志,打开一看,看不太懂,又从网上学习了一下,发现了一些线索,其中文件头的内容是这样的:
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGFPE (0x8) at pc=0x40008026, pid=31393, tid=2283477936
#
# JRE version: 6.0_29-b08
# Java VM: Java HotSpot(TM) Client VM (20.4-b01 mixed mode linux-x86 )
# Problematic frame:
# C [ld-linux.so.2+0x8026] double+0xb6
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
说明确实是加载本地库导致的异常。
3、网上学习了Hadoop本地库的一些介绍,发现有两种方法解决本地库的问题:
方法一是通过配置让本地库设置失效,在Hadoop的配置文件conf/core-site.xml中可以设置是否使用本地库:
<property>
<name>hadoop.native.lib</name>
<value>false</value>
</property>
测试后发现问题依然存在。
方法二是重新编译本地库:
于是下载了个ant,在/etc/profile中添加如下内容:
export ANT_HOME=/opt/apache-ant-1.8.0
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$ANT_HOME/bin:$PATH
执行如下命令,让配置生效:
$. /etc/profile
在$HADOOP_HOME目录下,使用如下命令:
$ant compile-native
编译完成后,可以在$HADOOP_HOME/build/native目录下找到相应的文件,然后移动编译好的文件到默认目录下。
最后重启hadoop服务,重新执行测试示例,终于看到了任务执行完毕的日志:
11/10/18 08:01:33 INFO mapred.JobClient: map 0% reduce 0%
11/10/18 08:01:55 INFO mapred.JobClient: map 50% reduce 0%
11/10/18 08:01:58 INFO mapred.JobClient: map 100% reduce 0%
11/10/18 08:02:10 INFO mapred.JobClient: map 100% reduce 100%
11/10/18 08:02:15 INFO mapred.JobClient: Job complete: job_201110180800_0001
最后别忘了将build目录移走或删除,否则有些意想不到的结果。
至此,纠结的问题得以解决,来杯
分享到:
相关推荐
Hadoop的MapTask类源代码分析
本地hadoop程序bug ExitCodeException exitCode=-1073741701 case1.双击本地hadoop目录下的bin/winutils.exe 提示操作系统缺少 msvcr100.dll文件 case2.如过第一种情况不成功,则安装常用C++库合集(x86 + 64位)
ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: \tmp\hadoop-admin \mapred\local\ttprivate to 0700 at org.apache...
Starting with the basics of Apache Hadoop and Solr, this book then dives into advanced topics of optimizing search with some interesting real-world use cases and sample Java code.
Title: Deep Learning with Hadoop Author: Dipayan Dev Length: 259 pages Edition: 1 Language: English Publisher: Packt Publishing Publication Date: 2017-03-06 ISBN-10: 1787124762 ISBN-13: 9781787124769 ...
This book will give you a complete understanding of the data lifecycle management with Hadoop, followed by modeling of structured and unstructured data in Hadoop. It will also show you how to design ...
"Data Analytics with Hadoop: An Introduction for Data Scientists" ISBN: 1491913703 | 2016 | PDF | 288 pages | 7 MB Ready to use statistical and machine-learning techniques across large data sets? ...
you can learn all the skills and techniques you'll need to deploy each key component of a Hadoop platform in your local environment or in the cloud, building a fully functional Hadoop cluster and ...
Instead of deployment, operations, or software development usually associated with distributed computing, you’ll focus on particular analyses you can build, the data warehousing techniques that ...
Big Data Processing With Hadoop is an essential reference source that discusses possible solutions for millions of users working with a variety of data applications, who expect fast turnaround ...
Big Data, MapReduce, Hadoop, and Spark with Python: Master Big Data Analytics and Data Wrangling with MapReduce Fundamentals using Hadoop, Spark, and Python by LazyProgrammer English | 15 Aug 2016 | ...
windows下编译的Hadoop2.7.4,使用方法简单直接解压到本地即可,解决在本地运行mapreduce程序连接不到hadoop客户端的问题。
在windows环境下开发hadoop时,需要配置HADOOP_HOME环境变量,变量值D:\hadoop-common-2.7.3-bin-master,并在Path追加%HADOOP_HOME%\bin,有可能出现如下错误: org.apache.hadoop.io.nativeio.NativeIO$Windows....
hadoop with python 用python 操作 hadoop 大数据集群
Wangda Tan and Wei-Chiu Chuang the current status of Apache Hadoop 3.x—how it’s used today in deployments large and small, and they dive into the exciting present and future of Hadoop 3.x—features ...
Hadoop使用常见问题以及解决方法,简单实用
eclipse远程调试hadoop时 报出eclipse Hadoop Failed to set permissions of path错误 修改hadoop core包中FileUtil java文件 里面有checkReturnValue方法 将代码throw new IOException "Failed to set ...