我是在一次REDFLAG公司到我们学校来宣传他们的一个开源比赛的时候才开始关注LINUX的,两年以前,而且也只是拿着他们送的REDFLAG装着玩而已,很多东西都不懂的。当时还特意要了一张ASIANUX,不知道有没有拼错,就是他们的LINUX服务器系统,我还记得那个递给我光盘的人是REDFLAG教育学院的副院长淮晋阳老师,呵呵~
我是本科生,“搞HADOOP”说得有点太抬举了,我们只是配置一下然后跑一下示例程序而已~
关于上面提到的一些问题,我想我在这里描述说不清楚,我来把终端里的输出结果发出来给各位老师看看吧。
首先是我的JAVAHOME设置问题,这里主要涉及两个文件,一个是家目录下的.profile,一个是/etc/profile,下面分别是两个文件的关键部分的内容。
这是我的jdk安装位置,按老师给的说明设置的,在我的家目录下。有点不太合理,因为其它用户访问不到。
代码: 全选
neo@neo-laptop:~/jdk1.6.0_25$ ls
bin include man register_zh_CN.html
COPYRIGHT jre README.html sample
db lib register.html src.zip
demo LICENSE register_ja.html THIRDPARTYLICENSEREADME.txt
下面是/etc/profile文件:
代码: 全选
#set java environment
export JAVA_HOME=/home/neo/jdk1.6.0_25
export JRE_HOME=/home/neo/jdk1.6.0_25/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
下面是~/.profile文件的最后几行:
代码: 全选
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
PATH="$HOME/jdk1.6.0_25/bin:$PATH"
PATH="$HOME/hadoop/bin:$PATH"
这个是输入$JAVA_HOME的返回结果:
代码: 全选
neo@neo-laptop:~$ $JAVA_HOME
bash: /home/neo/jdk1.6.0_25: 是一个目录
这个是我登录localhost之后输入JAVAHOME的返回结果,在master里也是一样的。
代码: 全选
neo@neo-laptop:~$ ssh localhost
Linux neo-laptop 2.6.35-28-generic #50-Ubuntu SMP Fri Mar 18 19:00:26 UTC 2011 i686 GNU/Linux
Ubuntu 10.10
Welcome to Ubuntu!
* Documentation: https://help.ubuntu.com/
Last login: Thu May 12 08:53:27 2011 from localhost.localdomain
neo@neo-laptop:~$ $JAVA_HOME
-bash: /home/neo/jdk1.6.0_25: 是一个目录
这个是在格式化namenode之后,执行start-all.sh得到的结果,上面我提到的JAVAHOME问题:
代码: 全选
neo@neo-laptop:~$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-mapred.sh
starting namenode, logging to /home/neo/hadoop/bin/../logs/hadoop-neo-namenode-neo-laptop.out
localhost: Error: JAVA_HOME is not set.
master: Error: JAVA_HOME is not set.
starting jobtracker, logging to /home/neo/hadoop/bin/../logs/hadoop-neo-jobtracker-neo-laptop.out
localhost: Error: JAVA_HOME is not set.
下面是我提到的那个io错误的输出,这个有点长:
代码: 全选
neo@neo-laptop:~$ hadoop fs -put /home/neo/bcd.txt user/neo/input
11/05/12 16:07:15 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/05/12 16:07:16 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
11/05/12 16:07:16 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: File /user/neo/user/neo/input could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1448)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)
at org.apache.hadoop.ipc.Client.call(Client.java:905)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:928)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:811)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427)
put: File /user/neo/user/neo/input could only be replicated to 0 nodes, instead of 1
11/05/12 16:07:16 ERROR hdfs.DFSClient: Exception closing file /user/neo/user/neo/input : java.io.IOException: File /user/neo/user/neo/input could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1448)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)
java.io.IOException: File /user/neo/user/neo/input could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1448)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)
at org.apache.hadoop.ipc.Client.call(Client.java:905)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:928)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:811)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427)
JAVAHOME的问题最恼人。记得之前看过一个跟HADOOP有关的,也是JAVAHOME问题的帖子,当时也没怎么注意,我再找找看,看能不能找到。