Hadoop平台搭建说明 1. Hadoop节点规划本次安装规划使用三个节点,每个节点都使用centos系统。 三个节点的hostname分别规划为:centoshadoop1、centoshadoop2、centoshadoop3(此处为本教程参数,可根据实际环境情况修改) 三个节点的ip地址分别规划为:192.168.65.57、192.168.65.58、192.168.65.59(此处为本教程参数,根据实际环境情况修改) 2. 平台搭建使用的软件下载如下软件 操作系统安装包:Centos6.3_x64 Jdk安装包:jdk-6u37-linux-x64.bin Hadoop安装包:hadoop-1.1.2.tar.gz 3. 安装centos操作系统安装三个节点的操作系统,安装过程省略。 4. 配置centoshadoop1节点4.1. 修改节点hostname[root@localhost ~]# vi /etc/sysconfig/network HOSTNAME=centoshadoop1 [root@localhost ~]# vi /etc/hosts …… 192.168.65.57 centoshadoop1 192.168.65.58 centoshadoop2 192.168.65.59 centoshadoop3 [root@localhost ~]#reboot 4.2. 关闭iptables防火墙[root@ centoshadoop1~]#service iptablesstop 注意每次操作系统重启后都要操作 4.3. 建立无ssh密码登陆生成签名文件 [root@ centoshadoop1~]#cd /root [root@ centoshadoop1~]#ssh-keygen -t dsa -P'' -f ~/.ssh/id_dsa [root@ centoshadoop1~]#cat~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys [root@ centoshadoop1~]# 测试本地SSH无密码登录 [root@ centoshadoop1~]#ssh centoshadoop1 4.4. 安装jdk上传jdk-6u37-linux-x64.bin到/root目录下 [root@ centoshadoop1~]#chmod 777 jdk-6u37-linux-x64.bin [root@ centoshadoop1~]#./jdk-6u37-linux-x64.bin [root@ centoshadoop1~]#ll 查看生成jdk-6u37-linux-x64目录 4.5. 安装hadoop软件上传hadoop-1.1.2.tar.gz到/root目录下 [root@ centoshadoop1~]#tar -zvxf hadoop-1.1.2.tar.gz [root@ centoshadoop1~]#ll 查看生成hadoop-1.1.2目录 [root@ centoshadoop1~]#vi /conf/core-site.xml <?xml version="1.0"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <!-- Put site-specific propertyoverrides in this file. --> <configuration> <property> <name>fs.default.name</name> <value>hdfs://192.168.65.57:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/root/hadoop-1.1.2/tmp</value> </property> </configuration> [root@ centoshadoop1~]#vi hdfs-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl"href="configuration.xsl"?> <!-- Put site-specific propertyoverrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> [root@ centoshadoop1~]#vi mapred-site.xml <?xml version="1.0"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <!-- Put site-specific propertyoverrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>192.168.65.57:9001</value> </property> </configuration> 4.6. 配置hadoop数据节点[root@ centoshadoop1~]#vi /root/hadoop-1.1.2/conf/masters 192.168.65.57 #secondaryNameNode [root@ centoshadoop1~]#vi /root/hadoop-1.1.2/conf/slaves 192.168.65.58 #datanode 192.168.65.59 #datanode 4.7. 添加环境变量[root@ centoshadoop1~]#cd /root [root@ centoshadoop1~]#vi /etc/profile export JAVA_HOME=/root/jdk1.6.0_37 export JRE_HOME=/root/jdk1.6.0_37/jre export HADOOP_HOME=/root/hadoop-1.1.2 export HADOOP_HOME_WARN_SUPPRESS=1 export CLASSPATH=. JAVA_HOME/lib JRE_HOME/lib HADOOP_HOME/lib:$CLASSPATH exportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH [root@ centoshadoop1~]#vi ~/.bashrc export JAVA_HOME=/root/jdk1.6.0_37 export JRE_HOME=/root/jdk1.6.0_37/jre export HADOOP_HOME=/root/hadoop-1.1.2 export HADOOP_DEV_HOME=/root/hadoop-1.1.2 export HADOOP_COMMON_HOME=/root/hadoop-1.1.2 export HADOOP_HDFS_HOME=/root/hadoop-1.1.2 export HADOOP_CONF_DIR=/root/hadoop-1.1.2/conf [root@ centoshadoop1~]#vi /root/hadoop-1.1.2/conf/hadoop-env.sh export JAVA_HOME=/root/jdk1.6.0_37 export HADOOP_HOME_WARN_SUPPRESS=1 export HADOOP_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib 5. 配置centoshadoop2节点5.1. 修改节点hostname[root@localhost ~]# vi /etc/sysconfig/network HOSTNAME=centoshadoop2 [root@localhost ~]# vi /etc/hosts …… 192.168.65.57 centoshadoop1 192.168.65.58 centoshadoop2 192.168.65.59 centoshadoop3 [root@localhost ~]#reboot 5.2. 关闭iptables防火墙[root@ centoshadoop1~]#service iptablesstop 注意每次操作系统重启后都要操作 5.3. 建立无ssh密码登陆将centoshadoop1节点上/root/.ssh目录下的authorized_keys、known_hosts 、id_rsa.pub、id_rsa四个文件拷贝到本机的/root/.ssh目录下 5.4. 安装jdk上传jdk-6u37-linux-x64.bin到/root目录下 [root@ centoshadoop1~]#chmod 777 jdk-6u37-linux-x64.bin [root@ centoshadoop1~]#./jdk-6u37-linux-x64.bin [root@ centoshadoop1~]#ll 查看生成jdk-6u37-linux-x64目录 5.5. 安装hadoop软件上传hadoop-1.1.2.tar.gz到/root目录下 [root@ centoshadoop1~]#tar -zvxf hadoop-1.1.2.tar.gz [root@ centoshadoop1~]#ll 查看生成hadoop-1.1.2目录 [root@ centoshadoop1~]#vi /conf/core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl"href="configuration.xsl"?> <!-- Put site-specific propertyoverrides in this file. --> <configuration> <property> <name>fs.default.name</name> <value>hdfs://192.168.65.57:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/root/hadoop-1.1.2/tmp</value> </property> </configuration> [root@ centoshadoop1~]#vi hdfs-site.xml <?xml version="1.0"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <!-- Put site-specific propertyoverrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> [root@ centoshadoop1~]#vi mapred-site.xml <?xml version="1.0"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overridesin this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>192.168.65.57:9001</value> </property> </configuration> 5.6. 添加环境变量[root@ centoshadoop1~]#cd /root [root@ centoshadoop1~]#vi /etc/profile export JAVA_HOME=/root/jdk1.6.0_37 export JRE_HOME=/root/jdk1.6.0_37/jre export HADOOP_HOME=/root/hadoop-1.1.2 export HADOOP_HOME_WARN_SUPPRESS=1 exportCLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH exportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH [root@ centoshadoop1~]#vi ~/.bashrc export JAVA_HOME=/root/jdk1.6.0_37 export JRE_HOME=/root/jdk1.6.0_37/jre export HADOOP_HOME=/root/hadoop-1.1.2 export HADOOP_DEV_HOME=/root/hadoop-1.1.2 export HADOOP_COMMON_HOME=/root/hadoop-1.1.2 export HADOOP_HDFS_HOME=/root/hadoop-1.1.2 export HADOOP_CONF_DIR=/root/hadoop-1.1.2/conf [root@ centoshadoop1~]#vi /root/hadoop-1.1.2/conf/hadoop-env.sh export JAVA_HOME=/root/jdk1.6.0_37 export HADOOP_HOME_WARN_SUPPRESS=1 export HADOOP_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib 6. 配置centoshadoop3节点6.1. 修改节点hostname[root@localhost ~]# vi /etc/sysconfig/network HOSTNAME=centoshadoop3 [root@localhost ~]# vi /etc/hosts …… 192.168.65.57 centoshadoop1 192.168.65.58 centoshadoop2 192.168.65.59 centoshadoop3 [root@localhost ~]#reboot 关闭iptables防火墙、建立无ssh密码登陆、安装jdk、安装hadoop软件、添加环境变量等操作跟“配置centoshadoop2节点”完全相同 7. 启动运行hadoop平台7.1. 首先执行格式化[root@ centoshadoop1~]#hadoop namenode–format 注意:core-site.xml、hdfs-site.xml、mapred-site.xml等配置文件变化后都需要执行格式化 7.2. 启动hadoop[root@ centoshadoop1~]#start-all.sh 7.3. 停止hadoop[root@ centoshadoop1~]#stop-all.sh 7.4. 查看hadoop运行状态查看hadoop启动的进程 [root@ centoshadoop1~]#jps 查看hadoop集群状态 [root@ centoshadoop1~]#hadoop dfsadmin -report 查看hadoop中的目录情况 [root@ centoshadoop1~]#hadoop fs –ls /
|