MySQL社区

 找回密码
 注册

QQ登录

只需一步,快速开始

扫一扫,访问微社区

搜索
查看: 1936|回复: 0

hadoop平台搭建_多节点详细教程_一次就能搭建成功

[复制链接]
发表于 2016-10-30 10:27:22 | 显示全部楼层 |阅读模式
Hadoop平台搭建说明
1.  Hadoop节点规划
本次安装规划使用三个节点,每个节点都使用centos系统。
三个节点的hostname分别规划为:centoshadoop1centoshadoop2centoshadoop3(此处为本教程参数,可根据实际环境情况修改)
三个节点的ip地址分别规划为:192.168.65.57192.168.65.58192.168.65.59(此处为本教程参数,根据实际环境情况修改)
2.  平台搭建使用的软件
下载如下软件
操作系统安装包:Centos6.3_x64
Jdk安装包:jdk-6u37-linux-x64.bin
Hadoop安装包:hadoop-1.1.2.tar.gz
3.  安装centos操作系统
安装三个节点的操作系统,安装过程省略。
4.  配置centoshadoop1节点4.1.  修改节点hostname
[root@localhost ~]# vi /etc/sysconfig/network
HOSTNAME=centoshadoop1
[root@localhost ~]# vi /etc/hosts
……
192.168.65.57 centoshadoop1
192.168.65.58 centoshadoop2
192.168.65.59 centoshadoop3
[root@localhost ~]#reboot
4.2.  关闭iptables防火墙
[root@ centoshadoop1~]#service iptablesstop
注意每次操作系统重启后都要操作
4.3.  建立无ssh密码登陆
生成签名文件
[root@ centoshadoop1~]#cd /root
[root@ centoshadoop1~]#ssh-keygen -t dsa -P'' -f ~/.ssh/id_dsa
[root@ centoshadoop1~]#cat~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[root@ centoshadoop1~]#
测试本地SSH无密码登录
[root@ centoshadoop1~]#ssh centoshadoop1
4.4.  安装jdk
上传jdk-6u37-linux-x64.bin/root目录下
[root@ centoshadoop1~]#chmod 777 jdk-6u37-linux-x64.bin
[root@ centoshadoop1~]#./jdk-6u37-linux-x64.bin
[root@ centoshadoop1~]#ll
查看生成jdk-6u37-linux-x64目录
4.5.  安装hadoop软件
上传hadoop-1.1.2.tar.gz/root目录下
[root@ centoshadoop1~]#tar -zvxf hadoop-1.1.2.tar.gz
[root@ centoshadoop1~]#ll
查看生成hadoop-1.1.2目录
[root@ centoshadoop1~]#vi /conf/core-site.xml
<?xml version="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific propertyoverrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.65.57:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/root/hadoop-1.1.2/tmp</value>
</property>
</configuration>
[root@ centoshadoop1~]#vi hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl"href="configuration.xsl"?>
<!-- Put site-specific propertyoverrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
[root@ centoshadoop1~]#vi mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific propertyoverrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>192.168.65.57:9001</value>
</property>
</configuration>
4.6.  配置hadoop数据节点
[root@ centoshadoop1~]#vi /root/hadoop-1.1.2/conf/masters
192.168.65.57 #secondaryNameNode
[root@ centoshadoop1~]#vi /root/hadoop-1.1.2/conf/slaves
192.168.65.58  #datanode
192.168.65.59  #datanode
4.7.  添加环境变量
[root@ centoshadoop1~]#cd /root
[root@ centoshadoop1~]#vi /etc/profile
export JAVA_HOME=/root/jdk1.6.0_37
export JRE_HOME=/root/jdk1.6.0_37/jre
export HADOOP_HOME=/root/hadoop-1.1.2
export HADOOP_HOME_WARN_SUPPRESS=1
export CLASSPATH=.JAVA_HOME/libJRE_HOME/libHADOOP_HOME/lib:$CLASSPATH
exportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
[root@ centoshadoop1~]#vi ~/.bashrc
export JAVA_HOME=/root/jdk1.6.0_37
export JRE_HOME=/root/jdk1.6.0_37/jre
export HADOOP_HOME=/root/hadoop-1.1.2
export HADOOP_DEV_HOME=/root/hadoop-1.1.2
export HADOOP_COMMON_HOME=/root/hadoop-1.1.2
export HADOOP_HDFS_HOME=/root/hadoop-1.1.2
export HADOOP_CONF_DIR=/root/hadoop-1.1.2/conf
[root@ centoshadoop1~]#vi /root/hadoop-1.1.2/conf/hadoop-env.sh
export JAVA_HOME=/root/jdk1.6.0_37
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib
5.  配置centoshadoop2节点5.1.  修改节点hostname
[root@localhost ~]# vi /etc/sysconfig/network
HOSTNAME=centoshadoop2
[root@localhost ~]# vi /etc/hosts
……
192.168.65.57 centoshadoop1
192.168.65.58 centoshadoop2
192.168.65.59 centoshadoop3
[root@localhost ~]#reboot
5.2.  关闭iptables防火墙
[root@ centoshadoop1~]#service iptablesstop
注意每次操作系统重启后都要操作
5.3.  建立无ssh密码登陆
centoshadoop1节点上/root/.ssh目录下的authorized_keysknown_hosts id_rsa.pubid_rsa四个文件拷贝到本机的/root/.ssh目录下
5.4.  安装jdk
上传jdk-6u37-linux-x64.bin/root目录下
[root@ centoshadoop1~]#chmod 777 jdk-6u37-linux-x64.bin
[root@ centoshadoop1~]#./jdk-6u37-linux-x64.bin
[root@ centoshadoop1~]#ll
查看生成jdk-6u37-linux-x64目录
5.5.  安装hadoop软件
上传hadoop-1.1.2.tar.gz/root目录下
[root@ centoshadoop1~]#tar -zvxf hadoop-1.1.2.tar.gz
[root@ centoshadoop1~]#ll
查看生成hadoop-1.1.2目录
[root@ centoshadoop1~]#vi /conf/core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl"href="configuration.xsl"?>
<!-- Put site-specific propertyoverrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.65.57:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/root/hadoop-1.1.2/tmp</value>
</property>
</configuration>
[root@ centoshadoop1~]#vi hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific propertyoverrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
[root@ centoshadoop1~]#vi mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overridesin this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>192.168.65.57:9001</value>
</property>
</configuration>
5.6.  添加环境变量
[root@ centoshadoop1~]#cd /root
[root@ centoshadoop1~]#vi /etc/profile
export JAVA_HOME=/root/jdk1.6.0_37
export JRE_HOME=/root/jdk1.6.0_37/jre
export HADOOP_HOME=/root/hadoop-1.1.2
export HADOOP_HOME_WARN_SUPPRESS=1
exportCLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH
exportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
[root@ centoshadoop1~]#vi ~/.bashrc
export JAVA_HOME=/root/jdk1.6.0_37
export JRE_HOME=/root/jdk1.6.0_37/jre
export HADOOP_HOME=/root/hadoop-1.1.2
export HADOOP_DEV_HOME=/root/hadoop-1.1.2
export HADOOP_COMMON_HOME=/root/hadoop-1.1.2
export HADOOP_HDFS_HOME=/root/hadoop-1.1.2
export HADOOP_CONF_DIR=/root/hadoop-1.1.2/conf
[root@ centoshadoop1~]#vi /root/hadoop-1.1.2/conf/hadoop-env.sh
export JAVA_HOME=/root/jdk1.6.0_37
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib
6.  配置centoshadoop3节点6.1.  修改节点hostname
[root@localhost ~]# vi /etc/sysconfig/network
HOSTNAME=centoshadoop3
[root@localhost ~]# vi /etc/hosts
……
192.168.65.57 centoshadoop1
192.168.65.58 centoshadoop2
192.168.65.59 centoshadoop3
[root@localhost ~]#reboot
关闭iptables防火墙、建立无ssh密码登陆、安装jdk、安装hadoop软件、添加环境变量等操作跟“配置centoshadoop2节点”完全相同
7.  启动运行hadoop平台7.1.  首先执行格式化
[root@ centoshadoop1~]#hadoop namenode–format
注意:core-site.xmlhdfs-site.xmlmapred-site.xml等配置文件变化后都需要执行格式化
7.2.  启动hadoop
[root@ centoshadoop1~]#start-all.sh
7.3.  停止hadoop
[root@ centoshadoop1~]#stop-all.sh
7.4.  查看hadoop运行状态
查看hadoop启动的进程
[root@ centoshadoop1~]#jps
查看hadoop集群状态
[root@ centoshadoop1~]#hadoop dfsadmin  -report
查看hadoop中的目录情况
[root@ centoshadoop1~]#hadoop fs –ls /

QQ|申请友链|小黑屋|Archiver|手机版|MySQL社区 ( 京ICP备07012489号   
联系人:周生; 联系电话:13911732319

GMT+8, 2024-4-22 08:25 , Processed in 0.072169 second(s), 31 queries , Gzip On.

Powered by Discuz! X3.2

© 2001-2013 Comsenz Inc.

快速回复 返回顶部 返回列表