Hadoop集群部署实战(cdh发行版)
1、概要java
因为工做须要，最近一段时间开始接触学习hadoop相关的东西，目前公司的实时任务和离线任务都跑在一个hadoop集群，离线任务的特色就是天天定时跑，任务跑完了资源就空闲了，为了合理的利用资源，咱们打算在搭一个集群用于跑离线任务，计算节点和储存节点分离，计算节点结合aws的Auto Scaling（自动扩容、缩容服务）以及竞价实例，动态调整，在跑任务的时候拉起一批实例，任务跑完就自动释放掉服务器，本文记录下hadoop集群的搭建过程，方便本身往后查看，也但愿能帮到初学者，本文全部软件都是经过yum安装，你们也能够下载相应的二进制文件进行安装，使用哪一种方式安装，从属我的习惯。node
2、环境web
一、角色介绍shell
10.10.103.246 NameNode zkfc journalNode QuorumaPeerMain DataNode ResourceManager NodeManager WebAppProxyServer JobHistoryServer 10.10.103.144 NameNode zkfc journalNode QuorumaPeerMain DataNode ResourceManager NodeManager WebAppProxyServer 10.10.103.62 zkfc journalNode QuorumaPeerMain DataNode NodeManager
10.10.20.64 journalNode QuorumaPeerMain NameNode zkfc 10.10.40.212 journalNode QuorumaPeerMain NameNode zkfc 10.10.102.207 journalNode QuorumaPeerMain 10.10.103.15 ResourceManager WebAppProxyServer JobHistoryServer NodeManager DataNode 10.10.30.83 ResourceManager WebAppProxyServer JobHistoryServer NodeManager DataNode
二、基础环境说明apache
a、系统版本bootstrap
咱们用的是aws的ec2，用的aws本身定制过的系统，不过和redhat基本相同，内核版本：4.9.20-10.30.amzn1.x86_64vim
b、java版本浏览器
java version "1.8.0_121"
c、hadoop版本缓存
hadoop-2.6.0
d、cdh版本bash
cdh5.11.0
e、关于主机名，由于我这里用的aws的ec2，默认已有主机名，而且内网能够解析，故就不单独作主机名的配置了，若是你的主机名内网不能解析，请必定要配置主机名，集群内部通信不少组件使用的是主机名
3、配置部署
一、设置yum源
vim /etc/yum.repos.d/cloudera.repo [cloudera-cdh5-11-0] # Packages for Cloudera's Distribution for Hadoop, Version 5.11.0, on RedHat or CentOS 6 x86_64 name=Cloudera's Distribution for Hadoop, Version 5.11.0 baseurl=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.11.0/ gpgkey=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera gpgcheck=1 [cloudera-gplextras5b2] # Packages for Cloudera's GPLExtras, Version 5.11.0, on RedHat or CentOS 6 x86_64 name=Cloudera's GPLExtras, Version 5.11.0 baseurl=http://archive.cloudera.com/gplextras5/redhat/6/x86_64/gplextras/5.11.0/ gpgkey=http://archive.cloudera.com/gplextras5/redhat/6/x86_64/gplextras/RPM-GPG-KEY-cloudera gpgcheck=1
PS：我这里安装的5.11.0，若是想安装低版本或者高版本，根据本身的需求修改版本号便可
二、安装配置zookeeper集群
yum -y install zookeeper zookeeper-server
vi /etc/zookeeper/conf/zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper clientPort=2181 maxClientCnxns=0 server.1=10.10.103.144:2888:3888 server.2=10.10.103.226:2888:3888 server.3=10.10.103.62:2888:3888 autopurge.snapRetainCount=3 autopurge.purgeInterval=1
mkdir /data/zookeeper #建立datadir目录 /etc/init.d/zookeeper-server init #全部节点先初始化 echo 1 > /data/zookeeper/myid #10.10.103.144上操做 echo 2 > /data/zookeeper/myid #10.10.103.226上操做 echo 3 > /data/zookeeper/myid #10.10.103.62上操做 /etc/init.d/zookeeper-server #启动服务 /usr/lib/zookeeper/bin/zkServer.sh status #查看全部节点状态，其中只有一个节点是Mode: leader就正常 了
三、安装
a、10.10.103.246和10.10.103.144安装
yum -y install hadoop hadoop-client hadoop-hdfs hadoop-hdfs-namenode hadoop-hdfs-zkfc hadoop-hdfs-journalnode hadoop-hdfs-datanode hadoop-mapreduce-historyserver hadoop-yarn-nodemanager hadoop-yarn-proxyserver hadoop-yarn hadoop-mapreduce hadoop-yarn-resourcemanager hadoop-lzo* impala-lzo
b、10.10.103.62上安装
yum -y install hadoop hadoop-client hadoop-hdfs hadoop-hdfs-journalnode hadoop-hdfs-datanode hadoop-lzo* impala-lzo hadoop-yarn hadoop-mapreduce hadoop-yarn-nodemanager
PS：
一、通常小公司，计算节点(ResourceManager)和储存节点(NameNode)的主节点部署在两台服务器上作HA，计算节点（NodeManager）和储存节点（DataNode）部署在多台服务器上，每台服务器上都启动NodeManager和DataNode服务。
二、若是大集群，可能须要计算资源和储存资源分离，集群的各个角色都有服务器单独部署，我的建议划分以下:
a、储存节点
NameNode：
须要安装hadoop hadoop-client hadoop-hdfs hadoop-hdfs-namenode hadoop-hdfs-zkfc hadoop-lzo* impala-lzo
DataNode：
须要安装hadoop hadoop-client hadoop-hdfs hadoop-hdfs-datanode hadoop-lzo* impala-lzo
QJM集群：
须要安装hadoop hadoop-hdfs hadoop-hdfs-journalnode zookeeper zookeeper-server
b、计算节点
ResourceManager：
须要安装hadoop hadoop-client hadoop-yarn hadoop-mapreduce hadoop-yarn-resourcemanager
WebAppProxyServer：
须要安装 hadoop hadoop-yarn hadoop-mapreduce hadoop-yarn-proxyserver
JobHistoryServer：
须要安装 hadoop hadoop-yarn hadoop-mapreduce hadoop-mapreduce-historyserver
NodeManager：
须要安装hadoop hadoop-client hadoop-yarn hadoop-mapreduce hadoop-yarn-nodemanager
四、配置
a、建立目录并设置权限
mkdir -p /data/hadoop/dfs/nn #datanode上操做 chown hdfs:hdfs /data/hadoop/dfs/nn/ -R #datanode上操做 mkdir -p /data/hadoop/dfs/dn #namenode上操做 chown hdfs:hdfs /data/hadoop/dfs/dn/ -R #namenode上操做 mkdir -p /data/hadoop/dfs/jn #journalnode上操做 chown hdfs:hdfs /data/hadoop/dfs/jn/ -R #journalnode上操做 mkdir /data/hadoop/yarn -p #nodemanager上操做 chown yarn:yarn /data/hadoop/yarn -R #nodemanager上操做
b、撰写配置文件
vim /etc/hadoop/conf/capacity-scheduler.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property><name>yarn.scheduler.capacity.maximum-applications</name><value>10000</value></property> <property><name>yarn.scheduler.capacity.maximum-am-resource-percent</name><value>0.4</value></property> <property><name>yarn.scheduler.capacity.resource-calculator</name><value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value></property> <property><name>yarn.scheduler.capacity.node-locality-delay</name><value>30</value></property> <property><name>yarn.scheduler.capacity.root.queues</name><value>default,server,offline</value></property> <property><name>yarn.scheduler.capacity.root.default.capacity</name><value>95</value></property> <property><name>yarn.scheduler.capacity.root.default.maximum-capacity</name><value>100</value></property> <property><name>yarn.scheduler.capacity.root.default.user-limit-factor</name><value>100</value></property> <property><name>yarn.scheduler.capacity.root.default.state</name><value>running</value></property> <property><name>yarn.scheduler.capacity.root.default.acl_submit_applications</name><value>*</value></property> <property><name>yarn.scheduler.capacity.root.default.acl_administer_queue</name><value>*</value></property> <property><name>yarn.scheduler.capacity.root.server.capacity</name><value>0</value></property> <property><name>yarn.scheduler.capacity.root.server.maximum-capacity</name><value>5</value></property> <property><name>yarn.scheduler.capacity.root.server.user-limit-factor</name><value>100</value></property> <property><name>yarn.scheduler.capacity.root.server.acl_submit_applications</name><value>haijun.zhao</value></property> <property><name>yarn.scheduler.capacity.root.server.acl_administer_queue</name><value>haijun.zhao</value></property> <property><name>yarn.scheduler.capacity.root.server.maximum-am-resource-percent</name><value>0.05</value></property> <property><name>yarn.scheduler.capacity.root.server.state</name><value>running</value></property> <property><name>yarn.scheduler.capacity.root.offline.capacity</name><value>5</value></property> <property><name>yarn.scheduler.capacity.root.offline.maximum-capacity</name><value>100</value></property> <property><name>yarn.scheduler.capacity.root.offline.user-limit-factor</name><value>100</value></property> <property><name>yarn.scheduler.capacity.root.offline.acl_submit_applications</name><value>hadoop,haifeng.huang,hongan.pan,rujing.zhang,lingjing.li</value></property> <property><name>yarn.scheduler.capacity.root.offline.acl_administer_queue</name><value>hadoop,haifeng.huang,hongan.pan,rujing.zhang,linjing.li</value></property> <property><name>yarn.scheduler.capacity.root.offline.maximum-am-resource-percent</name><value>0.8</value></property> <property><name>yarn.scheduler.capacity.root.offline.state</name><value>running</value></property> </configuration> ~~~~~~~~~~~~~~~~~分界线~~~~~~~~~~~~~~~~~~~
vim /etc/hadoop/conf/core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <!-- 定义默认的文件系统--> <property><name>fs.defaultFS</name><value>hdfs://mycluster/</value></property> <!-- 开启垃圾回收，默认不开启，单位分钟 --> <property><name>fs.trash.interval</name><value>1440</value></property> <!-- 可用于压缩/解压缩的压缩编×××类，逗号分隔列表 --> <property><name>io.compression.codecs</name><value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.SnappyCodec</value></property> <!-- 开启lzo支持--> <property><name>io.compression.codec.lzo.class</name><value>com.hadoop.compression.lzo.LzoCodec</value></property> <!-- oozie user setting --> <property><name>hadoop.proxyuser.oozie.hosts</name><value>*</value></property> <property><name>hadoop.proxyuser.oozie.groups</name><value>*</value></property> <!-- HTTPFS proxy user setting --> <property><name>hadoop.proxyuser.httpfs.hosts</name><value>*</value></property> <property><name>hadoop.proxyuser.httpfs.groups</name><value>*</value></property> <!-- aws key id --> <property><name>fs.s3n.awsAccessKeyId</name><value>AKIAIXxxx</value></property> <property><name>fs.s3.awsAccessKeyId</name><value>AKIAIXxxx</value></property> <property><name>fs.s3bfs.awsAccessKeyId</name><value>AKIAIXxxx</value></property> <property><name>fs.s3bfs.awsSecretAccessKey</name><value>Hdne1k/2c90Ixxxxxx</value></property> <property><name>fs.s3.awsSecretAccessKey</name><value>Hdne1k/2c90Ixxxxxx</value></property> <property><name>fs.s3n.awsSecretAccessKey</name><value>Hdne1k/2c90Ixxxxxx</value></property> <property><name>fs.s3n.endpoint</name><value>s3.amazonaws.com</value></property> <property><name>fs.s3bfs.impl</name><value>org.apache.hadoop.fs.s3.S3FileSystem</value></property> <property><name>fs.s3.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value></property> </configuration> ~~~~~~~~~~~~~~~~~分界线~~~~~~~~~~~~~~~~~~~
vim /etc/hadoop/conf/hdfs-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <!-- 定义nameservice列表 --> <property><name>dfs.nameservices</name><value>mycluster</value></property> <!-- 定义副本数，默认3 --> <property><name>dfs.replication</name><value>2</value></property> <!-- 定义namenode的数据存储路径 --> <property><name>dfs.namenode.name.dir</name><value>/data/hadoop/dfs/nn</value></property> <!-- 定义datanode的数据存储路径 --> <property><name>dfs.datanode.data.dir</name><value>/data/hadoop/dfs/dn</value></property> <!-- 设置集群的超级用户 --> <property><name>dfs.permissions.superusergroup</name><value>hdfs</value></property> <!-- 在hdfs上启用权限检查 --> <property><name>dfs.permissions.enabled</name><value>false</value></property> <!-- 定义namenode的地址列表 --> <property><name>dfs.ha.namenodes.mycluster</name><value>10.10.20.64,10.10.40.212</value></property> <!-- 定义namenode的rpc通信地址和端口 --> <property><name>dfs.namenode.rpc-address.mycluster.10.10.20.64</name><value>10.10.20.64:8020</value></property> <property><name>dfs.namenode.rpc-address.mycluster.10.10.40.212</name><value>10.10.40.212:8020</value></property> <!-- 定义namenode web管理地址和端口 --> <property><name>dfs.namenode.http-address.mycluster.10.10.20.64</name><value>10.10.20.64:50070</value></property> <property><name>dfs.namenode.http-address.mycluster.10.10.40.212</name><value>10.10.40.212:50070</value></property> <!-- 定义namenode的edits文件存储的journal集群地址--> <property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://10.10.20.64:8485;10.10.40.212:8485;10.10.102.207:8485/mycluster</value></property> <!-- 定义journalnode的文件存储路径 --> <property><name>dfs.journalnode.edits.dir</name><value>/data/hadoop/dfs/jn</value></property> <!-- 启用客户端自动切换到Active的NameNode --> <property><name>dfs.client.failover.proxy.provider.mycluster</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property> <!-- run an arbitrary shell command to fence the Active NameNode --> <property><name>dfs.ha.fencing.methods</name><value>shell(/bin/true)</value></property> <!-- 开启namenode自动切换 --> <property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property> <!-- 存储namenode的ha相关信息的zk地址 --> <property><name>ha.zookeeper.quorum</name><value>10.10.20.64:2181,10.10.40.212:2181,10.10.102.207:2181</value></property> <!--默认文件的块大小，以字节为单位，默认是134217728 --> <property><name>dfs.blocksize</name><value>134217728</value></property> <!-- namenode中用于处理RPC的线程数，默认是10 --> <property><name>dfs.namenode.handler.count</name><value>128</value></property> <!-- datanode上用于处理RPC的线程数。默认为3 --> <property><name>dfs.datanode.handler.count</name><value>64</value></property> <!--datanode预留空间，默认是0，单位是字节，好比该节点有1T空间，想让hdfs预留100G给系统使用，能够设置为107374182400 --> <property><name>dfs.datanode.du.reserved</name><value>107374182400</value></property> <!-- 定义了每一个datanode直接平衡操做所允许的最大带宽，单位byte --> <property><name>dfs.balance.bandwidthPerSec</name><value>1048576</value></property> <!-- 定义不容许链接到NameNode的主机列表的文件。该文件的完整路径必须指定。若是该值是空的，没有主机被排除在外 --> <property><name>dfs.hosts.exclude</name><value>/etc/hadoop/conf.mycluster/datanodes.exclude</value></property> <!-- 指定要用于进出DN的数据传输线程的最大数量，默认4096 --> <property><name>dfs.datanode.max.transfer.threads</name><value>4096</value></property> <!-- datanode数据副本存放磁盘选择策略有两种方式：第一种是沿用hadoop1.0的磁盘目录轮询方式，实现类：RoundRobinVolumeChoosingPolicy.Java；第二种是选择可用空间足够多的磁盘方式存储，实现类：AvailableSpaceVolumeChoosingPolicy.java--> <property><name>dfs.datanode.fsdataset.volume.choosing.policy</name><value>org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy</value></property> <property><name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold</name><value>10737418240</value></property> <property><name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction</name><value>0.75</value></property> <!-- A Hadoop HDFS DataNode has an upper bound on the number of files that it can serve at any one time --> <property><name>dfs.datanode.max.xcievers</name><value>4096</value></property> <!-- enable webhdfs --> <property><name>dfs.webhdfs.enabled</name><value>true</value></property> <property><name>dfs.checksum.type</name><value>CRC32</value></property> <property><name>dfs.client.file-block-storage-locations.timeout</name><value>3000</value></property> <property><name>dfs.datanode.hdfs-blocks-metadata.enabled</name><value>true</value></property> <property><name>dfs.namenode.safemode.threshold-pct</name><value>0.85</value></property> <property><name>fs.s3.awsAccessKeyId</name><value>AKIAIXxxx</value></property> <property><name>fs.s3.awsSecretAccessKey</name><value>Hdne1k/2c90Ixxxxxx</value></property> <property><name>fs.s3n.awsAccessKeyId</name><value>AKIAIXxxx</value></property> <property><name>fs.s3n.awsSecretAccessKey</name><value>Hdne1k/2c90Ixxxxxx</value></property> </configuration> ~~~~~~~~~~~~~~~~~分界线~~~~~~~~~~~~~~~~~~~
vim /etc/hadoop/conf/mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <!-- 用于执行MapReduce做业的运行时框架,能够是local, classic or yarn，默认是local --> <property><name>mapreduce.framework.name</name><value>yarn</value></property> <!-- MapReduce JobHistory服务器地址--> <property><name>mapreduce.jobhistory.address</name><value>10.10.103.15:10020</value></property> <!--MapReduce JobHistory服务器Web地址 --> <property><name>mapreduce.jobhistory.webapp.address</name><value>10.10.103.15:19888</value></property> <!-- 提交做业时使用的目录 --> <property><name>yarn.app.mapreduce.am.staging-dir</name><value>/user</value></property> <!-- 每一个map任务的调度程序请求的内存量。若是没有指定，则从mapreduce.map.java.opts和mapreduce.job.heap.memory-mb.ratio推断出来。若是尚未指定java-opts，咱们将其设置为1024 --> <property><name>mapreduce.map.memory.mb</name><value>1536</value></property> <!--每一个reduce任务的调度程序请求的内存量 --> <property><name>mapreduce.reduce.memory.mb</name><value>2880</value></property> <!--MR AppMaster须要的内存量 --> <property><name>yarn.app.mapreduce.am.resource.mb</name><value>3072</value></property> <!--map的java最大内存配置 --> <property><name>mapreduce.map.java.opts</name><value>-Xmx1228m</value></property> <!--reducejava最大内存配置 --> <property><name>mapreduce.reduce.java.opts</name><value>-Xmx2456m</value></property> <!-- --> <property><name>yarn.app.mapreduce.am.command-opts</name><value>-Xmx2457m</value></property> <!--JobTracker的服务器线程数 --> <property><name>mapreduce.jobtracker.handler.count</name><value>128</value></property> <!--namenode用于处理RPC的线程数，默认64 --> <property><name>dfs.namenode.handler.count</name><value>128</value></property> <!--每一个map任务须要的虚拟cpu核数 --> <property><name>mapreduce.map.cpu.vcores</name><value>1</value></property> <!--个reduce任务须要的虚拟cpu核数 --> <property><name>mapreduce.reduce.cpu.vcores</name><value>1</value></property> <!--MR AppMaster须要的虚拟CPU核数量 --> <property><name>yarn.app.mapreduce.am.resource.cpu-vcores</name><value>1</value></property> <!-- --> <property><name>mapred.output.direct.EmrFileSystem</name><value>true</value></property> <!--在排序文件时一次合并的数量。这决定了打开文件句柄的数量,默认10 --> <property><name>mapreduce.task.io.sort.factor</name><value>48</value></property> <!--做业完成后，用户日志将被保留的最长时间（以小时为单位 --> <property><name>mapreduce.job.userlog.retain.hours</name><value>48</value></property> <!--reuduce shuffle阶段并行传输数据的数量，默认5 --> <property><name>mapreduce.reduce.shuffle.parallelcopies</name><value>20</value></property> <!-- --> <property><name>hadoop.job.history.user.location</name><value>none</value></property> <!--是否对Map Task启用推测执行机制 --> <property><name>mapreduce.map.speculative</name><value>true</value></property> <!--是否对Reduce Task启用推测执行机制 --> <property><name>mapreduce.reduce.speculative</name><value>true</value></property> <!--启用Map 输出到aws s3的支持 --> <property><name>mapred.output.direct.NativeS3FileSystem</name><value>true</value></property> <!--在经过网络发送以前，应该对映射的输出进行压缩。使用SequenceFile压缩 --> <property><name>mapreduce.map.output.compress</name><value>true</value></property> <!--用于从远程任务处理MR AppMaster中的RPC调用的线程数，默认30 --> <property><name>yarn.app.mapreduce.am.job.task.listener.thread-count</name><value>60</value></property> <!--每一个jvm运行多少个任务。若是设置为-1，则没有限制，默认1 --> <property><name>mapreduce.job.jvm.numtasks</name><value>20</value></property> <!--定义map任务输出压缩的方式，可使用lzo或者Snappy --> <property><name>mapreduce.map.output.compress.codec</name><value>org.apache.hadoop.io.compress.SnappyCodec</value></property> </configuration> ~~~~~~~~~~~~~~~~~分界线~~~~~~~~~~~~~~~~~~~
vim /etc/hadoop/conf/yarn-env.sh
#!/bin/bash
export HADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn}
export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"
if [ "$JAVA_HOME" != "" ]; then
#echo "run java in $JAVA_HOME"
JAVA_HOME=$JAVA_HOME
fi
if [ "$JAVA_HOME" = "" ]; then
echo "Error: JAVA_HOME is not set."
exit 1
fi
JAVA=$JAVA_HOME/bin/java
JAVA_HEAP_MAX=-Xmx1000m
if [ "$YARN_HEAPSIZE" != "" ]; then
JAVA_HEAP_MAX="-Xmx""$YARN_HEAPSIZE""m"
fi
IFS=
if [ "$YARN_LOG_DIR" = "" ]; then
YARN_LOG_DIR="$HADOOP_YARN_HOME/logs"
fi
if [ "$YARN_LOGFILE" = "" ]; then
YARN_LOGFILE='yarn.log'
fi
if [ "$YARN_POLICYFILE" = "" ]; then
YARN_POLICYFILE="hadoop-policy.xml"
fi
unset IFS
YARN_OPTS="$YARN_OPTS -Dhadoop.log.dir=$YARN_LOG_DIR"
YARN_OPTS="$YARN_OPTS -Dyarn.log.dir=$YARN_LOG_DIR"
YARN_OPTS="$YARN_OPTS -Dhadoop.log.file=$YARN_LOGFILE"
YARN_OPTS="$YARN_OPTS -Dyarn.log.file=$YARN_LOGFILE"
YARN_OPTS="$YARN_OPTS -Dyarn.home.dir=$YARN_COMMON_HOME"
YARN_OPTS="$YARN_OPTS -Dyarn.id.str=$YARN_IDENT_STRING"
YARN_OPTS="$YARN_OPTS -Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
YARN_OPTS="$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
YARN_OPTS="$YARN_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
fi
YARN_OPTS="$YARN_OPTS -Dyarn.policy.file=$YARN_POLICYFILE"
~~~~~~~~~~~~~~~~~分界线~~~~~~~~~~~~~~~~~~~
vim /etc/hadoop/conf/yarn-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <!-- the most important properties --> <property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property> <property><name>yarn.nodemanager.aux-services.spark_shuffle.class</name><value>org.apache.spark.network.yarn.YarnShuffleService</value></property> <property><name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property> <!--用做资源调度程序的类,咱们使用capacity作资源管理 --> <property><name>yarn.resourcemanager.scheduler.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value></property> <!--用于YARN应用程序的CLASSPATH，以逗号分隔的CLASSPATH条目列表 --> <property><name>yarn.application.classpath</name><value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value></property> <!-- specify the directories --> <!--用于存储本地化文件的目录列表,包括任务临时生产的文件以及缓存等等 -- > <property><name>yarn.nodemanager.local-dirs</name><value>/data/hadoop/yarn/local</value></property> <property><name>yarn.nodemanager.log-dirs</name><value>/data/hadoop/yarn/logs</value></property> <!--是否启用日志聚合。日志聚合收集每一个任务的日志，并在应用程序完成后将这些日志移动到hhdfs上,经过yarn.nodemanager.remote-app-log-dir参数 指定hdfs的存放路径 --> <property><name>yarn.log-aggregation-enable</name><value>true</value></property> <property><name>yarn.nodemanager.remote-app-log-dir</name><value>hdfs://mycluster/var/log/hadoop-yarn/apps</value></property> <!--设置任务内存限制时，虚拟内存与物理内存的比例。任务分配以物理内存表示，而且经过此比例容许虚拟内存使用超过此分配。任务每使用1MB物理内存，最多可以使用虚拟内存量，默认是2.1 --> <property><name>yarn.nodemanager.vmem-pmem-ratio</name><value>10</value></property> <!--该节点的物理内存大小，默认是8192，具体设置该参数时，建议给系统预留1-2G --> <property><name>yarn.nodemanager.resource.memory-mb</name><value>10360</value></property> <!--能够为任务分配的cpu虚拟核数。RM分配资源用于任务时使用。这个参数不是用来限制YARN任务使用的物理内核的数量.--> <property><name>yarn.nodemanager.resource.cpu-vcores</name><value>4</value></property> <!--否检查任务正在使用的虚拟内存量--> <property><name>yarn.nodemanager.pmem-check-enabled</name><value>true</value></property> <!--是否检查任务正在使用的虚拟内存量 --> <property><name>yarn.nodemanager.vmem-check-enabled</name><value>true</value></property> <!--RM处理调度器接口的线程数,默认50个--> <property><name>yarn.resourcemanager.scheduler.client.thread-count</name><value>64</value></property> <!--NM用于container管理的线程数，默认20个--> <property><name>yarn.nodemanager.container-manager.thread-count</name><value>64</value></property> <!--处理资源跟踪器调用的线程数，默认50个--> <property><name>yarn.resourcemanager.resource-tracker.client.thread-count</name><value>64</value></property> <!--用于处理应用程序管理器请求的线程数，默认50个--> <property><name>yarn.resourcemanager.client.thread-count</name><value>64</value></property> <!--处理本地请求的线程数，默认5个--> <property><name>yarn.nodemanager.localizer.client.thread-count</name><value>20</value></property> <!--用于本地化获取的线程数，默认4个--> <property><name>yarn.nodemanager.localizer.fetch.thread-count</name><value>20</value></property> <!-- 单个任务可申请的最少物理内存量，默认是1024（MB），若是一个任务申请的物理内存量少于该值，则该对应的值改成这个数--> <property><name>yarn.scheduler.minimum-allocation-mb</name><value>1536</value></property> <!--单个任务可申请的最多物理内存量，默认是8192（MB）。 默认状况下，YARN采用了线程监控的方法判断任务是否超量使用内存，一旦发现超量，则直接将其杀死。因为Cgroups对内存的控制缺少灵活性（即任务任什么时候刻不能超过内存上限，若是超过，则直接将其杀死或者报OOM），而Java进程在建立瞬间内存将翻倍，以后骤降到正常值，这种状况下，采用线程监控的方式更加灵活（当发现进程树内存瞬间翻倍超过设定值时，可认为是正常现象，不会将任务杀死），所以YARN未提供Cgroups内存隔离机制 --> <property><name>yarn.scheduler.maximum-allocation-mb</name><value>9192</value></property> <!--是否开启日志聚合功能 --> <property><name>yarn.log-aggregation-enable</name><value>true</value></property> <!-- 是否开启label功能 --> <property><name>yarn.label.enabled</name><value>true</value></property> <!-- enable resource manager HA --> <!--重试的时间间隔，默认是30000 --> <property><name>yarn.resourcemanager.connect.retry-interval.ms</name><value>2000</value></property> <!--是否启用RM高可用 -- > <property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property> <!--启用自动故障切换 -- > <property><name>yarn.resourcemanager.ha.automatic-failover.enabled</name><value>true</value></property> <!--启用RM后恢复状态。若是为true，则必须指定yarn.resourcemanager.store.class-- > <property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value></property> <!--集群名称。在HA设置中，这用于确保RM参与此群集的首选，并确保其不影响其余群集 -- > <property><name>yarn.resourcemanager.cluster-id</name><value>my-cluster</value></property> <!--启用HA时，群集中的RM节点列表 -- > <property><name>yarn.resourcemanager.ha.rm-ids</name><value>10.10.103.15,10.10.30.83</value></property> <!--当前RM的id，咱们常常部署集群会把一台服务器的配置问题同步到全部服务器上，另外一台RM记得修改这个ID为本身的IP地址 -- > <property><name>yarn.resourcemanager.ha.id</name><value>10.10.103.15</value></property> <!--配合yarn.resourcemanager.recovery.enabled参数一块儿使用 -- > <property><name>yarn.resourcemanager.store.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value></property> <!-- yarn proxyserver 的地址 --> <property><name>yarn.web-proxy.address</name><value>10.10.103.15:8100</value></property> <!--jobhistor的地址 --> <property><name>yarn.log.server.url</name><value>http://10.10.103.15:19888/jobhistory/logs/</value></property> <!--RM的地址--> <property><name>yarn.resourcemanager.hostname.10.10.103.15</name><value>10.10.103.15</value></property> <property><name>yarn.resourcemanager.hostname.10.10.30.83</name><value>10.10.30.83</value></property> <!--由RM使用的ZooKeeper服务器的端口。当在HA设置中使用基于ZooKeeper的RM状态存储和自动故障切换时，必须提供此功能 --> <property><name>yarn.resourcemanager.zk-address</name><value>10.10.20.64:2181,10.10.40.212:2181,10.10.102.207:2181</value></property> <!--RM中应用程序管理器界面的地址 --> <property><name>yarn.resourcemanager.address.10.10.103.15</name><value>10.10.103.15:23140</value></property> <!--调度器接口的地址 --> <property><name>yarn.resourcemanager.scheduler.address.10.10.103.15</name><value>10.10.103.15:23130</value></property> <!--RM Web应用程序的https adddress --> <property><name>yarn.resourcemanager.webapp.https.address.10.10.103.15</name><value>10.10.103.15:23189</value></property> <!--RM Web应用程序的http地址 --> <property><name>yarn.resourcemanager.webapp.address.10.10.103.15</name><value>10.10.103.15:8088</value></property> <!-- --> <property><name>yarn.resourcemanager.resource-tracker.address.10.10.103.15</name><value>10.10.103.15:23125</value></property> <!--M管理界面的地址--> <property><name>yarn.resourcemanager.admin.address.10.10.103.15</name><value>10.10.103.15:23141</value></property> <property><name>yarn.resourcemanager.address.10.10.30.83</name><value>10.10.30.83:23140</value></property> <property><name>yarn.resourcemanager.scheduler.address.10.10.30.83</name><value>10.10.30.83:23130</value></property> <property><name>yarn.resourcemanager.webapp.https.address.10.10.30.83</name><value>10.10.30.83:23189</value></property> <property><name>yarn.resourcemanager.webapp.address.10.10.30.83</name><value>10.10.30.83:8088</value></property> <property><name>yarn.resourcemanager.resource-tracker.address.10.10.30.83</name><value>10.10.30.83:23125</value></property> <property><name>yarn.resourcemanager.admin.address.10.10.30.83</name><value>10.10.30.83:23141</value></property> <property><name>yarn.nodemanager.localizer.address</name><value>0.0.0.0:23344</value></property> <!--NM Webapp地址--> <property><name>yarn.nodemanager.webapp.address</name><value>0.0.0.0:23999</value></property> <property><name>mapreduce.shuffle.port</name><value>23080</value></property> <!--启用RM工做保留恢复。此配置对YARN是私有的--> <property><name>yarn.resourcemanager.work-preserving-recovery.enabled</name><value>true</value></property> </configuration> ~~~~~~~~~~~~~~~~~分界线~~~~~~~~~~~~~~~~~~~
PS:配置参数解释参考下面连接
https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-common/core-default.xml
http://dongxicheng.org/mapreduce-nextgen/hadoop-yarn-configurations-capacity-scheduler/
五、服务启动
a、启动journalnode(三台服务器上都启动)
/etc/init.d/hadoop-hdfs-journalnode start
b、格式化namenode(在其中一台namenode10.10.103.246上操做)
sudo -u hdfs hadoop namenode -format
c、初始化zk中HA的状态(在其中一台namenode10.10.103.246上操做)
sudo -u hdfs hdfs zkfc -formatZK
d、初始化共享Edits文件(在其中一台namenode10.10.103.246上操做)
sudo -u hdfs hdfs namenode -initializeSharedEdits
e、启动10.10.103.246上namenode
/etc/init.d/hadoop-hdfs-namenode start
f、同步源数据并启动10.10.103.144上namenode
sudo -u hdfs hdfs namenode -bootstrapStandby /etc/init.d/hadoop-hdfs-namenode start
g、在两台namenode上启动zkfc
/etc/init.d/hadoop-hdfs-zkfc start
h、启动datanode(全部机器上操做)
/etc/init.d/hadoop-hdfs-journalnode start
i、在10.10.103.246上启动WebAppProxyServer、JobHistoryServer、httpfs
/etc/init.d/hadoop-yarn-proxyserver start /etc/init.d/hadoop-mapreduce-historyserver start /etc/init.d/hadoop-httpfs start
j、在全部机器上启动nodemanager
/etc/init.d/hadoop-yarn-nodemanager restart
4、功能验证
一、hadoop功能
a、查看hdfs根目录
[root@ip-10-10-103-246 ~]# hadoop fs -ls / Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 Found 3 items drwxr-xr-x - hdfs hdfs 0 2017-05-11 11:40 /tmp drwxrwx--- - mapred hdfs 0 2017-05-11 11:28 /user drwxr-xr-x - yarn hdfs 0 2017-05-11 11:28 /var
b、上传一个文件到根目录
[root@ip-10-10-103-246 ~]# hadoop fs -put /tmp/test.txt / Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 [root@ip-10-10-103-246 ~]# hadoop fs -ls / Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 Found 4 items -rw-r--r-- 2 root hdfs 22 2017-05-11 15:47 /test.txt drwxr-xr-x - hdfs hdfs 0 2017-05-11 11:40 /tmp drwxrwx--- - mapred hdfs 0 2017-05-11 11:28 /user drwxr-xr-x - yarn hdfs 0 2017-05-11 11:28 /var
c、直接删除文件不放回收站
[root@ip-10-10-103-246 ~]# hadoop fs -rm -skipTrash /test.txt Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 Deleted /test.txt
d、跑一个wordcount用例
[root@ip-10-10-103-246 ~]# hadoop fs -put /tmp/test.txt /user/hdfs/rand/ Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 [root@ip-10-10-103-246 conf]# sudo -u hdfs hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.11.0.jar wordcount /user/hdfs/rand/ /tmp OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 17/05/11 11:40:08 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to 10.10.103.246 17/05/11 11:40:09 INFO input.FileInputFormat: Total input paths to process : 1 17/05/11 11:40:09 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 17/05/11 11:40:09 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 674c65bbf0f779edc3e00a00c953b121f1988fe1] 17/05/11 11:40:09 INFO mapreduce.JobSubmitter: number of splits:1 17/05/11 11:40:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1494472050574_0003 17/05/11 11:40:09 INFO impl.YarnClientImpl: Submitted application application_1494472050574_0003 17/05/11 11:40:09 INFO mapreduce.Job: The url to track the job: http://10.10.103.246:8100/proxy/application_1494472050574_0003/ 17/05/11 11:40:09 INFO mapreduce.Job: Running job: job_1494472050574_0003 17/05/11 11:40:15 INFO mapreduce.Job: Job job_1494472050574_0003 running in uber mode : false 17/05/11 11:40:15 INFO mapreduce.Job: map 0% reduce 0% 17/05/11 11:40:20 INFO mapreduce.Job: map 100% reduce 0% 17/05/11 11:40:25 INFO mapreduce.Job: map 100% reduce 100% 17/05/11 11:40:25 INFO mapreduce.Job: Job job_1494472050574_0003 completed successfully 17/05/11 11:40:25 INFO mapreduce.Job: Counters: 53 File System Counters FILE: Number of bytes read=1897 FILE: Number of bytes written=262703 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=6431 HDFS: Number of bytes written=6219 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=2592 Total time spent by all reduces in occupied slots (ms)=5360 Total time spent by all map tasks (ms)=2592 Total time spent by all reduce tasks (ms)=2680 Total vcore-milliseconds taken by all map tasks=2592 Total vcore-milliseconds taken by all reduce tasks=2680 Total megabyte-milliseconds taken by all map tasks=3981312 Total megabyte-milliseconds taken by all reduce tasks=8232960 Map-Reduce Framework Map input records=102 Map output records=96 Map output bytes=6586 Map output materialized bytes=1893 Input split bytes=110 Combine input records=96 Combine output records=82 Reduce input groups=82 Reduce shuffle bytes=1893 Reduce input records=82 Reduce output records=82 Spilled Records=164 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=120 CPU time spent (ms)=1570 Physical memory (bytes) snapshot=501379072 Virtual memory (bytes) snapshot=7842639872 Total committed heap usage (bytes)=525860864 Peak Map Physical memory (bytes)=300183552 Peak Map Virtual memory (bytes)=3244224512 Peak Reduce Physical memory (bytes)=201195520 Peak Reduce Virtual memory (bytes)=4598415360 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=6321 File Output Format Counters Bytes Written=6219 [root@ip-10-10-103-246 conf]#
二、namenode高可用验证
查看http://10.10.103.246:50070
查看http://10.10.103.144:50070
停掉10.10.103.246节点的namenode进程，查看10.10.103.144节点是否会提高为active节点
三、resourcemanager高可用验证
查看http://10.10.103.246:8088
查看http://10.10.103.144:8088
在浏览器输入http://10.10.103.144:8088，会跳转到http://ip-10-10-103-246.ec2.internal:8088/，ip-10-10-103-246.ec2.internal是10.10.103.246的主机名，说明resourcemanager高可用配置ok，停掉10.10.103.144的
resourcemanager进程，在浏览器输入http://10.10.103.144:8088，就不会在跳转了，说明10.10.103.144已经切成了master。
5、总结
一、hadoop集群能成本部署完成，这才是开始，后期的维护，业务方问题的解决这些经验须要一点一点积累，多出差多折腾老是好的。
二、对应上面部署的集群后期须要扩容，直接把10.10.103.62这台机器作个镜像，用镜像启动服务器便可，服务会自动启动而且加入到集群
三、云上hadoop集群的成本优化，这里只针对aws而言
a、冷数据存在在s3上，hdfs能够直接支持s3，在hdfs-site.xml里面添加s3的key参数(fs.s3n.awsAccessKeyId和fs.s3n.awsSecretAccessKey)便可，须要注意的是程序上传、下载的逻辑须要多加几个重试机制，s3有时候不稳定会致使上传或者下载不成功
b、使用Auto Scaling服务结合竞价实例，配置扩展策略，好比当cpu大于50%的时候就扩容5台服务器，当cpu小于10%的时候就缩容5台服务器，固然你能够配置更多阶梯级的扩容、缩容策略，Auto Scaling还有一个计划任务的功能，你能够向设置crontab同样设置，让Auto Scaling帮你扩容、缩容服务器