Hadoop Yarn Capacity Scheduler

时间 2019-11-11 标签 hadoop yarn capacity scheduler

Capacity 调度器配置apache

<property>
  <name>yarn.resourcemanager.scheduler.class<name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

加载配置文件app

$yarn rmadmin -refreshQueues

Root --- QueueA --- A1, A2
|-- QueueB --- B1, B2
|-- QueueC --- C1, C2oop

队列spa

<property>
  <name>yarn.scheduler.capacity.root.queues<name>
  <value>QueueA, QueueB, QueueC</value>
</property>

<property>
  <name>yarn.scheduler.capacity.QueueA.queues<name>
  <value>A1, A2</value>
</property>

队列访问权限控制code

<property>
  <name>yarn.scheduler.capacity.root.QueueA.acl_submit_applications<name>
  <value>u1, u2 g1</value>
  <description>限定哪些Linux用户/用户组可向给定队列中提交应用程序</description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.QueueA.acl_administer_queue<name>
  <value>g1</value>
  <description>为队列指定一个管理员，该管理员可控制该队列的全部应用程序，好比杀死任意一个应用程序等</description>
</property>

层级队列Capacity管理server

<property>
  <name>yarn.scheduler.capacity.root.QueueA.capacity<name>
  <value>60</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.QueueB.capacity<name>
  <value>10</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.QueueC.capacity<name>
  <value>30</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.QueueA.A1.capacity<name>
  <value>20</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.QueueA.A2.capacity<name>
  <value>80</value>
</property>

队列A1不能超过父队列queueA的40%的资源。blog

<property>
  <name>yarn.scheduler.capacity.root.QueueA.A1.maximum-capacity<name>
  <value>40</value>
</property>

队列的状态队列

<property>
  <name>yarn.scheduler.capacity.root.QueueB.state<name>
  <value>RUNNING</value>
</property>

应用程序限制ip

<property>
  <name>yarn.scheduler.capacity.maximum-applications<name>
  <value>10000</value>
  <description>集群同时运行的最大做业数</description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.QueueA.maximum-applications<name>
  <value>10000</value>
  <description>队列root.QueueA同时运行的最大做业数</description>
</property>

<property>
  <name>yarn.scheduler.capacity.maximum-am-resource-percent<name>
  <value>0.1</value>
  <description>AM所占资源最大比例，用来控制同时运行的做业数</description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.QueueA.maximum-am-resource-percent<name>
  <value>0.1</value>
  <description>AM所占资源最大比例，用来控制队列root.QueueA同时运行的做业数</description>
</property>