基于 IBM BigInsights 的 Hadoop 多租户实践

  从这个例子可以看到,用户不断提交任务,有可能锁定其它队列的资源,直到资源释放或获得抢占。为了避免这种情况,Capacity Scheduler 支持在队列设置弹性增长限制。例如,以限制 root.Engineering.Development 队列,管理员可以设置 Max Capacity 为 40%。一旦设置,root.Engineering.Development 队列的用户仍然可以超越其 1.8 GB 的容量,但他们不会被分配超过 40% 的 root.Engineering 父队列的能力,即 9 GB 的 40% = 3.6 GB。

  通过 Capacity 属性和 Max Capacity 属性的设置,可用于控制组织和子机构资源弹性共享。管理员应当平衡这些特性,以避免严格的限制导致降低集群利用率,也要避免过度的跨组织资源共享。

  队列的映射

  管理员可以定义一个默认的映射策略,指定用户提交的申请将被自动提交到队列。有了一个默认的映射策略,用户无需递交申请时指定的队列名称。默认映射策略可以被配置成如果所提交的应用程序指定的队列名被覆盖。

  队列映射是用逗号分隔的映射分配的列表中定义,队列映射分配可以为用户(使用 u)或一组用户(使用 g)来定义。

  图 12. 队列映射分配

物联网
物联网

  完成配置

  IBM BigInsights 通过图形化界面配置,产生了新的 Capacity Scheduler 配置,极大简化了管理,同时也避免了错误。保存配置后可以不重启服务,直接在 Capacity-Scheduler 视图 保存并刷新队列 Refresh Queues ResourceManager,当然也可以重新启动YARN 服务使配置生效。

  清单 2. Capacity Scheduler 修改后的配置

  yarn.scheduler.capacity.maximum-am-resource-percent=0.2

  yarn.scheduler.capacity.maximum-applications=10000

  yarn.scheduler.capacity.node-locality-delay=40

  yarn.scheduler.capacity.queue-mappings=u:dev1:Development,u:qa1:QA,u:adv1:Advertising,u:sal1:Sales,u:serv1:Services,u:tr1:Training

  yarn.scheduler.capacity.queue-mappings-override.enable=false

  yarn.scheduler.capacity.root.accessible-node-labels=*

  yarn.scheduler.capacity.root.acl_administer_queue=*

  yarn.scheduler.capacity.root.capacity=100

  yarn.scheduler.capacity.root.default.acl_submit_applications=*

  yarn.scheduler.capacity.root.default.capacity=10

  yarn.scheduler.capacity.root.default.maximum-capacity=10

  yarn.scheduler.capacity.root.default.state=RUNNING

  yarn.scheduler.capacity.root.default.user-limit-factor=1

  yarn.scheduler.capacity.root.Engineering.acl_administer_queue=*

  yarn.scheduler.capacity.root.Engineering.acl_submit_applications=*

  yarn.scheduler.capacity.root.Engineering.capacity=30

  yarn.scheduler.capacity.root.Engineering.Development.acl_administer_queue=*

  yarn.scheduler.capacity.root.Engineering.Development.acl_submit_applications=*

  yarn.scheduler.capacity.root.Engineering.Development.capacity=20

  yarn.scheduler.capacity.root.Engineering.Development.maximum-capacity=100

  yarn.scheduler.capacity.root.Engineering.Development.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Engineering.Development.ordering-policy=fifo

  yarn.scheduler.capacity.root.Engineering.Development.state=RUNNING

  yarn.scheduler.capacity.root.Engineering.Development.user-limit-factor=1

  yarn.scheduler.capacity.root.Engineering.maximum-capacity=30

  yarn.scheduler.capacity.root.Engineering.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Engineering.ordering-policy=fifo

  yarn.scheduler.capacity.root.Engineering.QA.acl_administer_queue=*

  yarn.scheduler.capacity.root.Engineering.QA.acl_submit_applications=*

  yarn.scheduler.capacity.root.Engineering.QA.capacity=80

  yarn.scheduler.capacity.root.Engineering.QA.maximum-capacity=80

  yarn.scheduler.capacity.root.Engineering.QA.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Engineering.QA.ordering-policy=fifo

  yarn.scheduler.capacity.root.Engineering.QA.state=RUNNING

  yarn.scheduler.capacity.root.Engineering.QA.user-limit-factor=1

  yarn.scheduler.capacity.root.Engineering.queues=Development,QA

  yarn.scheduler.capacity.root.Engineering.state=RUNNING

  yarn.scheduler.capacity.root.Engineering.user-limit-factor=1

  yarn.scheduler.capacity.root.Marketing.acl_administer_queue=*