基于 IBM BigInsights 的 Hadoop 多租户实践

  yarn.scheduler.capacity.root.Marketing.acl_submit_applications=*

  yarn.scheduler.capacity.root.Marketing.Advertising.acl_administer_queue=*

  yarn.scheduler.capacity.root.Marketing.Advertising.acl_submit_applications=*

  yarn.scheduler.capacity.root.Marketing.Advertising.capacity=30

  yarn.scheduler.capacity.root.Marketing.Advertising.maximum-capacity=30

  yarn.scheduler.capacity.root.Marketing.Advertising.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Marketing.Advertising.ordering-policy=fifo

  yarn.scheduler.capacity.root.Marketing.Advertising.state=RUNNING

  yarn.scheduler.capacity.root.Marketing.Advertising.user-limit-factor=1

  yarn.scheduler.capacity.root.Marketing.capacity=40

  yarn.scheduler.capacity.root.Marketing.maximum-capacity=100

  yarn.scheduler.capacity.root.Marketing.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Marketing.ordering-policy=fifo

  yarn.scheduler.capacity.root.Marketing.queues=Advertising,Sales

  yarn.scheduler.capacity.root.Marketing.Sales.acl_administer_queue=*

  yarn.scheduler.capacity.root.Marketing.Sales.acl_submit_applications=*

  yarn.scheduler.capacity.root.Marketing.Sales.capacity=70

  yarn.scheduler.capacity.root.Marketing.Sales.maximum-capacity=70

  yarn.scheduler.capacity.root.Marketing.Sales.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Marketing.Sales.ordering-policy=fifo

  yarn.scheduler.capacity.root.Marketing.Sales.state=RUNNING

  yarn.scheduler.capacity.root.Marketing.Sales.user-limit-factor=1

  yarn.scheduler.capacity.root.Marketing.state=RUNNING

  yarn.scheduler.capacity.root.Marketing.user-limit-factor=1

  yarn.scheduler.capacity.root.queues=Engineering,Marketing,Support,default

  yarn.scheduler.capacity.root.Support.acl_administer_queue=*

  yarn.scheduler.capacity.root.Support.acl_submit_applications=*

  yarn.scheduler.capacity.root.Support.capacity=20

  yarn.scheduler.capacity.root.Support.maximum-capacity=24

  yarn.scheduler.capacity.root.Support.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Support.ordering-policy=fifo

  yarn.scheduler.capacity.root.Support.queues=Services,Training

  yarn.scheduler.capacity.root.Support.Services.acl_administer_queue=*

  yarn.scheduler.capacity.root.Support.Services.acl_submit_applications=*

  yarn.scheduler.capacity.root.Support.Services.capacity=80

  yarn.scheduler.capacity.root.Support.Services.maximum-capacity=80

  yarn.scheduler.capacity.root.Support.Services.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Support.Services.ordering-policy=fifo

  yarn.scheduler.capacity.root.Support.Services.state=RUNNING

  yarn.scheduler.capacity.root.Support.Services.user-limit-factor=1

  yarn.scheduler.capacity.root.Support.state=RUNNING

  yarn.scheduler.capacity.root.Support.Training.acl_administer_queue=*

  yarn.scheduler.capacity.root.Support.Training.acl_submit_applications=*

  yarn.scheduler.capacity.root.Support.Training.capacity=20

  yarn.scheduler.capacity.root.Support.Training.maximum-capacity=20

  yarn.scheduler.capacity.root.Support.Training.minimum-user-limit-percent=100

  yarn.scheduler.capacity.root.Support.Training.ordering-policy=fifo

  yarn.scheduler.capacity.root.Support.Training.state=RUNNING

  yarn.scheduler.capacity.root.Support.Training.user-limit-factor=1

  yarn.scheduler.capacity.root.Support.user-limit-factor=1

  多租户的验证

  下面由不同的用户 dev1、qa1、adv1、sal1、serv1、tr1 分别提交任务,在 ResourceManager UI 的 Scheduler 视图可以监控到任务被分配到不同的队列。

  用户 dev1 提交的作业分配到 Development 队列

  用户 qa1 提交的作业分配到 QA 队列

  用户 adv1 提交的作业分配到 Advertising 队列

  用户 sal1 提交的作业分配到 Sales 队列

  用户 serv1 提交的作业分配到 Services 队列

  用户 tr1 提交的作业分配到 Training 队列

  同时可以监控到队列资源的使用情况,绿色是占用的资源,黄色是已经超过 Capacity 属性的定义值,灰色是已经到达 Max Capacity 的设置。从视图可以看出 Engineering.Development 队列、Marketing.Advertising 队列和 Support.Training 队列目前已经超过 Capacity 属性的定义值,正在占用父队列的资源,但还没有到达 Max Capacity 属性的定义值。

  图 13. IBM BigInsights 多租户的验证

物联网