[Close as Dup] display resource utility per vc/queue metrics

[xudifsd]

What would you like to be added:

A graph/list to show resource util per vc/queue

Why is this needed:

User sometime found their job is always in waiting state, this may be due to resource not enough in VC or resource fragment in node, need a graph to help them debug the issue

Without this feature, how does the current module work：

User check yarn's page manually.

Components that may involve changes:

• yarn-exporter
• grafana

[Anbang-Hu]

Is it possible to have metrics showing current resource fragmentation status (per VC)? We should manage users' expectation properly.

[xudifsd]

I'm not sure how to show resource fragmentation status, showing node count with certain free card number? How about memory? Like what hadoop showing disk?

But this is the whole cluster status, I think no vc view about this. Am I right? @mzmssg

[fanyangCS]

related. #2013

[fanyangCS]

@xudifsd , a high priority issue is to show how many jobs are in waiting state in a VC.

[scarlett2018]

This is fit for the "Virtual Cluster" Page refactoring. Currently, when click on the VC instance name it goes to the single VC page. The current single VC page only has the generic VC info. We can add this info above the current table.

VC Table

Single VC Page

[xudifsd]

@fanyangCS yes, yarn-exporter should get that metric. We will need to display that and node count with certain free gpu card, memory, cpu graph in some page, so user can get a sense why their job is waiting.

[mzmssg]

@fanyangCS

a high priority issue is to show how many jobs are in waiting state in a VC.

What's the definition of waiting status? AM launcher but no job container is regarded as RUNNING or WAITING?

[yqwang-ms]

Job Waiting: Job Not Complete and Not Exist Container is Running

[xudifsd]

As discussed in meeting, I think above picture may captured what user may find useful. It has three tables and one histogram. Meaning of each table:

• jobs pending: pending job count of each VC
• containers pending: pending container count of each VC
• free resources: resource available in each VC, this table has already shown in virtual cluster page right now
• free gpu histogram: show node count with certain free GPU, so user could know if certain task can fit in cluster, take the picture as example, tasks require 8 GPU will never be scheduled, since there are no nodes with 8 free GPU.

All required metrics has been exported by #2289

@scarlett2018 Maybe experience team can take over this job?

[fanyangCS]

#1943

[fanyangCS]

#1989

[scarlett2018]

@xudifsd - sure, but I don't think experience team has capacity to do this in 0.11.0.
cc @weixingzhang - do your team want to take this? or are there any comments around this design?

[scarlett2018]

Will address this in #2539