doc for drf fair share #193
Conversation
|
Related: volcano-retired/scheduler#14 |
|
@lminzhw why don't you add this into volcanosh/kube-batch repo? |
|
put every discussion here. only code change in kube-batch |
| ```go | ||
| type Session struct { | ||
| ... | ||
| NamespaceInfo map[string]*kbapi.NamespaceInfo |
There was a problem hiding this comment.
Prefer to UserInfo, similar to other place.
There was a problem hiding this comment.
Prefer to
UserInfo, similar to other place.
|
LGTM overall, let's wait for other's feedback on that. |
| volcano.sh/namespace.weight: 1 <- this field represent the weight of this namespace | ||
| ``` | ||
|
|
||
| If many `ResourceQuota` in the same namespace have weight, the weight for this namespace is the highest one of them. |
There was a problem hiding this comment.
I don't quite understand the assumption here. Do you mean if user create several ResourceQuota in this namespace, the one with highest weight wins?
There was a problem hiding this comment.
Sure. Just for corner case.
There was a problem hiding this comment.
@lminzhw need reword these words to make it more clear.
There was a problem hiding this comment.
The hierarchy is Queue -> PodGroup -> Pod now. And there is no namespace now. It means that we could create pods in the same namespace for different queues. How do we deal with it?
There was a problem hiding this comment.
It means that we could create pods in the same namespace for different queues
Yes, we support that; namespace is considered as a user, so we will balance the user's resources cross queue. If there's one queue per namespace, this feature will not take effect.
/cc @lminzhw
|
|
||
| ### Cases: | ||
|
|
||
| - cluster have __16 cpu__, queue and namespace have default weight. |
There was a problem hiding this comment.
When namespace requests more resources (calculated from pods) than it's resources quota(from v1.ResourceQuota). At the same time, this namespace has higher weight than the other namespaces in the same queue? Is it a possibility some resources allocated by volcano but failed resource quota check?
There was a problem hiding this comment.
For this case, the request of requiring more resources than it's resources quota will cause error like this
pods \"lm-fairshare-wnchr-tb-3\" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=500m, used: limits.cpu=2, limited: limits.cpu=2
So, if that pod is Pending and wait for allocation from volcano, it is already satisfied the resource quota.
|
|
||
| So, we need a more fine-grained strategy to balance resource usage among users in the same `Queue`. | ||
|
|
||
| In consideration of multi-user model in kubernetes, we use namespace to distinguish different user. Each namespace would have its weight to control resources usage. |
| spec: | ||
| hard: | ||
| limits.memory: 2Gi | ||
| volcano.sh/namespace.weight: 1 <- this field represent the weight of this namespace |
There was a problem hiding this comment.
# this field represent the weight of this namespace
volcano.sh/namespace.weight: 1| volcano.sh/namespace.weight: 1 <- this field represent the weight of this namespace | ||
| ``` | ||
|
|
||
| If many `ResourceQuota` in the same namespace have weight, the weight for this namespace is the highest one of them. |
There was a problem hiding this comment.
@lminzhw need reword these words to make it more clear.
|
|
||
| For namespace n1 having weight w1 and namespace n2 having weight w2, we can compute the `share` of resource and recorded as u1 and u2. Now, the resource usage of n1 less than n2 can be defined as (u1 / w1 < u2 / w2) | ||
|
|
||
| `e.g.` ns1 having weight w1=2 use 6cpu, ns2 having weight w2=1 use 2cpu. In the scope of cpu, the ns1 use less resource than ns2. (6 / 3 < 3 / 1) |
There was a problem hiding this comment.
How we get this equation:
6/3 < 3/1
| | q1 w1 | ns1 w3 | 5 cpu | 4 cpu | 3 cpu | | ||
| | | ns2 w1 | 10 cpu | | 1 cpu | | ||
| | q2 w3 | ns3 w2 | 10 cpu | 12 cpu | 10 cpu | | ||
| | | ns4 w6 | 2 cpu | | 2 cpu | |
There was a problem hiding this comment.
namespace4 gets more compared with namespace3 why?
There was a problem hiding this comment.
According to the design, ns3 should have only 3 * CPU and 9 * CPU for ns4, i think.
| volcano.sh/namespace.weight: 1 <- this field represent the weight of this namespace | ||
| ``` | ||
|
|
||
| If many `ResourceQuota` in the same namespace have weight, the weight for this namespace is the highest one of them. |
There was a problem hiding this comment.
The hierarchy is Queue -> PodGroup -> Pod now. And there is no namespace now. It means that we could create pods in the same namespace for different queues. How do we deal with it?
| apiVersion: v1 | ||
| kind: ResourceQuota | ||
| metadata: | ||
| namespace: default |
There was a problem hiding this comment.
Is ResourceQuota same as Kubernetes ResourceQuota? If not, could volcano.sh/namespace.weight be named to weight because the whole type is volcano.sh related?
Does it need a name? Then we could create different quotas of same namespace for different queues.
There was a problem hiding this comment.
it's Kubernetes ResourceQuota as far as I known :)
There was a problem hiding this comment.
If it is same, it is a little inappropriate to put volcano.sh/namespace.weight in .spec.hard, because hard is the set of desired hard limits for each named resource and weight is not a hard limit.
There was a problem hiding this comment.
hard is the set of desired hard limits for each named resource and weight is not a hard limit.
It's a kind of hard as scheduler will make sure namespace will not be overused.
There was a problem hiding this comment.
It is OK if it is a hard limit.
There was a problem hiding this comment.
It is OK if it is a hard limit.
And once the resource reached the limit, there will be no more pods created for the namespace.
|
LGTM overall 👍 , let's wait for other comments :) |
|
/lgtm |
volcano-sh-bot
added
approved
|
Get it merged firstly; let's open other PRs if any enhancements. |
doc for drf fair share
add doc.