k8s调度器

导语 | kubernetes调度器，通过watch机制来发现集群中新创建且未调度的pod，通过过滤node列表，打分策略，以及各个时机的插件调用机制，选择合适的node与之绑定。

一、调度队列

二、单次调度

注意：一个集群中可以有多个调度器，所以首先需要根据pod中的spec参数获取调度器名称

• 对于小规模集群，node数小于100， 遍历所有node。
• 对于大规模集群，node数大于100，且配置的百分比小于100%时：按照node数量的一定百分比遍历，区间范围是[5，100]。
• 计算公式是

三、调度过程

// node快照
if err := sched.Cache.UpdateSnapshot(sched.nodeInfoSnapshot); err != nil {
  return result, err
}
// 过滤
feasibleNodes, diagnosis, err := sched.findNodesThatFitPod(ctx, fwk, state, pod)
if err != nil {
  return result, err
}
// 打分
priorityList, err := prioritizeNodes(ctx, sched.Extenders, fwk, state, pod, feasibleNodes)
if err != nil {
  return result, err
}
// 随机筛选
host, err := selectHost(priorityList)

（一）过滤

• 插件过滤：<font style="color:rgb(34, 34, 34);">RunPreFilterPlugins</font>如果插件执行失败，那么返回所有node可用，如果插件返回不可调度，则返回失败，终止本次调度。
• 获取<font style="color:rgb(34, 34, 34);">node allNodes, err := sched.nodeInfoSnapshot.NodeInfos().列表()</font>
• 抢占式pod的status字段中<font style="color:rgb(34, 34, 34);">NominatedNodeName</font>设置后，会优先抢占同名的node。
• <font style="color:rgb(34, 34, 34);">allNode</font>和<font style="color:rgb(34, 34, 34);">preFilter</font>返回的node求交集。

（二）打分

四、插件机制

（一）插件类型

// QueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.
QueueSort PluginSet `json:"queueSort,omitempty"`
// PreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.
PreFilter PluginSet `json:"preFilter,omitempty"`
// Filter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.
Filter PluginSet `json:"filter,omitempty"`
// PostFilter is a list of plugins that are invoked after filtering phase, but only when no feasible nodes were found for the pod.
PostFilter PluginSet `json:"postFilter,omitempty"`
// PreScore is a list of plugins that are invoked before scoring.
PreScore PluginSet `json:"preScore,omitempty"`
// Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.
Score PluginSet `json:"score,omitempty"`
// Reserve is a list of plugins invoked when reserving/unreserving resources
// after a node is assigned to run the pod.
Reserve PluginSet `json:"reserve,omitempty"`
// Permit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.
Permit PluginSet `json:"permit,omitempty"`
// PreBind is a list of plugins that should be invoked before a pod is bound.
PreBind PluginSet `json:"preBind,omitempty"`
// Bind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework.
// The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.
Bind PluginSet `json:"bind,omitempty"`
// PostBind is a list of plugins that should be invoked after a pod is successfully bound.
PostBind PluginSet `json:"postBind,omitempty"`
// MultiPoint is a simplified config section to enable plugins for all valid extension points.
MultiPoint PluginSet `json:"multiPoint,omitempty"`

（二）插件列表

"PrioritySort" : 
"DefaultBinder"
"DefaultPreemption"
"ImageLocality"
"InterPodAffinity"
"NodeAffinity"
"NodeName"
"NodePorts"
"NodeResourcesBalancedAllocation"
"NodeResourcesFit"
"NodeUnschedulable"
"NodeVolumeLimits"
"AzureDiskLimits"
"CinderLimits"
"EBSLimits"
"GCEPDLimits"
"PodTopologySpread"
"SelectorSpread"
"ServiceAffinity"
"TaintToleration"
"VolumeBinding"
"VolumeRestrictions"
"VolumeZone"

type ScorePlugin interface {
    Plugin
    // Score is called on each filtered node. It must return success and an integer
    // indicating the rank of the node. All scoring plugins must return success or
    // the pod will be rejected.
    Score(ctx context.Context, state *CycleState, p *v1.Pod, nodeName string) (int64, *Status)
    // ScoreExtensions returns a ScoreExtensions interface if it implements one, or nil if does not.
    ScoreExtensions() ScoreExtensions
}

（三）插件特性

图像定位

<font style="color:rgb(34, 34, 34);">ImageLocality</font>：本地镜像打分插件，计算分数规则如下：

注意：这里不是指单个containner，而是一个pod中的所有container的打分之和。为什么范围是23mb到1000mb？可以想一想。

节点关联

<font style="color:rgb(34, 34, 34);">NodeAffinity</font>：node亲和性和反亲和性，提供了两种策略配置。

• 对于必选策略<font style="color:rgb(34, 34, 34);">RequiredDuringSchedulingIgnoredDuringExecution</font>，如果匹配未成功，则在<font style="color:rgb(34, 34, 34);">PreFilter</font>阶段返回失败，终止调度。
• 对于首先策略<font style="color:rgb(34, 34, 34);">PreferredDuringSchedulingIgnoredDuringExecution</font>，如果匹配未成功，则尝试其他node也失败，调度器仍然会调度改pod。亲和性，反亲和性是用来影响打分数值（正负分）weight

// 亲和性，反亲和性判定判定
if hasPreferredAffinityConstraints || hasPreferredAntiAffinityConstraints {
    for _, existingPod := range podsToProcess {
      pl.processExistingPod(state, existingPod, nodeInfo, pod, topoScore)
    }
  topoScores[atomic.AddInt32(&index, 1)] = topoScore
}

污点耐受

<font style="color:rgb(34, 34, 34);">TaintToleration</font>：污点插件，提供了过滤，预打分，打分，打分标准化（平行扩展到0到100）接口。

// 尽可能不调度
TaintEffectPreferNoSchedule TaintEffect = "PreferNoSchedule"    
// 一定不调度
TaintEffectNoSchedule TaintEffect = "NoSchedule"
// 一定不调度且驱逐
TaintEffectNoExecute TaintEffect = "NoExecute"

五、调度器配置

{
  "name": "BalancedResourceAllocation",
  "weight": 1
},
{
  "name": "EvenPodsSpreadPriority",
  "weight": 1
},
{
  "name": "InterPodAffinityPriority",
  "weight": 1
},
{
  "name": "LeastRequestedPriority",
  "weight": 1
},
{
  "name": "NodeAffinityPriority",
  "weight": 1
},
{
  "name": "NodePreferAvoidPodsPriority",
  "weight": 10000
},
{
  "name": "SelectorSpreadPriority",
  "weight": 1
},
{
  "name": "TaintTolerationPriority",
  "weight": 1
}

六、如何自定义pod调度

（一）扩展模式

"extenders": [{
    "urlPrefix": "http://xxx/prefix",
    "filterVerb": "filter",
    "weight": 1,
    "bindVerb": "bind",
    "enableHttps": false
}]

（二）多调度器

import (
    scheduler "k8s.io/kubernetes/cmd/kube-scheduler/app"
)
func main() {
    command := scheduler.NewSchedulerCommand(
            scheduler.WithPlugin("my-plugin", MyPlugin))
    if err := command.Execute(); err != nil {
        fmt.Fprintf(os.Stderr, "%v\n", err)
    }
}

七、总结