kubernetes组件开发-自定义调度器（三）

isekiro 收录于 Kubernetes

2023-01-23 约 1581 字预计阅读 4 分钟

部署自定义调度器.

部署方式

scheduler-plugins 提供了两种安装调度器插件组件的方法: 作为第二个调度器和作为默认调度器。

他们的优点和缺点如下:

第二个调度器

优点是容易部署，很容易安装。

缺点是运行多个调度器的时候，在集群资源短缺时不可避免地会遇到资源冲突，因此不推荐在生产环境中使用。不过这种方式很适合用来当作调度框架插件的练习和开发。
默认调度程序（推荐）

优点是您将使用一个统一的调度程序，从而避免了资源冲突引起的调度问题。在生产环境中推荐使用。

缺点是需要在控制平面上操作，另外，安装起来也比较麻烦。

编译打包

进入 scheduler-plugins/cmd/scheduler 目录

执行：go build -ldflags '-s -w' -o bin/kube-scheduler ./main.go

Dockerfile ，打包成 docker 镜像，并推送至远程仓库。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10



FROM alpine:3.12

COPY ./kube-scheduler /bin/kube-scheduler

RUN chmod +x /bin/kube-scheduler

WORKDIR /bin

CMD ["kube-scheduler"]

第二调度器

作为第二个调度器部署到集群。

准备好 rbac 权限配置：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85



apiVersion: v1
kind: ServiceAccount
metadata:
  name: scheduler-plugins-scheduler
  namespace: scheduler-plugins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: scheduler-plugins-scheduler
rules:
- apiGroups: ["", "events.k8s.io"]
  resources: ["events"]
  verbs: ["create", "patch", "update"]
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["create"]
- apiGroups: ["coordination.k8s.io"]
  resourceNames: ["kube-scheduler"]
  resources: ["leases"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["create"]
- apiGroups: [""]
  resourceNames: ["kube-scheduler"]
  resources: ["endpoints"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["delete", "get", "list", "watch"]
- apiGroups: [""]
  resources: ["bindings", "pods/binding"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods/status"]
  verbs: ["patch", "update"]
- apiGroups: [""]
  resources: ["replicationcontrollers", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps", "extensions"]
  resources: ["replicasets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["statefulsets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims", "persistentvolumes"]
  verbs: ["get", "list", "watch", "patch", "update"]
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
- apiGroups: ["authorization.k8s.io"]
  resources: ["subjectaccessreviews"]
  verbs: ["create"]
- apiGroups: ["storage.k8s.io"]
  resources: ["csinodes", "storageclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["scheduling.sigs.k8s.io"]
  resources: ["podgroups", "elasticquotas"]
  verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: scheduler-plugins-scheduler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: scheduler-plugins-scheduler
subjects:
- kind: ServiceAccount
  name: scheduler-plugins-scheduler
  namespace: scheduler-plugins

准备好 scheduler-plugins 用的 configmap ：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18



apiVersion: v1
kind: ConfigMap
metadata:
  name: scheduler-config
  namespace: scheduler-plugins
data:
  scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1beta1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: false
    profiles:
    - schedulerName: scheduler-plugins-scheduler
      plugins:
        score:
          enabled:
          - name: PodCapacityOfNode    

准备好 Deployment 配置文件：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52



apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: scheduler
  name: scheduler-plugins-scheduler
  namespace: scheduler-plugins
spec:
  selector:
    matchLabels:
      component: scheduler
  replicas: 1
  template:
    metadata:
      labels:
        component: scheduler
    spec:
      serviceAccountName: scheduler-plugins-scheduler
      containers:
      - command:
        - /bin/kube-scheduler
        - --address=0.0.0.0
        - --leader-elect=false
        - --config=/etc/kubernetes/scheduler-config.yaml
        - --scheduler-name=scheduler-plugins-scheduler
        image: xxx/kube-scheduler:v0.20.10
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10251
          initialDelaySeconds: 15
        name: scheduler-plugins-scheduler
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10251
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts:
        - name: scheduler-config
          mountPath: /etc/kubernetes
          readOnly: true
      hostNetwork: false
      hostPID: false
      volumes:
      - name: scheduler-config
        configMap:
          name: scheduler-config

使用 kubectl 将 yaml 配置文件应用到 kubernetes 集群内。

本次实验环境

3 个 worker 节点
node1 21 个 pod
node2 17 个 pod
node3 25 个 pod

/kubernetes%E7%BB%84%E4%BB%B6%E5%BC%80%E5%8F%91-%E8%87%AA%E5%AE%9A%E4%B9%89%E8%B0%83%E5%BA%A6%E5%99%A8%E4%B8%89/kube-scheduler-plugins-logs.png — 调度日志

根据日志可以看出，worker 节点上面的 Pods 个数影响到此次调度，使 Pod 被调度到 node2 节点上。

默认调度器

使用这种模式，我们不仅不会丢失 kubernetes 本身默认的调度能力，还能额外获得我们自己插件的调度能力。¹

登录到控制平面节点，即 master 节点。
备份文件 kube-scheduler.yaml

cp /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/kube-scheduler.yaml
创建文件 /etc/kubernetes/sched-cc.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14



apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
leaderElection:
  # (Optional) Change true to false if you are not running a HA control-plane.
  leaderElect: true
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
profiles:
- schedulerName: default-scheduler
  plugins:
    score:
      enabled:
      - name: PodCapacityOfNode

如果我们要启用社区 scheduler-plugins 自带的一些插件，需要安装他们提供的 CRD 资源和控制器，根据启用功能的不同，安装不同的 CRD 资源。文件位于 manifests 目录。我们这里没有用到，只用我们自己的调度插件，暂不做此步骤。
修改 /etc/kubernetes/manifests/kube-scheduler.yaml

我们需要做以下变更:
- 新增 –config 参数指定 scheduler-config 配置文件
- (可选) 移除 - –kubeconfig=/etc/kubernetes/scheduler.conf 配置文件，如果不是高可用集群，移除 - –leader-elect=true
- 使用自己的 scheduler-plugin 镜像替换掉默认的镜像
- 挂载 scheduler-config 文件

< 新增， > 移除

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19



16d15
<     - --config=/etc/kubernetes/sched-cc.yaml
17a17,18
>     - --kubeconfig=/etc/kubernetes/scheduler.conf
>     - --leader-elect=true
19,20c20
<     image: k8s.gcr.io/scheduler-plugins/kube-scheduler:v0.19.9
---
>     image: k8s.gcr.io/kube-scheduler:v1.19.9
50,52d49
<     - mountPath: /etc/kubernetes/sched-cc.yaml
<       name: sched-cc
<       readOnly: true
60,63d56
<   - hostPath:
<       path: /etc/kubernetes/sched-cc.yaml
<       type: FileOrCreate
<     name: sched-cc

验证 kube-scheduler pod 有没有正确重启并是否处于 running 状态。

验证 Pod 调度：

/kubernetes%E7%BB%84%E4%BB%B6%E5%BC%80%E5%8F%91-%E8%87%AA%E5%AE%9A%E4%B9%89%E8%B0%83%E5%BA%A6%E5%99%A8%E4%B8%89/kube-scheduler-plugins-logs-single.png — 调度日志

1
2
3
4
5
6
7
8


......
I0123 14:08:00.061224       1 pod_capacity_of_node.go:78] pod busybox-687cf69c47-dtt59 node ops-k8s-node2 score is 84                                                                  
I0123 14:08:00.085898       1 pod_capacity_of_node.go:78] pod busybox-687cf69c47-dtt59 node ops-k8s-node3 score is 77                                                                  
I0123 14:08:00.087837       1 pod_capacity_of_node.go:78] pod busybox-687cf69c47-dtt59 node ops-k8s-node1 score is 80                                                                  
I0123 14:08:00.088682       1 default_binder.go:51] Attempting to bind default/busybox-687cf69c47-dtt59 to ops-k8s-node2                                                               
I0123 14:08:00.101755       1 eventhandlers.go:209] delete event for unscheduled pod default/busybox-687cf69c47-dtt59                                                                  
I0123 14:08:00.101817       1 eventhandlers.go:229] add event for scheduled pod default/busybox-687cf69c47-dtt59                                                                       
I0123 14:08:00.102414       1 scheduler.go:604] "Successfully bound pod to node" pod="default/busybox-687cf69c47-dtt59" node="ops-k8s-node2" evaluatedNodes=3 feasibleNodes=3 

As a single scheduler(replacing the vanilla default-scheduler)：https://github.com/kubernetes-sigs/scheduler-plugins/blob/release-1.20/doc/install.md#as-a-single-schedulerreplacing-the-vanilla-default-scheduler ↩︎