1. 项目背景与核心需求在Kubernetes集群中管理GPU资源一直是个颇具挑战性的任务。传统方式直接使用nvidia.com/gpu资源分配存在诸多限制无法实现细粒度共享、缺乏灵活的调度策略、资源利用率低下。而vGPU技术能够将物理GPU切分为多个虚拟GPU显著提升资源利用率。本项目通过开发一个Kubernetes Webhook实现了以下核心功能自动将Pod声明的nvidia.com/gpu资源转换为Volcano调度器支持的vGPU资源格式动态设置vGPU内存大小支持按模型大小预设配置强制使用Volcano调度器以获得更好的GPU调度能力保持资源请求与限制的完整性避免Kubernetes校验失败2. 技术架构解析2.1 整体工作流程整个系统的工作流程可分为四个关键阶段证书准备阶段使用cert-manager创建自签名CA为Webhook服务签发TLS证书Webhook部署阶段包含Python处理逻辑的Deployment、对外服务的Service、配置存储的ConfigMap注册阶段通过MutatingWebhookConfiguration注册我们的Webhook到Kubernetes API运行时阶段当用户创建Pod时API Server会调用我们的Webhook进行资源转换2.2 核心组件交互---------------- --------------- ----------------- | kubectl | | API Server | | Webhook Server | | apply pod ------- (k8s) ------- (我们的服务) | ---------------- -------------- ---------------- ^ | | | | v | ---------------- --------------- cert-manager | | (证书管理) | -----------------3. 证书管理实现3.1 cert-manager部署首先需要部署cert-manager来处理证书生命周期kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml3.2 创建自签名ClusterIssuer创建自签名证书颁发机构CAapiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned spec: selfSigned: {}3.3 签发Webhook TLS证书为Webhook服务创建证书资源apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: volcano-injector-cert namespace: gpustack spec: secretName: volcano-injector-tls dnsNames: - volcano-injector.gpustack.svc issuerRef: name: selfsigned kind: ClusterIssuer关键点证书必须包含Webhook Service的完整DNS名称service.namespace.svc4. Webhook服务实现4.1 核心处理逻辑Webhook的核心Python代码实现了以下关键功能def mutate(): # 1. 解析请求 req request.get_json() request_obj req.get(request, {}) pod request_obj.get(object, {}) # 2. 初始化patch列表 patch [] # 3. 强制使用Volcano调度器 if spec.get(schedulerName) ! volcano: add_op(patch, /spec/schedulerName, volcano) # 4. 处理每个容器 for i, container in enumerate(containers): base_path f/spec/containers/{i}/resources # 5. GPU→vGPU转换 if nvidia.com/gpu in limits: # 先删除原GPU请求 remove_op(patch, f{base_path}/limits/nvidia.com~1gpu) remove_op(patch, f{base_path}/requests/nvidia.com~1gpu) # 添加vGPU配置 add_op(patch, f{base_path}/limits/volcano.sh~1vgpu-number, gpu) add_op(patch, f{base_path}/limits/volcano.sh~1vgpu-memory, vgpu_memory) # ...同样处理requests... # 确保有CPU和内存配置 add_op(patch, f{base_path}/limits/cpu, DEFAULT_CPU) add_op(patch, f{base_path}/requests/cpu, DEFAULT_CPU) # ...内存处理类似... # 6. 返回patch return jsonify({ response: { allowed: True, patchType: JSONPatch, patch: base64.b64encode(json.dumps(patch).encode()).decode() } })4.2 关键设计决策操作顺序至关重要必须先删除原GPU配置然后添加vGPU配置最后补充CPU/内存等必填字段默认值设计预设了不同模型大小的vGPU内存配置为CPU和内存提供了合理的默认值错误处理捕获所有异常并返回允许创建详细日志记录便于排查问题5. 部署Webhook服务5.1 将代码打包为ConfigMapkubectl create configmap volcano-injector \ --from-filewebhook.pywebhook.py \ --namespacegpustack \ --dry-runclient -o yaml configmap-py.yaml5.2 构建自定义镜像Dockerfile关键内容FROM python:3.9-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txtrequirements.txt包含的依赖Flask2.3.3 Werkzeug2.3.7 jsonpatch1.335.3 部署Webhook DeploymentapiVersion: apps/v1 kind: Deployment metadata: name: volcano-injector namespace: gpustack spec: replicas: 1 template: spec: containers: - name: injector image: your-registry/python:v4 command: [python, /app/webhook.py] volumeMounts: - name: config mountPath: /app - name: tls mountPath: /tls volumes: - name: config configMap: name: volcano-injector - name: tls secret: secretName: volcano-injector-tls5.4 创建ServiceapiVersion: v1 kind: Service metadata: name: volcano-injector namespace: gpustack spec: selector: app: injector ports: - port: 443 targetPort: 443 type: ClusterIP6. 注册Webhook配置6.1 MutatingWebhookConfigurationapiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: volcano-injector webhooks: - name: volcano.injector.k8s.io admissionReviewVersions: [v1] sideEffects: None failurePolicy: Fail namespaceSelector: matchExpressions: - key: kubernetes.io/metadata.name operator: NotIn values: [kube-system, cert-manager, gpustack] clientConfig: service: name: volcano-injector namespace: gpustack path: /mutate port: 443 caBundle: base64编码的CA证书 rules: - operations: [CREATE] apiGroups: [] apiVersions: [v1] resources: [pods]6.2 关键配置说明namespaceSelector排除了系统命名空间避免对Webhook自身Pod的递归调用failurePolicy设置为Fail确保转换失败时Pod创建会被拒绝生产环境建议从Ignore开始caBundle必须包含验证Webhook服务器证书的CA证书可通过kubectl get secret volcano-injector-tls -n gpustack -o jsonpath{.data.ca\.crt}获取7. 测试与验证7.1 测试Pod定义apiVersion: v1 kind: Pod metadata: name: gpustack-test spec: containers: - name: test image: nvidia/cuda:12.2.0-base-ubuntu22.04 command: [bash, -c, nvidia-smi sleep 3600] resources: limits: nvidia.com/gpu: 17.2 验证转换结果查看最终Pod配置kubectl get pod gpustack-test -o yaml预期看到schedulerName设置为volcanonvidia.com/gpu被移除替换为volcano.sh/vgpu-number和volcano.sh/vgpu-memory自动添加了CPU和内存资源7.3 查看Webhook日志kubectl logs -n gpustack deployment/volcano-injector -f正常日志应显示接收到的Pod名称执行的转换操作生成的完整patch内容8. 常见问题与解决方案8.1 证书相关问题问题现象Error from server (InternalError): failed calling webhook: Post https://...: x509: certificate signed by unknown authority解决方案确保证书Secret已正确创建检查MutatingWebhookConfiguration中的caBundle是否正确验证证书的DNS名称匹配Service地址8.2 Webhook调用失败问题现象failed calling webhook: service volcano-injector not found排查步骤检查Service是否存在且selector匹配Pod标签确认Service和Webhook配置在同一个namespace验证网络策略是否允许API Server访问Webhook Pod8.3 资源校验失败问题现象The Pod is invalid: spec.containers[0].resources.limits: Required value原因分析Webhook没有正确补充CPU/内存资源操作顺序不当导致校验失败修复方法确保在移除GPU后立即添加vGPU配置最后才补充CPU/内存等必填字段检查Python代码中的操作顺序9. 性能优化建议增加就绪探针readinessProbe: httpGet: path: /healthz port: 443 scheme: HTTPS实现缓存机制缓存模型大小到vGPU内存的映射减少每次请求的解析时间优化日志级别生产环境降低默认日志级别关键操作保留INFO级别横向扩展根据负载增加Deployment副本数考虑使用HPA自动扩缩容10. 安全加固措施细粒度访问控制通过namespaceSelector和objectSelector限制Webhook作用范围避免处理系统关键Pod证书轮换设置cert-manager自动轮换证书监控证书过期时间请求验证验证请求来源IP范围检查User-Agent是否为Kubernetes API Server资源限制resources: limits: cpu: 500m memory: 256Mi11. 项目扩展方向支持动态vGPU配置根据Pod注解动态设置vGPU内存支持百分比形式的显存分配多GPU卡支持识别不同型号GPU卡提供异构GPU分配能力监控集成暴露Prometheus指标记录转换统计信息策略引擎基于资源使用情况自动调整vGPU大小实现超额订阅策略这个Webhook实现已经过生产环境验证能够稳定处理每秒数十个Pod创建请求。关键在于确保证书配置正确和操作顺序合理。实际部署时建议先从少量命名空间开始试用逐步扩大范围。