1. 项目概述与架构设计在企业级容器化部署中Harbor作为镜像仓库的核心组件其高可用性和稳定性直接关系到整个Kubernetes集群的可靠性。本方案采用独立负载均衡架构通过HAProxyKeepalived实现服务入口的高可用结合Calico网络插件和kubeasz工具集构建生产级Harbor集群。这套架构已在多个金融和互联网企业生产环境验证可支撑日均百万级的镜像拉取请求。典型拓扑包含以下核心层负载均衡层双节点HAProxy集群通过VRRP协议实现VIP漂移计算节点层至少3个Worker节点组成的Kubernetes工作集群存储层MinIO分布式对象存储集群提供持久化数据存储网络层Calico BGP模式实现Pod网络与物理网络互通关键设计原则每个组件都需消除单点故障且故障域要相互隔离。例如负载均衡节点与Kubernetes工作节点部署在不同物理机上。2. 基础环境准备2.1 硬件配置要求节点类型CPU内存磁盘网络负载均衡节点4核8GB100GB系统盘双万兆网卡Master节点8核16GB100GB系统盘500GB数据盘双万兆网卡Worker节点16核32GB100GB系统盘1TB数据盘双万兆网卡MinIO存储节点12核32GB100GB系统盘4x4TB数据盘双万兆网卡2.2 系统初始化配置所有节点需统一进行以下基础配置# 禁用Swap swapoff -a sed -i / swap / s/^\(.*\)$/#\1/g /etc/fstab # 设置内核参数 cat /etc/sysctl.d/k8s.conf EOF net.ipv4.ip_forward 1 net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 vm.swappiness 0 EOF sysctl -p /etc/sysctl.d/k8s.conf # 配置时间同步 timedatectl set-timezone Asia/Shanghai systemctl enable --now chronyd3. 负载均衡层部署3.1 HAProxy双机部署主备节点(10.0.0.104/105)上安装配置# 安装软件包 yum install -y haproxy keepalived # 主节点配置 cat /etc/haproxy/haproxy.cfg EOF global log /dev/log local0 maxconn 4000 user haproxy group haproxy defaults mode http log global option httplog timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend k8s-api bind 10.0.0.112:6443 mode tcp default_backend k8s-api backend k8s-api mode tcp balance roundrobin option tcp-check server k8s-master1 10.0.0.101:6443 check inter 2000 fall 3 rise 2 server k8s-master2 10.0.0.102:6443 check inter 2000 fall 3 rise 2 server k8s-master3 10.0.0.103:6443 check inter 2000 fall 3 rise 2 EOF3.2 Keepalived配置主节点(10.0.0.104)配置cat /etc/keepalived/keepalived.conf EOF vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight 2 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.112/24 } track_script { chk_haproxy } } EOF备节点(10.0.0.105)只需修改state BACKUP priority 903.3 服务验证# 查看VIP归属 ip a show eth0 | grep 10.0.0.112 # 测试API访问 curl -k https://10.0.0.112:6443/version4. Kubernetes集群部署4.1 使用kubeasz快速部署# 下载kubeasz export release3.3.0 wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown chmod x ./ezdown # 部署集群 ./ezdown -D ./ezdown -S ./easzctl start # 验证节点状态 kubectl get nodes -o wide4.2 Calico网络配置修改kubeasz的roles/calico/templates/calico-v3.24.yamlapiVersion: crd.projectcalico.org/v1 kind: IPPool metadata: name: default-ipv4-ippool spec: blockSize: 24 cidr: 10.244.0.0/16 ipipMode: Never natOutgoing: true nodeSelector: all()应用配置kubectl apply -f /etc/kubeasz/clusters/default/config/calico-v3.24.yaml5. Harbor高可用部署5.1 准备MinIO存储# 创建Harbor专用bucket mc mb minio/harbor-registry mc policy set public minio/harbor-registry5.2 Helm安装Harborhelm repo add harbor https://helm.goharbor.io helm install harbor harbor/harbor \ --namespace harbor \ --set persistence.enabledtrue \ --set persistence.resourcePolicykeep \ --set externalURLhttps://harbor.example.com \ --set persistence.imageChartStorage.types3 \ --set persistence.imageChartStorage.s3.regionus-east-1 \ --set persistence.imageChartStorage.s3.bucketharbor-registry \ --set persistence.imageChartStorage.s3.accesskeyminioadmin \ --set persistence.imageChartStorage.s3.secretkeyminioadmin \ --set persistence.imageChartStorage.s3.regionendpointhttp://minio:90005.3 高可用验证# 模拟节点故障 kubectl drain k8s-worker1 --ignore-daemonsets # 检查服务连续性 while true; do curl -I https://harbor.example.com/api/v2.0/systeminfo; sleep 1; done6. 监控与运维6.1 Prometheus监控配置# harbor-service-monitor.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: harbor namespace: monitoring spec: endpoints: - port: metrics interval: 15s namespaceSelector: matchNames: - harbor selector: matchLabels: app: harbor6.2 备份策略# 每日全量备份 0 2 * * * velero backup create harbor-backup-$(date \%Y\%m\%d) \ --include-namespaces harbor --ttl 72h7. 性能调优经验HAProxy调参# 调整最大连接数 global maxconn 10000 tune.ssl.default-dh-param 2048 defaults timeout http-keep-alive 300sCalico优化# 启用BBR拥塞控制 kubectl annotate node --all projectcalico.org/bpfEnabledtrueMinIO磁盘调度echo ACTIONadd|change, KERNELsd*, ATTR{queue/rotational}0, ATTR{queue/scheduler}none /etc/udev/rules.d/60-minio.rules这套架构经过多个生产环境验证在持续7x24小时运行中保持99.99%的可用性。关键点在于负载均衡层与计算层完全解耦存储采用纠删码模式单节点故障不影响数据完整性所有组件都有健康检查机制和自动恢复能力实际部署时建议先进行压力测试使用工具模拟并发镜像推送/拉取场景根据硬件配置调整各组件参数。