1.使用cephadm部署ceph集群（ceph）
一、前言本文主要介绍如何通过cephadm部署ceph集群。环境说明部署方法cephadm操作系统ubuntu22.04ceph版本S版本部署过程中需要拉ceph镜像可以直接去我个人镜像仓库拉取二、架构图ceph的架构图如下后期博客会按照这个架构图详细更新解释大家可以先看下。本次内容主要还是“使用cephadm部署ceph集群”三、官网官网链接https://ceph.io/en/官网文档链接Welcome to Ceph — Ceph Documentation中文社区文档链接https://docs.ceph.net.cn 正常http://docs.ceph.org.cn (已停止维护官方软件下载仓库Index of /四、版本说明本文部署的是19.2.2 S版本1.看ceph的版本1大版本Ceph Releases (index) — Ceph Documentation2小版本https://docs.ceph.com/en/latest/releases/general/x.0.z 开发版本一般是用于内部测试。x.1.z 测试版本一般是开发人员测试使用。x.2.z 稳定版本生产环境中推荐使用。五、安装ceph的常用工具Installing Ceph — Ceph Documentation1.官网推荐安装方法一、方法一cephadm必须有podman或者docker环境podman是一种容器管理工具和docker一样podman是红帽的ceph也是红帽的二、方法二rook前提必须是先有k8s集群2.其他方法其他方法中之前有ceph-ansible和ceph-deployceph-deploy(ceph的原生部署工具):(1)ceph-deploy是一种部署ceph的方法它仅依赖于SSH访问服务器、而后借助sudo和一些Python模块就可以实现部署。(2)它完全在工作站管理主机上运行不需要任何服务、数据库或类似的东西。(3)它不是一个通用的部署系统它只是为Ceph设计的并且是为那些希望在不需要安装Chef、Puppet或Juju的情况下使用合理的初始设置快速运行Ceph的用户而设计的。(4)除了推送Ceph配置文件之外它不会处理客户端配置想要对安全设置、分区或目录位置进行精确控制的用户应该使用Chef或Puppet之类的工具。ceph-ansible:(1)我们可以使用ansible的playbook来部署Ceph;(2)ceph的GitHub地址为: https://github.com/ceph/ceph-ansibleceph-chef:对应的GitHub地址: https://github.com/ceph/ceph-chefpuppet-ceph:即使用puppet工具来部署ceph。Cephadm:和k8s的kubeadm一样好用部署服务非常方便推荐使用。参考链接:https://docs.ceph.com/en/latest/install/#other-methodsceph:https://github.com/ceph/ceph3.cephadm部署ceph集群Using Cephadm to Deploy a New Ceph Cluster — Ceph Documentation六、使用cephadm部署ceph官网链接Using Cephadm to Deploy a New Ceph Cluster — Ceph Documentation1.cephadm基础配置环境准备不会准备的虚拟机的可以参考我之前博客https://blog.csdn.net/2402_84026123/article/details/167703896?spm1011.2415.3001.10575sharefrommp_manage_link基于cephadm部署前提条件官方提的要求Ubuntu 22.04 LTS出了容器运行时其他都满足Python 3SystemdPodman or Docker for running containersTime synchronization (such as Chrony or the legacy ntpd)LVM2 for provisioning storage devicesUsing Cephadm to Deploy a New Ceph Cluster — Ceph Documentation1.虚拟机环境准备准备3台ubt22.04LTS的虚拟机ceph141:ip10.0.0.141CPU: 1c Memory: 2G /dev/sdb300GB /dev/sdc: 500GB /dev/sdd: 1024GBceph142:ip10.0.0.142CPU: 1c Memory: 2G /dev/sdb300GB /dev/sdc: 500GB /dev/sdd: 1024GBceph143:ip10.0.0.0.143CPU: 1c Memory: 2G /dev/sdb300GB /dev/sdc: 500GB /dev/sdd: 1024GB[rootceph141 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 63.9M 1 loop /snap/core20/2318 loop1 7:1 0 87M 1 loop /snap/lxd/29351 loop2 7:2 0 64M 1 loop /snap/core20/2379 loop3 7:3 0 89.4M 1 loop /snap/lxd/31333 loop4 7:4 0 40.4M 1 loop /snap/snapd/20671 loop5 7:5 0 38.8M 1 loop /snap/snapd/21759 sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 2G 0 part /boot └─sda3 8:3 0 98G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 49G 0 lvm / sdb 8:16 0 300G 0 disk sdc 8:32 0 500G 0 disk sdd 8:48 0 1T 0 disk sr0 11:0 1 1024M 0 rom [rootceph141 ~]# [rootceph142 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop1 7:1 0 87M 1 loop /snap/lxd/29351 loop2 7:2 0 38.8M 1 loop /snap/snapd/21759 loop3 7:3 0 40.4M 1 loop /snap/snapd/20671 loop4 7:4 0 64M 1 loop /snap/core20/2379 loop5 7:5 0 89.4M 1 loop /snap/lxd/31333 loop6 7:6 0 63.7M 1 loop /snap/core20/2496 sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 2G 0 part /boot └─sda3 8:3 0 98G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 49G 0 lvm / sdb 8:16 0 300G 0 disk sdc 8:32 0 500G 0 disk sdd 8:48 0 1T 0 disk sr0 11:0 1 1024M 0 rom [rootceph142 ~]# [rootceph143 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 89.4M 1 loop /snap/lxd/31333 loop1 7:1 0 64M 1 loop /snap/core20/2379 loop2 7:2 0 87M 1 loop /snap/lxd/29351 loop3 7:3 0 40.4M 1 loop /snap/snapd/20671 loop5 7:5 0 38.8M 1 loop /snap/snapd/21759 loop6 7:6 0 63.7M 1 loop /snap/core20/2496 sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 2G 0 part /boot └─sda3 8:3 0 98G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 49G 0 lvm / sdb 8:16 0 300G 0 disk sdc 8:32 0 500G 0 disk sdd 8:48 0 1T 0 disk sr0 11:0 1 1024M 0 rom [rootceph143 ~]#不会安装操作系统的参考我之前博客Ubuntu操作系统安装-CSDN博客2.三台服务器都设置时区timedatectl set-timezone Asia/Shanghai ll /etc/localtime3.安装docker环境三台都安装不会安装docker的参考我之前博客二进制部署dockerDocker进阶之路1-CSDN博客4.所有节点添加hosts文件解析三台都执行cat /etc/hosts EOF 10.0.0.141 ceph141 10.0.0.142 ceph142 10.0.0.143 ceph143 EOF2.安装cephadm工具官网下载链接Index of /1.下载需要安装ceph版本的cephadm方法一、直接用curl安装[rootceph141 ~]# CEPH_RELEASE19.2.2 curl --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm [rootceph141 ~]# ll cephadm -rw-r--r-- 1 root root 787576 Oct 8 15:56 cephadm方法二、在浏览器下载下来然后再传到服务器里面打开网站搜索19.2.2版本这里我们用这个版本建议统一和我一个版本2 将cephadm添加到PATH环境变量[rootceph141 ~]# mv cephadm /usr/local/bin/cephadm [rootceph141 ~]# chmod x /usr/local/bin/cephadm [rootceph141 ~]# ll /usr/local/bin/cephadm -rwxr-xr-x 1 root root 787576 Oct 8 15:44 /usr/local/bin/cephadm* [rootceph141 ~]#3 检查版本是否正确[rootceph141 ~]# cephadm version cephadm version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) [rootceph141 ~]#3.基于cephadm快速初始化ceph集群(141操作)1.拉取镜像方法一、去官网拉取镜像有下列这些镜像[rootceph141 ~]# docker image ls | awk {print $1:$2} REPOSITORY:TAG quay.io/ceph/ceph:v19 quay.io/prometheus/prometheus:v2.51.0 grafana/grafana:10.4.0 quay.io/prometheus/node-exporter:v1.7.0 quay.io/prometheus/alertmanager:v0.25.0 [rootceph141 ~]#把镜像导入到本地[rootceph141 ~]#docker load -i 镜像包方法二、直接去我个人的镜像仓库拉取镜像连接crpi-2qt1qcwq3ji0zfnc.cn-chengdu.personal.cr.aliyuncs.com/wuyuhua666888/ceph:v19crpi-2qt1qcwq3ji0zfnc.cn-chengdu.personal.cr.aliyuncs.com/wuyuhua666888/prometheus:v2.51.0crpi-2qt1qcwq3ji0zfnc.cn-chengdu.personal.cr.aliyuncs.com/wuyuhua666888/node-exporter:v1.7.0crpi-2qt1qcwq3ji0zfnc.cn-chengdu.personal.cr.aliyuncs.com/wuyuhua666888/alertmanager:v0.25.0crpi-2qt1qcwq3ji0zfnc.cn-chengdu.personal.cr.aliyuncs.com/wuyuhua666888/grafana:10.4.0拉下来自己打个tag就可以了示例[rootceph141 ~]# docker tag crpi-2qt1qcwq3ji0zfnc.cn-chengdu.personal.cr.aliyuncs.com/wuyuhua666888/node-exporter:v1.7.0 quay.io/prometheus/node-exporter:v1.7.02 创建新集群--skip-pull直接到本地拉取镜像不加会走外网拉镜像[rootceph141 ~]# cephadm bootstrap --mon-ip 10.0.0.141 --cluster-network 10.0.0.0/24 --allow-fqdn-hostname --skip-pull3.浏览器访问登录url和账号密码都在上一步初始化的结果里面设置一个新密码成功进入主界面4.配置ceph集群的管理节点的两种方式方式一、ceph客户端直接管理需要下载ceph-common工具方式二、cephadm shell容器管理1.在宿主机安装ceph管理命令1 添加软件源[rootceph141 ~]# cephadm add-repo --release squid Installing repo GPG key from https://download.ceph.com/keys/release.gpg... Installing repo file at /etc/apt/sources.list.d/ceph.list... Updating package list... Completed adding repo. [rootceph141 ~]# [rootceph141 ~]# ll /etc/apt/sources.list.d/ceph.list -rw-r--r-- 1 root root 55 Mar 31 16:38 /etc/apt/sources.list.d/ceph.list [rootceph141 ~]# [rootceph141 ~]# ll /etc/apt/trusted.gpg.d/ceph.release.gpg -rw-r--r-- 1 root root 1143 Mar 31 16:38 /etc/apt/trusted.gpg.d/ceph.release.gpg [rootceph141 ~]# [rootceph141 ~]# ll /etc/ceph/ total 20 drwxr-xr-x 2 root root 4096 Mar 31 15:37 ./ drwxr-xr-x 130 root root 4096 Mar 31 15:30 ../ -rw------- 1 root root 151 Mar 31 15:37 ceph.client.admin.keyring -rw-r--r-- 1 root root 171 Mar 31 15:37 ceph.conf -rw-r--r-- 1 root root 595 Mar 31 15:33 ceph.pub [rootceph141 ~]#替换成国内阿里的加速源[rootceph141 ~]# sed -i s#download.ceph.com#mirrors.aliyun.com/ceph# /etc/apt/sources.list.d/ceph.list [rootceph141 ~]# cat /etc/apt/sources.list.d/ceph.list deb https://mirrors.aliyun.com/ceph/debian-squid/ jammy main [rootceph141 ~]#2 安装ceph通用工具包[rootceph141 ~]# apt update [rootceph141 ~]# apt -y install ceph-common3.测试使用2.基于容器管理ceph集群1 启动shell工具执行cephadm shell会自动进入一个容器可以在容器里面直接使用ceph命令管理ceph2 本质上是启动了一个容器【当命令执行结束时容器会自动删除】5.ceph集群添加主机、删除主机1.添加主机1.镜像导入其他两个节点将docker和镜像重新从ceph141节点导入到ceph142和ceph143节点。docker load导入即可。2.把秘钥放到其他服务器上[rootceph141 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub ceph142 [rootceph141 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub ceph1433.添加主机[rootceph141 ~]# ceph orch host add ceph142 10.0.0.142 Added host ceph142 with addr 10.0.0.142 [rootceph141 ~]# [rootceph141 ~]# ceph orch host add ceph143 10.0.0.143 Added host ceph143 with addr 10.0.0.143 [rootceph141 ~]#温馨提示 将集群加入成功后会自动创建/var/lib/ceph/Ceph_Cluster_ID相关数据目录。4.查看主机列表[rootceph141 ~]# ceph orch host ls HOST ADDR LABELS STATUS ceph141 10.0.0.141 _admin ceph142 10.0.0.142 ceph143 10.0.0.143 3 hosts in cluster [rootceph141 ~]#5.通过浏览器查看集群列表发现三台都可用了2.删除主机删除ceph143主机来演示下“如何删除主机”1.驱逐已经调度到节点的服务[rootceph141 ~]# ceph orch host drain ceph143 Scheduled to remove the following daemons from host ceph143 type id -------------------- --------------- node-exporter ceph143 mon ceph143 ceph-exporter ceph143 crash ceph143 [rootceph141 ~]#2.驱逐后主机将被标记为不可调度[rootceph141 ~]# ceph orch host ls HOST ADDR LABELS STATUS ceph141 10.0.0.141 _admin ceph142 10.0.0.142 ceph143 10.0.0.143 _no_schedule,_no_conf_keyring 3 hosts in cluster [rootceph141 ~]#3.观察已经驱逐完毕[rootceph141 ~]# ceph orch host drain ceph143 Scheduled to remove the following daemons from host ceph143 type id -------------------- --------------- [rootceph141 ~]#也可以看下daemon运行情况驱逐后ceph143节点上没有daemon在运行[rootceph141 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID alertmanager.ceph141 ceph141 *:9093,9094 running (5h) 12s ago 27h 24.9M - 0.25.0 c8568f914cd2 5bfdbc4589fa ceph-exporter.ceph141 ceph141 *:9926 running (5h) 12s ago 27h 7487k - 19.2.6 829c8a87e883 430386604cc7 ceph-exporter.ceph142 ceph142 *:9926 running (5h) 6m ago 5h 5231k - 19.2.6 829c8a87e883 0d3a5f0031df crash.ceph141 ceph141 running (5h) 12s ago 27h 18.8M - 19.2.6 829c8a87e883 2afb5e7f6b5c crash.ceph142 ceph142 running (5h) 6m ago 5h 4503k - 19.2.6 829c8a87e883 325bf42faec4 grafana.ceph141 ceph141 *:3000 running (5h) 12s ago 27h 179M - 12.3.1 5cdab57891ea 90fbbc9f2188 mgr.ceph141.hdvoti ceph141 *:9283,8765,8443 running (5h) 12s ago 27h 186M - 19.2.6 829c8a87e883 e949c78b86f3 mgr.ceph142.zicwnb ceph142 *:8443,9283,8765 running (5h) 6m ago 5h 111M - 19.2.6 829c8a87e883 46322e43fdca mon.ceph141 ceph141 running (5h) 12s ago 27h 78.1M 2048M 19.2.6 829c8a87e883 261a121614e5 mon.ceph142 ceph142 running (5h) 6m ago 5h 87.9M 2048M 19.2.6 829c8a87e883 c4cfa477d901 node-exporter.ceph141 ceph141 *:9100 running (5h) 12s ago 27h 16.2M - 1.7.0 72c9c2088986 c8a30176f66f node-exporter.ceph142 ceph142 *:9100 running (5h) 6m ago 5h 17.9M - 1.7.0 72c9c2088986 d21699c4fde7 prometheus.ceph141 ceph141 *:9095 running (48s) 12s ago 27h 46.2M - 2.51.0 1d3b7f56885b 572df0ed27a74.移除节点[rootceph141 ~]# ceph orch host rm ceph143 Removed host ceph143 [rootceph141 ~]# [rootceph141 ~]# ceph orch host ls HOST ADDR LABELS STATUS ceph141 10.0.0.141 _admin ceph142 10.0.0.142 2 hosts in cluster [rootceph141 ~]#5.删除数据日志等一些列ceph相关的 数据【或者直接重新安装操作系统即可避免数据泄露。推荐操作】[rootceph143 ~]# ll /var/lib/ceph/ total 12 drwxr-xr-x 3 root root 4096 Mar 31 17:02 ./ drwxr-xr-x 62 root root 4096 Mar 31 17:02 ../ drwx------ 4 167 167 4096 Mar 31 17:08 11e66474-0e02-11f0-82d6-4dcae3d59070/ [rootceph143 ~]# [rootceph143 ~]# [rootceph143 ~]# ll /var/log/ceph/ total 112 drwxr-xr-x 3 root root 4096 Mar 31 17:05 ./ drwxrwxr-x 10 root syslog 4096 Mar 31 17:02 ../ drwxrwx--- 2 167 167 4096 Mar 31 17:06 11e66474-0e02-11f0-82d6-4dcae3d59070/ -rw-r--r-- 1 root root 96321 Mar 31 17:09 cephadm.log [rootceph143 ~]#因为我们只是演示下如何删除主机所以删除测试后再把主机加回来[rootceph141 ~]# ceph orch host add ceph143 10.0.0.143 Added host ceph143 with addr 10.0.0.143 [rootceph141 ~]# [rootceph141 ~]# ceph orch host ls HOST ADDR LABELS STATUS ceph141 10.0.0.141 _admin ceph142 10.0.0.142 ceph143 10.0.0.143 3 hosts in cluster [rootceph141 ~]#6.添加OSD设备到ceph集群温馨提示: 如果一个设备想要加入ceph集群要求满足3个条件:1.设备未被使用;2.设备的存储大小必须大于5GB;3.需要等待一段时间快则30s慢则3分钟线下教学有人笔记本性能不高甚至有等了25min或者40min才等出现设备结果; 如果想要快速显示则可以在相应节点手动导入quay.io/ceph/ceph镜像即可。1 查看集群可用的设备【每个设备想要加入到集群则其大小不得小于5GB】[rootceph141 ~]# ceph orch device ls HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph141 /dev/sdb hdd 300G Yes 28m ago ceph141 /dev/sdc hdd 500G Yes 28m ago ceph141 /dev/sdd hdd 1024G Yes 28m ago ceph141 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 28m ago Failed to determine if device is BlueStore, Insufficient space (5GB) ceph142 /dev/sdb hdd 300G Yes 3m ago ceph142 /dev/sdc hdd 500G Yes 3m ago ceph142 /dev/sdd hdd 1024G Yes 3m ago ceph142 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 3m ago Failed to determine if device is BlueStore, Insufficient space (5GB) ceph143 /dev/sdb hdd 300G Yes 20m ago ceph143 /dev/sdc hdd 500G Yes 20m ago ceph143 /dev/sdd hdd 1024G Yes 20m ago ceph143 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 20m ago Failed to determine if device is BlueStore, Insufficient space (5GB) [rootceph141 ~]#2.添加osd设备到集群[rootceph141 ~]# ceph orch daemon add osd ceph141:/dev/sdb Created osd(s) 0 on host ceph141 [rootceph141 ~]# [rootceph141 ~]# ceph orch daemon add osd ceph141:/dev/sdc Created osd(s) 1 on host ceph141 [rootceph141 ~]# [rootceph141 ~]# ceph orch daemon add osd ceph141:/dev/sdd Created osd(s) 2 on host ceph141 [rootceph141 ~]# [rootceph141 ~]# [rootceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdb Created osd(s) 3 on host ceph142 [rootceph141 ~]# [rootceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdc Created osd(s) 4 on host ceph142 [rootceph141 ~]# [rootceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdd Created osd(s) 5 on host ceph142 [rootceph141 ~]# [rootceph141 ~]# ceph orch daemon add osd ceph143:/dev/sdb Created osd(s) 6 on host ceph143 [rootceph141 ~]# [rootceph141 ~]# ceph orch daemon add osd ceph143:/dev/sdc Created osd(s) 7 on host ceph143 [rootceph141 ~]# [rootceph141 ~]# ceph orch daemon add osd ceph143:/dev/sdd Created osd(s) 8 on host ceph143 [rootceph141 ~]#3.添加后验证查看osd查看磁盘总量查看集群健康度到这里“使用cephadm部署ceph集群”就全部完成了后续会更新ceph的进阶使用本文来自csdn作者Yuhua 吴玉桦。当你的能力还驾驭不了你的生活的时候你就应该沉下心来历练。问问自己想要怎样的人生。