You can support us by downloading this article as PDF from the Link below. Download the guide as PDF

In this guide, we will cover the installation and configuration of a fully functional Ceph Storage Cluster on Ubuntu 18.04 LTS server. Ceph is an open source storage solution that implements distributed object storage cluster, and provides interfaces for an object, block, and file-level storage.

Components of Ceph storage cluster

The basic components of a Ceph storage cluster

  • Monitors: A Ceph Monitor (ceph-mon) maintains maps of the cluster state, including the monitor map, manager map, the OSD map, and the CRUSH map
  • Ceph OSDs: A Ceph OSD (object storage daemon, ceph-osd) stores data, handles data replication, recovery, rebalancing, and provides some monitoring information to Ceph Monitors and Managers by checking other Ceph OSD Daemons for a heartbeat. At least 3 Ceph OSDs are normally required for redundancy and high availability.
  • MDSs: A Ceph Metadata Server (MDS, ceph-mds) stores metadata on behalf of the Ceph Filesystem (i.e., Ceph Block Devices and Ceph Object Storage do not use MDS). Ceph Metadata Servers allow POSIX file system users to execute basic commands (like,ls, find etc.) without placing an enormous burden on the Ceph Storage Cluster.\
  • Managers: A Ceph Manager daemon (ceph-mgr) is responsible for keeping track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load.

Ceph Cluster on Ubuntu 18.04 Architecture

My Ceph cluster deployment is based on the following system diagram

The hostnames of each server should be set accordingly, and /etc/hosts file configured to look like below on all Ceph Cluster Nodes:

192.168.18.70 rgw.example.com   rgw
192.168.18.71 mon01.example.com mon01
192.168.18.72 mon02.example.com mon02
192.168.18.73 mon03.example.com mon03
192.168.18.75 osd01.example.com osd01
192.168.18.76 osd02.example.com osd02
192.168.18.77 osd03.example.com osd03

Change the hostnames and IP addresses to fit your setup. As a pre-requisite, ensure your system is updated

sudo apt update
sudo apt -y upgrade
sudo reboot

Once the systems are rebooted, you can begin Ceph cluster deployment. It is recommended to have an additional node, whose work is deploying ceph on all Ceph nodes. This node is called admin node

192.168.18.74 ceph-admin.example.com ceph-admin

Prepare the Ceph Admin Node

You need to add Ceph repositories to the ceph-deploy admin node, then, install ceph-deploy package. My admin  node is on IP 192.168.18.74

Import repository key

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

Add the Ceph repository to your system. This installation will do Ceph nautilus

echo deb https://download.ceph.com/debian-nautilus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

Update your repository and install ceph-deploy:

sudo apt update
sudo apt -y install ceph-deploy

Prepare Ceph  Nodes

The admin node must have password-less SSH access to Ceph nodes. When ceph-deploy logs in to a Ceph node as a user, that particular user must have passwordless sudo privileges.

Add SSH user on All Ceph Nodesrgw,osd nodes & Monitors

export USER_NAME="ceph-admin"
export USER_PASS="[email protected]"
sudo useradd --create-home -s /bin/bash ${USER_NAME}
echo "${USER_NAME}:${USER_PASS}"|sudo chpasswd
echo "${USER_NAME} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/${USER_NAME}
sudo chmod 0440 /etc/sudoers.d/${USER_NAME}

Confirm that the user can run sudo without a password prompt

[email protected]:~$ su - ceph-admin
Password: 
[email protected]:~$ sudo su -
[email protected]:~#

Generate the SSH keys on ceph-admin node, but do not use sudo or the root user. Leave the passphrase empty:

# su - ceph-admin
$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph-admin/.ssh/id_rsa): 
Created directory '/home/ceph-admin/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ceph-admin/.ssh/id_rsa.
Your public key has been saved in /home/ceph-admin/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:DZZdfRS1Yo+unWAkDum7juShEF67pm7VdSkfWlsCpbA [email protected]
The key's randomart image is:
+---[RSA 2048]----+
|     .  ..  .. o=|
|      o..o .  . o|
|     E .= o  o o |
|       +.O .. +  |
| . .. .oS.*. . . |
|. o.....ooo .    |
| o.. o . . o .   |
| ...= o . . + .  |
|oooo o.+.  . o   |
+----[SHA256]-----+

$ ls /home/ceph-admin/.ssh/
config  id_rsa  id_rsa.pub  known_hosts

Configure your ~/.ssh/config

$ cat /home/ceph-admin/.ssh/config 
Host osd01
  Hostname osd01
  User ceph-admin
Host osd02
  Hostname osd02
  User ceph-admin
Host osd03
  Hostname osd03
  User ceph-admin
Host osd01
  Hostname osd01
  User ceph-admin
Host mon01
  Hostname mon01
  User ceph-admin
Host mon02
  Hostname mon02
  User ceph-admin
Host mon03
  Hostname mon03
  User ceph-admin
Host rgw
  Hostname rgw
  User ceph-admin

Copy the key to each Ceph Node ( Do this from Ceph Admin Node as the ceph-admin user)

for i in rgw mon01 mon02 mon03 osd01 osd02 osd03; do
 ssh-copy-id $i
done

If your remote user on all Ceph Nodes is not,ceph-admin replace it with correct username.

Deploy Ceph  Storage Cluster

Ceph is sensitive to Clock drift, so ensure you setup NTP on all Ceph Nodes, especially Monitors.

sudo apt install ntp

We’re going to create a three Ceph Node cluster with one Ceph Monitor and three Ceph OSD Daemons. Once the cluster reaches a active + clean state, expand it by adding a Metadata Server and two more Ceph Monitors

Let’s start by creating a directory on our admin node for maintaining the configuration files and keys that ceph-deploy generates for the cluster.

mkdir ceph-deploy
cd ceph-deploy

The ceph-deploy utility will output files to the current directory. Ensure you are in this directory when executing ceph-deploy.

Step 1: Initialize ceph monitor nodes

Run the following commands on your admin node from the ceph-deploy directory you created for holding your configuration details

$ ceph-deploy new mon01 mon02 mon03

Sample output

[email protected]:~/ceph-deploy$ ceph-deploy new  mon01 mon02 mon03
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy new mon01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4c0720a950>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['mon01']
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f4c07456d70>
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mon01][DEBUG ] connected to host: ceph-admin 
[mon01][INFO  ] Running command: ssh -CT -o BatchMode=yes mon01
[mon01][DEBUG ] connection detected need for sudo
[mon01][DEBUG ] connected to host: mon01 
[mon01][DEBUG ] detect platform information from remote host
[mon01][DEBUG ] detect machine type
[mon01][DEBUG ] find the location of an executable
[mon01][INFO  ] Running command: sudo /bin/ip link show
[mon01][INFO  ] Running command: sudo /bin/ip addr show
[mon01][DEBUG ] IP addresses found: [u'192.168.18.71', u'192.168.19.71', u'192.168.121.23']
[ceph_deploy.new][DEBUG ] Resolving host mon01
[ceph_deploy.new][DEBUG ] Monitor mon01 at 192.168.18.71
[ceph_deploy.new][DEBUG ] Monitor initial members are ['mon01']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.18.71']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf..

Step 2: Install Ceph packages:

Install Ceph Packages on all nodes.

ceph-deploy install mon01 mon02 mon03 osd01 osd02 osd03 rgw

The ceph-deploy utility will install Ceph on each node.

Sample output

..........................
[osd03][DEBUG ] Setting up ceph-base (13.2.2-1bionic) ...
[osd03][DEBUG ] Setting up python-pecan (1.2.1-2) ...
[osd03][DEBUG ] update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode
[osd03][DEBUG ] update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode
[osd03][DEBUG ] Setting up ceph-osd (13.2.2-1bionic) ...
[osd03][DEBUG ] chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
[osd03][DEBUG ] Setting up ceph-mds (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
[osd03][DEBUG ] Setting up ceph-mon (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
[osd03][DEBUG ] Setting up ceph-mgr (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
[osd03][DEBUG ] Setting up ceph (13.2.2-1bionic) ...
[osd03][DEBUG ] Processing triggers for libc-bin (2.27-3ubuntu1) ...
[osd03][DEBUG ] Processing triggers for ureadahead (0.100.0-20) ...
[osd03][DEBUG ] Processing triggers for systemd (237-3ubuntu10) ...
[osd03][INFO  ] Running command: sudo ceph --version
[osd03][DEBUG ] ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)

Step 3: Deploy the initial monitor(s) and gather the keys:

Create initial monitors by running the command:

ceph-deploy mon create-initial

A number of keyrings will be placed in your working directory.

Deploy a manager daemon:

ceph-deploy mgr create mon01 mon02 mon03

Add a Metadata Servers:

ceph-deploy mds create mon01 mon02 mon03

Step 4: Copy Ceph Admin Key

Copy the configuration file and admin key to your admin node and your Ceph Nodes:

ceph-deploy admin mon01 mon02 mon03 osd01 osd02 osd03

Step 5: Add three OSDs

I assume you have an unused disk in each node called /dev/device. Mine look like below:

[email protected]:~# lsblk 
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   32G  0 disk 
|-sda1   8:1    0  487M  0 part /boot
|-sda2   8:2    0  1.9G  0 part [SWAP]
`-sda3   8:3    0 29.6G  0 part /
vdb    252:0    0    5G  0 disk 
vdc    252:16   0    5G  0 disk 
vdd    252:32   0    5G  0 disk

I have three devices to use:

  • /dev/vda
  • dev/vdb
  • dev/vdc

Be sure that the device is not currently in use and does not contain any important data. The syntax to use is:

ceph-deploy osd create --data {device} {ceph-node}

In my case, I’ll run:

for i in vdb vdc vdd; do
  for j in osd01 osd02 osd03; do
    ceph-deploy osd create --data /dev/$i $j
done
done

Note: If you are creating an OSD on an LVM volume, the argument to --data must be volume_group/lv_name, rather than the path to the volume’s block device.

The output of the lsblkcommand should have changed

# lsblk 
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                                                                     8:0    0   32G  0 disk 
|-sda1                                                                                                  8:1    0  487M  0 part /boot
|-sda2                                                                                                  8:2    0  1.9G  0 part [SWAP]
`-sda3                                                                                                  8:3    0 29.6G  0 part /
vdb                                                                                                   252:0    0    5G  0 disk 
`-ceph--908c8792--04e8--414f--8430--faa78e9b18eb-osd--block--275c9d8b--3825--4898--9b3b--5ea080fd7137 253:0    0    5G  0 lvm  
vdc                                                                                                   252:16   0    5G  0 disk 
`-ceph--c79a5159--3980--47e8--b649--ed0c44d32d51-osd--block--a50c2ebc--8d65--4d16--9196--6f741606b3a2 253:1    0    5G  0 lvm  
vdd                                                                                                   252:32   0    5G  0 disk 
`-ceph--594ff477--943e--49d8--8e08--addf5dbccca3-osd--block--5b71bad9--7fa8--41af--a3af--48af1219aafd 253:2    0    5G  0 lvm

Step 6: Check your cluster’s health.

Check Ceph Cluster status

[email protected]:~# sudo ceph health
HEALTH_OK

[email protected]:~# sudo ceph status
  cluster:
    id:     5594514c-4d70-4ec8-a74e-e54ac2c1994f
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum mon01
    mgr: osd01(active)
    osd: 9 osds: 9 up, 9 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   9.0 GiB used, 36 GiB / 45 GiB avail
    pgs:

Once you have added your new Ceph Monitors, Ceph will begin synchronizing the monitors and form a quorum. You can check the quorum status by executing the following:

ceph quorum_status --format json-pretty

Step 7: Enable Ceph Dashboard

Enable the Ceph Dashboard module:

sudo ceph mgr module enable dashboard
sudo ceph mgr module ls

Generate self signed certificates for the dashboard:

sudo ceph dashboard create-self-signed-cert

Create a user for Dashboard:

sudo ceph dashboard ac-user-create admin '[email protected]' administrator

Enabling the Object Gateway Management Frontend:

$ sudo radosgw-admin user create --uid=admin --display-name='Ceph Admin' --system

Finally, provide the credentials to the dashboard:

sudo ceph dashboard set-rgw-api-access-key <api-access-key>
sudo ceph dashboard set-rgw-api-secret-key <api-secret-key>

If you are using a self-signed certificate in your Object Gateway setup, then you should disable certificate verification:

sudo ceph dashboard set-rgw-api-ssl-verify False

Step 8: Add Rados Gateway

To use the Ceph Object Gateway component of Ceph, you must deploy an instance of RGW. Execute the following to create a new instance of Rados Gateway:

$ ceph-deploy rgw create {gateway-node}

Example:

$ ceph-deploy rgw create rgw

By default, the RGW instance will listen on port 7480. This can be changed by editing ceph.conf on the node running the RGW as follows:

[client]
rgw frontends = civetweb port=80

Resetting your Ceph Cluster

If at any point you run into trouble and you want to start over, execute the following to purge the Ceph packages, and erase all its data and configuration:

ceph-deploy purge {ceph-node} [{ceph-node}]
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys
rm ceph.*

If you execute purge, you must re-install Ceph. The last rm command removes any files that were written out by ceph-deploy locally during a previous installation.

In our next article, I’ll cover how to store and retrieve objects on Ceph and configuring Ceph Clients.

Check other similar guide:

Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7

How to Setup S3 Compatible Object Storage Server with Minio

Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7

You can support us by downloading this article as PDF from the Link below. Download the guide as PDF