(Last Updated On: October 22, 2018)

In this guide, we will cover the installation and configuration of a fully functional Ceph Storage Cluster on Ubuntu 18.04 LTS server. Ceph is an open source storage solution that implements distributed object storage cluster, and provides interfaces for an object, block, and file-level storage.

Components of Ceph storage cluster

The basic components of a Ceph storage cluster

  • Monitors: A Ceph Monitor (ceph-mon) maintains maps of the cluster state, including the monitor map, manager map, the OSD map, and the CRUSH map
  • Ceph OSDs: A Ceph OSD (object storage daemon, ceph-osd) stores data, handles data replication, recovery, rebalancing, and provides some monitoring information to Ceph Monitors and Managers by checking other Ceph OSD Daemons for a heartbeat. At least 3 Ceph OSDs are normally required for redundancy and high availability.
  • MDSs: A Ceph Metadata Server (MDS, ceph-mds) stores metadata on behalf of the Ceph Filesystem (i.e., Ceph Block Devices and Ceph Object Storage do not use MDS). Ceph Metadata Servers allow POSIX file system users to execute basic commands (like,ls, find etc.) without placing an enormous burden on the Ceph Storage Cluster.\
  • Managers: A Ceph Manager daemon (ceph-mgr) is responsible for keeping track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load.

My Ceph Storage cluster Architecture

My Ceph cluster deployment is based on the following system diagram

The hostnames of each server should be set accordingly, and /etc/hosts file configured to look like below on all Ceph Cluster Nodes:

192.168.18.70 rgw.example.com   rgw
192.168.18.71 mon01.example.com mon01
192.168.18.72 mon02.example.com mon02
192.168.18.73 mon03.example.com mon03
192.168.18.75 osd01.example.com osd01
192.168.18.76 osd02.example.com osd02
192.168.18.77 osd03.example.com osd03

Change the hostnames and IP addresses to fit your setup. As a pre-requisite, ensure your system is updated

sudo apt update
sudo apt upgrade
sudo reboot

Once the systems are rebooted, you can begin Ceph cluster deployment. It is recommended to have an additional node, whose work is deploying ceph on all Ceph nodes. This node is called admin node

192.168.18.74 ceph-admin.example.com ceph-admin

Step 1: Prepare the Ceph Admin Node

You need to add Ceph repositories to the ceph-deploy admin node, then, install ceph-deploy package. My admin  node is on IP 192.168.18.74

Import repository key

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

Add the Ceph repository to your system. Replace {ceph-stable-release} with a stable Ceph release (e.g., mimic.)

echo deb https://download.ceph.com/debian-{ceph-stable-release}/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

This installation will do Ceph mimic

echo deb https://download.ceph.com/debian-mimic/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

Update your repository and install ceph-deploy:

sudo apt update
sudo apt -y install ceph-deploy

Step 2: Prepare Ceph  Nodes

The admin node must have password-less SSH access to Ceph nodes. When ceph-deploy logs in to a Ceph node as a user, that particular user must have passwordless sudo privileges.

Add SSH user on All Ceph Nodesrgw,osd nodes & Monitors

export USERNAME="ceph-admin"
export USER_PASS="StrongPassword"
sudo useradd --create-home -s /bin/bash ${USERNAME}
echo "${USERNAME}:${USER_PASS}"|chpasswd
echo "${USERNAME} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/${USERNAME}
sudo chmod 0440 /etc/sudoers.d/${USERNAME}

Confirm that the user can run sudo without a password prompt

[email protected]:~$ su - ceph-admin
Password: 
[email protected]:~$ sudo su -
[email protected]:~#

Generate the SSH keys on ceph-admin node, but do not use sudo or the root user. Leave the passphrase empty:

# su - ceph-admin
$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph-admin/.ssh/id_rsa): 
Created directory '/home/ceph-admin/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ceph-admin/.ssh/id_rsa.
Your public key has been saved in /home/ceph-admin/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:DZZdfRS1Yo+unWAkDum7juShEF67pm7VdSkfWlsCpbA [email protected]
The key's randomart image is:
+---[RSA 2048]----+
|     .  ..  .. o=|
|      o..o .  . o|
|     E .= o  o o |
|       +.O .. +  |
| . .. .oS.*. . . |
|. o.....ooo .    |
| o.. o . . o .   |
| ...= o . . + .  |
|oooo o.+.  . o   |
+----[SHA256]-----+

$ ls /home/ceph-admin/.ssh/
config  id_rsa  id_rsa.pub  known_hosts

Configure your ~/.ssh/config

$ cat /home/ceph-admin/.ssh/config 
Host osd01
  Hostname osd01
  User ceph-admin
Host osd02
  Hostname osd02
  User ceph-admin
Host osd03
  Hostname osd03
  User ceph-admin
Host osd01
  Hostname osd01
  User ceph-admin
Host mon01
  Hostname mon01
  User ceph-admin
Host mon02
  Hostname mon02
  User ceph-admin
Host mon03
  Hostname mon03
  User ceph-admin
Host rgw
  Hostname rgw
  User ceph-admin

Copy the key to each Ceph Node ( Do this from Ceph Admin Node as the ceph-admin user)

for i in rgw mon01 mon02 mon03 osd01 osd02 osd03; do
 ssh-copy-id $i
done

If your remote user on all Ceph Nodes is not,ceph-admin replace it with correct username.

Step 3: Deploy Ceph  Storage Cluster

Ceph is sensitive to Clock drift, so ensure you setup NTP on all Ceph Nodes, especially Monitors.

sudo apt install ntp

We’re going to create a three Ceph Node cluster with one Ceph Monitor and three Ceph OSD Daemons. Once the cluster reaches a active + clean state, expand it by adding a Metadata Server and two more Ceph Monitors

Let’s start by creating a directory on our admin node for maintaining the configuration files and keys that ceph-deploy generates for the cluster.

mkdir ceph-deploy
cd ceph-deploy

The ceph-deploy utility will output files to the current directory. Ensure you are in this directory when executing ceph-deploy.

Step 1: Initialize ceph monitor nodes

Run the following commands on your admin node from the ceph-deploy directory you created for holding your configuration details

$ ceph-deploy new mon01 mon02 mon03

Sample output

[email protected]:~/ceph-deploy$ ceph-deploy new mon01
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy new mon01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4c0720a950>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['mon01']
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f4c07456d70>
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mon01][DEBUG ] connected to host: ceph-admin 
[mon01][INFO  ] Running command: ssh -CT -o BatchMode=yes mon01
[mon01][DEBUG ] connection detected need for sudo
[mon01][DEBUG ] connected to host: mon01 
[mon01][DEBUG ] detect platform information from remote host
[mon01][DEBUG ] detect machine type
[mon01][DEBUG ] find the location of an executable
[mon01][INFO  ] Running command: sudo /bin/ip link show
[mon01][INFO  ] Running command: sudo /bin/ip addr show
[mon01][DEBUG ] IP addresses found: [u'192.168.18.71', u'192.168.19.71', u'192.168.121.23']
[ceph_deploy.new][DEBUG ] Resolving host mon01
[ceph_deploy.new][DEBUG ] Monitor mon01 at 192.168.18.71
[ceph_deploy.new][DEBUG ] Monitor initial members are ['mon01']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.18.71']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf..

Step 2: Install Ceph packages.:

ceph-deploy install mon01 mon02 mon03 osd01 osd02 osd03 rgw

The ceph-deploy utility will install Ceph on each node.

Sample output

..........................
[osd03][DEBUG ] Setting up ceph-base (13.2.2-1bionic) ...
[osd03][DEBUG ] Setting up python-pecan (1.2.1-2) ...
[osd03][DEBUG ] update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode
[osd03][DEBUG ] update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode
[osd03][DEBUG ] Setting up ceph-osd (13.2.2-1bionic) ...
[osd03][DEBUG ] chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
[osd03][DEBUG ] Setting up ceph-mds (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
[osd03][DEBUG ] Setting up ceph-mon (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
[osd03][DEBUG ] Setting up ceph-mgr (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
[osd03][DEBUG ] Setting up ceph (13.2.2-1bionic) ...
[osd03][DEBUG ] Processing triggers for libc-bin (2.27-3ubuntu1) ...
[osd03][DEBUG ] Processing triggers for ureadahead (0.100.0-20) ...
[osd03][DEBUG ] Processing triggers for systemd (237-3ubuntu10) ...
[osd03][INFO  ] Running command: sudo ceph --version
[osd03][DEBUG ] ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)

Step 3: Deploy the initial monitor(s) and gather the keys:

ceph-deploy mon create-initial

A number of keyrings will be placed in your working directory.

Step 4: Copy ceph configuration file and keyring to Nodes

Copy the configuration file and admin key to your admin node and your Ceph Nodes

ceph-deploy admin osd01 osd02 osd03

Step 5: Deploy a manager daemon.

Deploy a manager daemon by running the command:

ceph-deploy mgr create osd01
Step 6: Add three OSDs

I assume you have an unused disk in each node called /dev/device. Mine look like below:

[email protected]:~# lsblk 
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   32G  0 disk 
|-sda1   8:1    0  487M  0 part /boot
|-sda2   8:2    0  1.9G  0 part [SWAP]
`-sda3   8:3    0 29.6G  0 part /
vda    252:0    0    5G  0 disk 
vdb    252:16   0    5G  0 disk 
vdc    252:32   0    5G  0 disk

I have three devices to use:

  • /dev/vda
  • dev/vdb
  • dev/vdc

Be sure that the device is not currently in use and does not contain any important data. The syntax to use is:

ceph-deploy osd create --data {device} {ceph-node}

Example

ceph-deploy osd create --data /dev/vda osd01
ceph-deploy osd create --data /dev/vda osd02
ceph-deploy osd create --data /dev/vda osd03

ceph-deploy osd create --data /dev/vdb osd01
ceph-deploy osd create --data /dev/vdb osd02
ceph-deploy osd create --data /dev/vdb osd03

ceph-deploy osd create --data /dev/vdc osd01
ceph-deploy osd create --data /dev/vdc osd02
ceph-deploy osd create --data /dev/vdc osd03

Note: If you are creating an OSD on an LVM volume, the argument to --data must be volume_group/lv_name, rather than the path to the volume’s block device.

The output of the lsblkcommand should have changed

# lsblk 
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                                                                     8:0    0   32G  0 disk 
|-sda1                                                                                                  8:1    0  487M  0 part /boot
|-sda2                                                                                                  8:2    0  1.9G  0 part [SWAP]
`-sda3                                                                                                  8:3    0 29.6G  0 part /
vda                                                                                                   252:0    0    5G  0 disk 
`-ceph--908c8792--04e8--414f--8430--faa78e9b18eb-osd--block--275c9d8b--3825--4898--9b3b--5ea080fd7137 253:0    0    5G  0 lvm  
vdb                                                                                                   252:16   0    5G  0 disk 
`-ceph--c79a5159--3980--47e8--b649--ed0c44d32d51-osd--block--a50c2ebc--8d65--4d16--9196--6f741606b3a2 253:1    0    5G  0 lvm  
vdc                                                                                                   252:32   0    5G  0 disk 
`-ceph--594ff477--943e--49d8--8e08--addf5dbccca3-osd--block--5b71bad9--7fa8--41af--a3af--48af1219aafd 253:2    0    5G  0 lvm

Step 7: Check your cluster’s health.

Check Ceph Cluster status

[email protected]:~# ceph health
HEALTH_OK

[email protected]:~# ceph status
  cluster:
    id:     5594514c-4d70-4ec8-a74e-e54ac2c1994f
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum mon01
    mgr: osd01(active)
    osd: 9 osds: 9 up, 9 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   9.0 GiB used, 36 GiB / 45 GiB avail
    pgs:

Expand your Ceph Storage Cluster

Once you have a basic cluster up and running, the next step is to expand cluster.

Step 1: Add Ceph Metadata Server

To use CephFS, you need at least one metadata server. Execute the following to add a Ceph Metadata Server to osd02 & osd03:

ceph-deploy mds create osd02 osd03

Step 2: Add other Ceph Monitor

A Ceph Storage Cluster requires at least one Ceph Monitor and Ceph Manager to run. For high availability, Ceph Storage Clusters typically run multiple Ceph Monitors so that the failure of a single Ceph Monitor will not bring down the Ceph Storage Cluster, to extend your monitors number, use:

ceph-deploy install <monitor>
ceph-deploy mon add <monitor>

Once you have added your new Ceph Monitors, Ceph will begin synchronizing the monitors and form a quorum. You can check the quorum status by executing the following:

ceph quorum_status --format json-pretty

Step 3: Add additional Manager Daemons

To deploy additional manager daemons:

ceph-deploy mgr create mon02 mon03

Step 4: Add RGW

To use the Ceph Object Gateway component of Ceph, you must deploy an instance of RGW. Execute the following to create a new instance of RGW:

ceph-deploy rgw create {gateway-node}

Example:

ceph-deploy rgw create rgw

By default, the RGW instance will listen on port 7480. This can be changed by editing ceph.conf on the node running the RGW as follows:

[client]
rgw frontends = civetweb port=80

Reset your Ceph Cluster

If at any point you run into trouble and you want to start over, execute the following to purge the Ceph packages, and erase all its data and configuration:

ceph-deploy purge {ceph-node} [{ceph-node}]
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys
rm ceph.*

If you execute purge, you must re-install Ceph. The last rm command removes any files that were written out by ceph-deploy locally during a previous installation.

In our next article, I’ll cover how to store and retrieve objects on Ceph and configuring Ceph Clients.