Software RAID under Linux

Introduction

This tutorial is about the installation, setup and administration of a mdadm software RAID on Linux systems.

Preconditions

An installed Linux OS
A dedicated server with several drives
Root access

Step 1 – Preparations

First you should think about which RAID system you want to run. This depends on the target and how many drives are installed in the server itself.

Note: A RAID should not be seen as a data backup as it does not provide protection against data loss. It only increases the availability of the data.

Step 1.1 – RAID level selection

Choosing the right RAID level is not easy and depends on several factors:

How many drives does the server have?
What are your goals?
- More storage space / Less availability
- Higher availability / Less storage space

Here is a list of the most used RAID levels:

RAID0:

If there is a group of two or more partitions, the partitions are logically combined to one partition. Here the availability is reduced. If one of the drives is defective automatically all data is lost.

Pro
- Combines the available storage space.
- Increases drive performance.
Contra
- In case of a drive failure the data of all drives are lost.

RAID1:

If there is a group of two or more partitions, the data is mirrored on each partition.

Pro
- Increases the reliability / availability of the data.
- Increases the reading speed of the data.
Contra
- The available storage space is halved.

RAID5:

Is a group of three or more partitions, where the data is mirrored to two of the three partitions. So-called “parities” are stored on the third partition, with the help of which it is possible to recover data on defective drives in the RAID.

Learn more about RAID5

Pro
- Increased reliability / availability of data.
- Optimum storage utilization
- Increases the reading speed of the data.
Contra
- Less performance with write accesses

A list of additional RAID levels that are used less frequently:

Linear: Merge multiple partitions
Multipath: No RAID, but a mapping of a file to two different paths on the same partition (mirroring).
Faulty: emulates a faulty RAID system for test cases
Level 4: Like Level 0, but with an additional device for parity bits (increased reliability).
Level 6: Like Level 5 but with two independent parity bits per segment (increased reliability).

Step 1.2 – List of drives in the system

For a short and clear list of all available block devices, the command lsblk can be used. Here is a sample output:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 232.9G 0 disk 
├─sda1 8:1 0 10G 0 part /
├─sda2 8:2 0 208.1G 0 part /home/han/data
&lt;font color=#38B0DE&gt;-=─sda3=- Proudly Presents 
sr0 11:0 1 1024M 0 Rome

For a list with more detailed information, fdisk -l can be used. Here is a sample output:

Disk /dev/sda: 80.0 GByte, 80026361856 Byte
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinder of 16065 × 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x92669266

   Device boot.     Start End Blocks Id System
/dev/sda1 * 1 1305 10482381 83 Linux
/dev/sda2 1306 3977 21462840 83 Linux
/dev/sda3 4508 9729 41945715 83 Linux
/dev/sda4 3978 4507 4257225 82 Linux Swap / Solaris

Note: For a software RAID, it is not necessary to add the entire drive to the RAID. Single partitions are sufficient.

Step 2 – Create a software RAID

Step 2.1 – Preparing the drives

First the drives must be formatted accordingly.

Create a new, empty partition table on the drive:

For large 2 TB drives or PCs with UEFI:sudo parted /dev/sda mklabel gpt
For drives smaller than 2 TB and BIOS:sudo parted /dev/sda mklabel msdos

Create a partition on the drive:

sudo parted -a optimal -- /dev/sda mkpart primary 2048s -8192s

If you want to use the whole drive, just enter “0% 100%” instead of “2048s -8192s”.

Hint: 8192 sectors at the end of the drive are deliberately left unused to be prepared for failures. It allows you to use drives that have a few sectors less as replacements due to the space left free.

Mark the newly created partition as a RAID partition:

sudo parted /dev/sda set 1 raid on

Step 2.2 – Create the software RAID

Under Linux, mdadm is the main tool. It is the interface to the RAID functions of the kernel.

To create a RAID 1 the following command is sufficient:

sudo mdadm --create /dev/md0 --auto md --level=1 --raid-devices=2 /dev/sda1 /dev/sdb2

The following command is sufficient to create a RAID 5:

sudo mdadm --create /dev/md0 --auto md --level=5 --raid-devices=4 /dev/sda1 /dev/sdb2 /dev/sdc3 /dev/sdd4

The parameters in detail:

--create /dev/md0 – Creates a new endpoint with the name md0. If there are already endpoints with the same name, choose another free name (md1,md2, etc.).
--auto md – Creates a “classic” endpoint without pre-partitioning.
--level= – The type of RAID level.
--raid-devices – The number of single devices the RAID should consist of.
/dev/sde1 /dev/sde2 ... – The individual devices to be combined. The order of the identifiers, or ideally those of the corresponding physical devices, should be written down if the RAID has to be reassembled manually in an emergency.

The newly created block device mdX can be used immediately and the system can also be shut down or restarted during this time. The current status of the RAID creation can be queried with the following command:

watch cat /proc/mdstat

An exemplary edition:

Personalities : [raid0] [raid1]
mdX : active raid1 sda1[1] sdb1[0]
      8384448 blocks super 1.2 [2/2] [UU]
      [==============>.......] check = 74.7% (6267520/8384448) finish=0.1min speed=202178K/sec

Format the newly created RAID:

sudo mkfs.ext4 /dev/md0

Integration of the RAID:

sudo mount /dev/mdX /mnt/md0

Automatic integration of the RAID:

Here the corresponding line must be entered into the file /etc/fstab:

/dev/mdX /mnt/md0 xt4 defaults 0 2

(Optional) Hotspare drives: Hotspare drives/partitions are those which are not normally used. These are used if one of the active drives/partitions of the RAID system has an error or is defective. If no hot spare drive is defined in a software raid, the rebuild of a defective RAID must be started manually. If a hot spare is present, the rebuild will be started automatically. A hot spare drive can be added with the following command.

mdadm --add /dev/md/X /dev/sdX1

Step 3 – Removing a software RAID

To remove a software RAID, the following steps must be performed:

Stop the RAID:sudo umount /dev/md0 sudo mdadm --stop /dev/md0
Remove automatic mount entries (e.g. /etc/fstab)
Delete RAID entries in mdadm.conf.
Delete the superblock of the used partitions:sudo mdadm --zero-superblock /dev/sda1 /dev/sdb1
Disable RAID flag:sudo parted /dev/sda set 1 raid off

Step 4 – Managing a software RAID

Step 4.1 – Get RAID status

A short list of all RAIDs in the system can be obtained with the output of the file /proc/mdstat.

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdb1[1] sda1[0]
      8380416 blocks super 1.2 [2/2] [UU]
      
md2 : active raid1 sdb3[1] sda3[0]
      536739840 blocks super 1.2 [2/2] [UU]
      bitmap: 3/4 pages [12KB], 65536KB chunk

md1 : active raid1 sdb2[1] sda2[0]
      1047552 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>

A more exact output is done with the command:

sudo mdadm --detail /dev/md/2

Here is an example output:

/dev/md/2:
           Version : 1.2
     Creation Time : Fri Feb 22 17:19:37 2019
        Raid Level : raid1
        Array Size : 536739840 (511.88 GiB 549.62 GB)
     Used Dev Size : 536739840 (511.88 GiB 549.62 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Sun May 26 13:49:02 2019
             State : clean 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : rescue:2
              UUID : c76e5446:f3c41fbc:0b62ca86:803a7e88
            Events : 2170

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3

Step 4.2 – Replace defective drive

To do this, the defective drive must first be removed from the RAID:

sudo mdadm --manage /dev/md/0 -r /dev/sda1

If no hot spare drive is available, a new drive must be partitioned. It is important that the new drive has the same partitioning as the defective drive!

To partition the new drive, it is sufficient to copy the partition table from an existing drive.

For MBR partitioned drives:

sfdisk --dump /dev/sda > sda_parttable_mbr.bak # Backs up partition table
sfdisk -d /dev/sda | sfdisk /dev/sdb # Copy partition table from sda to sdb

For GPT partitioned drives:

sgdisk --backup=sda_parttable_gpt.bak /dev/sda # Create a backup of the partition table
sgdisk --load-backup=sda_parttable_gpt.bak /dev/sdb # Copy the created backup of the partition table to sdb

If the new drive is partitioned correctly, it can be added to the RAID system again:

sudo mdadm --manage /dev/md/0 -a /dev/sda1

In order to start the recovery process, the newly added drive must be given the status faulty:

sudo mdadm --manage --set-faulty /dev/md/0 /dev/sda1

The progress can be monitored again with the command watch cat /proc/mdstat.

Once the rebuild of the RAID is complete, the partition must be removed from the RAID and added again to remove the status “faulty”. This is done with the commands:

sudo mdadm --manage /dev/md/0 -r /dev/sda1 # To be removed
sudo mdadm --manage /dev/md/0 -a /dev/sda1 # To be added again

**Note: If the system is on the RAID itself, it is necessary to install the bootloader on the appropriate drive. This is done with the following command:

update grub && grub-install /dev/sda

Step 4.3 – Expand RAID

Only RAIDs with levels 1, 5 and 6 can be expanded.

The new partition must first be added as a hot spare:

sudo mdadm /dev/md0 --add /dev/sda1

Now the RAID can be extended with the new drive:

sudo mdadm --grow --raid-devices=5 /dev/md0 --backup-file=/root/md0.bak

Note: In the file specified by --backup-file critical areas are saved (typically a few MiB). If the system crashes during the extension, the extension can be continued later using the following command:

sudo mdadm /dev/md0 --continue --backup-file=/tmp/md0.bak

The backup file must not be located on the RAID to be extended! The use of backup-file is not mandatory, but strongly recommended.

The file system must still be extended so that the newly created storage space can be used. The extension takes place with the following commands:

sudo umount /dev/md0 /mnt    # Unmount the file system
sudo fsck.ext4 -f /dev/md0   # Force the check, even if it has recently been checked
sudo resize2fs /dev/md0      # Extend file system to maximum size
sudo mount /dev/md0 /mnt     # Mount the file system again

Step 4.4 – RAID monitors

To monitor the RAID, this entry can be stored as a crontab (sudo crontab -e):

0 0 * * * /usr/share/mdadm/checkarray --cron --all --quiet >/dev/null 2>&1 # Runs every day at 00:00 AM

Conclusion

In this article we reported how to select a suitable RAID level for your project and then configure it on Linux systems using mdadm. Furthermore, administrative tasks like expanding a RAID or exchanging defective hard disks will be discussed.

Reprint:https://community.hetzner.com/tutorials/howto-setup-mdadm