How Disk Multipath works on Linux
Updated: Sep 5
Linux Multipath command is used to manage storage SAN (storage area network) disks on OS side. Linux multipath provides a way of organizing the I/O paths logically, by creating a single multipath device on top of the underlying devices.
Start multipath on Linux
We need to install the following two rpms.
start multipathd service with service command or systemctl command
service multipathd start
To make multipath work, we need make sure below module is loaded
# lsmod | grep dm_multipath dm_multipath 27427 4 dm_round_robin,dm_service_time # modinfo dm_multipath filename: /lib/modules/3.10.0-693.21.1.el7.x86_64/kernel/drivers/md/dm-multipath.ko.xz license: GPL author: Sistina Software <firstname.lastname@example.org> description: device-mapper multipath target retpoline: Y rhelversion: 7.4 srcversion: 063067F9F167E7B653A4773 depends: dm-mod intree: Y vermagic: 3.10.0-693.21.1.el7.x86_64 SMP mod_unload modversions signer: Red Hat Enterprise Linux kernel signing key sig_key: B1:C3:31:09:FB:DA:94:AD:2F:E1:E8:E5:C1:E5:52:BD:22:57:60:FE sig_hashalgo: sha256
List multipath devices on Linux
We can use this command to list all the multipath devices on Linux.
# multipath -l
Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging.
We can get the WWID using below commands.
# multipath -l | grep dm 3624a937079ebbf3c903141360001149b dm-5 PURE ,FlashArray 3624a937079ebbf3c903141360001149a dm-14 PURE ,FlashArray
OR using below command
# ls -ld /dev/disk/by-id/scsi-* lrwxrwxrwx 1 root root 9 May 22 18:29 /dev/disk/by-id/scsi-3600c0ff00013b88f9058e35a01000000 -> ../../sde lrwxrwxrwx 1 root root 9 May 22 18:29 /dev/disk/by-id/scsi-3600c0ff00013b88f9c58e35a01000000 -> ../../sdd
Get the WWID ( SCSI ID ) of disks on Linux
Each LUN is mapped to a disk on Linux. There is a unique WWID generated for each disk. This ID can be used for representing them in the configuration as the device name may change post reboot while the WWID will not change and are reboot persistent.
# lsscsi --scsi_id 3624a937079ebbf3c903141360001149c dm-2 PURE ,FlashArray size=1.0T features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=1 status=active |- 13:0:0:249 sdo 8:224 active ready running |- 11:0:0:249 sdr 65:16 active ready running |- 11:0:1:249 sdas 66:192 active ready running `- 13:0:1:249 sdan 66:112 active ready running
To get the WWID of the disks we can also use below command
# /usr/lib/udev/scsi_id -g -u -d /dev/sda 3600c0ff00013b88f9058e35a01000000
List of disks mapped via HBA on Linux box
A node with two HBAs attached to a storage controller with two ports sees four devices: /dev/sda, /dev/sdb, dev/sdc, and /dev/sdd. DM Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration.
In our case, we have 1 LUN on the storage side connected to the Linux box hence we see 4 devices
# ls -ld /sys/block/sd*/device lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sda/device -> ../../../0:0:0:0 lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdb/device -> ../../../0:0:0:1 lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdc/device -> ../../../0:0:1:0 lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdd/device -> ../../../0:0:1:1
we can also get the underlying disks this way.
# ls -l /dev/disk/by-path/ total 0 lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x207000c0ff13d3a7-lun-0 -> ../../sda lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x207000c0ff13d3a7-lun-1 -> ../../sdb lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x247000c0ff13d3a7-lun-0 -> ../../sdc lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x247000c0ff13d3a7-lun-1 -> ../../sdd
How multipath works in Linux?
In the multipath output, we can see that this device dm-9 has four underlying disks.
# multipath -l 3624a937079ebbf3c903141360001149c dm-9 PURE ,FlashArray size=1.0T features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=1 status=active |- 11:0:0:249 sdo 8:224 active ready running |- 11:0:1:249 sdao 66:128 active ready running |- 13:0:0:249 sdr 65:16 active ready running `- 13:0:1:249 sdas 66:192 active ready running
From the output, we can see that all the disks are active and ready. That means all the disks are available for IO on OS side. That also means that the storage we use is an active-active product. 11/13 means the HBA number on OS side. 0 1 means the controller ID on the storage side. 249 is the LUN ID on the storage side.
Below value explains what each field means
^ ^ ^ ^
| | | |
H C T L
Where H is the HBA number, C is the channel on the HBA, T is the SCSI target ID, and L is the LUN from the Storage.