15 Linux multipath command examples

Updated: Dec 3, 2020

Linux Multipath command is used to manage storage SAN (storage area network) disks on OS side. Linux multipath provides a way of organizing the I/O paths logically, by creating a single multipath device on top of the underlying devices.


Without DM Multipath, each path from a server node to a storage controller is treated by the system as a separate device, even when the I/O path connects the same server node to the same storage controller.


We use our Pure Storage as an example.




How to make multipath work on Linux?


To make multipath work, we need make sure below module is loaded


# lsmod | grep dm_multipath

dm_multipath 27427 4 dm_round_robin,dm_service_time

# modinfo dm_multipath

filename: /lib/modules/3.10.0-693.21.1.el7.x86_64/kernel/drivers/md/dm-multipath.ko.xz

license: GPL

author: Sistina Software <dm-devel@redhat.com>

description: device-mapper multipath target

retpoline: Y

rhelversion: 7.4

srcversion: 063067F9F167E7B653A4773

depends: dm-mod

intree: Y

vermagic: 3.10.0-693.21.1.el7.x86_64 SMP mod_unload modversions

signer: Red Hat Enterprise Linux kernel signing key

sig_key: B1:C3:31:09:FB:DA:94:AD:2F:E1:E8:E5:C1:E5:52:BD:22:57:60:FE

sig_hashalgo: sha256


multipath needs these two rpms.

device-mapper-multipath

device-mapper


start multipathd service with service command or systemctl command





List the connected and scanned multipath devices on Linux


# multipath -l

Get the WWID of the connected/scanned LUN

Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging.


You can get the WWID using below commands


# multipath -l | grep dm

3624a937079ebbf3c903141360001149b dm-5 PURE ,FlashArray

3624a937079ebbf3c903141360001149a dm-14 PURE ,FlashArray

OR using below command


# ls -ld /dev/disk/by-id/scsi-*

lrwxrwxrwx 1 root root 9 May 22 18:29 /dev/disk/by-id/scsi-3600c0ff00013b88f9058e35a01000000 -> ../../sde

lrwxrwxrwx 1 root root 9 May 22 18:29 /dev/disk/by-id/scsi-3600c0ff00013b88f9c58e35a01000000 -> ../../sdd


2 ways to Get the WWID ( SCSI ID ) of the disks on Linux

Each LUN is mapped to a disk for which also a unique WWID is generated and the same can be used for representing them in the configuration as the device name may change post reboot while the WWID will not change and are reboot persistent.


# lsscsi --scsi_id

3624a937079ebbf3c903141360001149c dm-2 PURE ,FlashArray

size=1.0T features='0' hwhandler='0' wp=rw

`-+- policy='queue-length 0' prio=1 status=active

|- 13:0:0:249 sdo 8:224 active ready running

|- 11:0:0:249 sdr 65:16 active ready running

|- 11:0:1:249 sdas 66:192 active ready running

`- 13:0:1:249 sdan 66:112 active ready running


To get the WWID of the disks you can also use below command


# /usr/lib/udev/scsi_id -g -u -d /dev/sda

3600c0ff00013b88f9058e35a01000000


2 ways to list of disks mapped via HBA on Linux box


A node with two HBAs attached to a storage controller with two ports by means of a single unzoned FC switch sees four devices: /dev/sda, /dev/sdb, dev/sdc, and /dev/sdd. DM Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration.


For our case since we have 1 LUN connected to the Linux box hence we see 4 devices


# ls -ld /sys/block/sd*/device

lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sda/device -> ../../../0:0:0:0

lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdb/device -> ../../../0:0:0:1

lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdc/device -> ../../../0:0:1:0

lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdd/device -> ../../../0:0:1:1


The same can be seen via below command where LUN to device mapping does easier on the eyes


# ls -l /dev/disk/by-path/

total 0

lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x207000c0ff13d3a7-lun-0 -> ../../sda

lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x207000c0ff13d3a7-lun-1 -> ../../sdb

lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x247000c0ff13d3a7-lun-0 -> ../../sdc

lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x247000c0ff13d3a7-lun-1 -> ../../sdd


How does Device ID work in multipath?

In multipath output we see below highlighted entry being used (and also in the commands above you would see similar representation)


# multipath -l

3624a937079ebbf3c903141360001149c dm-9 PURE ,FlashArray

size=1.0T features='0' hwhandler='0' wp=rw

`-+- policy='queue-length 0' prio=1 status=active

|- 11:0:0:249 sdo 8:224 active ready running

|- 11:0:1:249 sdao 66:128 active ready running

|- 13:0:0:249 sdr 65:16 active ready running

`- 13:0:1:249 sdas 66:192 active ready running


Below value explains what each field means


1:0:0:1

^ ^ ^ ^

| | | |

H C T L

Where H is the HBA number, C is the channel on the HBA, T is the SCSI target ID, and L is the LUN from the Storage.


Flush/Remove all the multipath device on Linux

# multipath -F


Flush/Remove selected multipath device

First of all the the WWID of the multipath device which you want to flush using multipath -ll


# multipath -ll

3600c0ff00013b88fdc53e35a01000000 dm-0 PURE ,P2000 G3 FC

size=186G features='1 queue_if_no_path' hwhandler='0' wp=rw

|-+- policy='service-time 0' prio=50 status=active

| |- 1:0:0:0 sda 8:0 active ready running

| `- 2:0:1:0 sdd 8:48 active ready running

`-+- policy='service-time 0' prio=10 status=enabled

|- 1:0:1:0 sdb 8:16 active ready running

`- 2:0:0:0 sdc 8:32 active ready running

Next flush the selected multipath using below command


# multipath -f 3600c0ff000144876065be35a01000000

NOTE: This multipath must not be in used state


Show the existing multipath configuration in detail on Linux


Below command will show the existing configuration loaded on your Linux box

# multipathd show config

defaults {

verbosity 2

polling_interval 5

max_polling_interval 20

reassign_maps "yes"

multipath_dir "/lib64/multipath"

path_selector "service-time 0"

path_grouping_policy "failover"

uid_attribute "ID_SERIAL"

prio "const"

prio_args ""

features "0"

path_checker "directio"

alias_prefix "mpath"

failback "immediate"

rr_min_io 1000

rr_min_io_rq 1

max_fds 1048576

rr_weight "uniform"

queue_without_daemon "no"

flush_on_last_del "no"

user_friendly_names "no"

fast_io_fail_tmo 5

bindings_file "/etc/multipath/bindings"

wwids_file /etc/multipath/wwids

prkeys_file /etc/multipath/prkeys

log_checker_err always

find_multipaths no

retain_attached_hw_handler no

detect_prio no

detect_path_checker no

hw_str_match no

force_sync no

deferred_remove no

ignore_new_boot_devs no

skip_kpartx no

config_dir "/etc/multipath/conf.d"

delay_watch_checks no

delay_wait_checks no

retrigger_tries 3

retrigger_delay 10

missing_uev_wait_timeout 30

new_bindings_in_boot no

remove_retries 0

disable_changed_wwids no

unpriv_sgio no

}

blacklist {

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"

devnode "^hd[a-z][[0-9]*]"

devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"

devnode "^(td|hd|vd)[a-z]"

devnode "^dcssblk[0-9]*"

device {

vendor "DGC"

product "LUNZ"

}

device {

vendor "EMC"

product "LUNZ"

}

device {

vendor "IBM"

product "Universal Xport"

}


}

}

multipaths {

}



How to Get complete HBA details on Linux


systool uses APIs provided by libsysfs to gather information and will give detailed information about the available HBA and the LUN status


NOTE: systool is delivered with sysfsutils rpm so make sure it is installed before using this tool


# systool -c fc_host -v

Class = "fc_host"

Class Device = "host0"

Class Device path = "/sys/devices/pci0000:00/0000:00:02.0/0000:04:00.2/host0/fc_host/host0"

active_fc4s = "0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 "

dev_loss_tmo = "30"

fabric_name = "0x2003547fee1188d9"

issue_lip = <store method only>

max_npiv_vports = "255"

maxframe_size = "2048 bytes"

node_name = "0x50060b0000c2a67d"

npiv_vports_inuse = "0"

port_id = "0x310305"

port_name = "0x50060b0000c2a67c"

port_state = "Online"

port_type = "NPort (fabric via point-to-point)"

speed = "unknown"

supported_classes = "Class 3"

supported_fc4s = "0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 "

supported_speeds = "10 Gbit"

symbolic_name = "Emulex 554FLB FV11.1.183.23 DV11.2.0.6 HN:ban33-be002-2b OS:Linux"

tgtid_bind_type = "wwpn (World Wide Port Name)"

uevent =

vport_create = <store method only>

vport_delete = <store method only>


Device = "host0"

Device path = "/sys/devices/pci0000:00/0000:00:02.0/0000:04:00.2/host0"

uevent = "DEVTYPE=scsi_host"


Class Device = "host1"

Class Device path = "/sys/devices/pci0000:00/0000:00:02.0/0000:04:00.3/host1/fc_host/host1"

active_fc4s = "0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 "

dev_loss_tmo = "30"

fabric_name = "0x2003547fee357a81"

issue_lip = <store method only>

max_npiv_vports = "255"

maxframe_size = "2048 bytes"

node_name = "0x50060b0000c2a67f"

npiv_vports_inuse = "0"

port_id = "0x25030f"

port_name = "0x50060b0000c2a67e"

port_state = "Online"

port_type = "NPort (fabric via point-to-point)"

speed = "unknown"

supported_classes = "Class 3"

supported_fc4s = "0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 "

supported_speeds = "10 Gbit"

symbolic_name = "Emulex 554FLB FV11.1.183.23 DV11.2.0.6 HN:ban33-be002-2b OS:Linux"

tgtid_bind_type = "wwpn (World Wide Port Name)"

uevent =

vport_create = <store method only>

vport_delete = <store method only>


Device = "host1"

Device path = "/sys/devices/pci0000:00/0000:00:02.0/0000:04:00.3/host1"

uevent = "DEVTYPE=scsi_host"


This tool can also be used to get other values using below syntax


How to get the HBA port name on Linux


# systool -c fc_host -A port_name

Class = "fc_host"

Class Device = "host0"

port_name = "0x50060b0000c2a67c"

Device = "host0"


Class Device = "host1"

port_name = "0x50060b0000c2a67e"

Device = "host1"


To get the port id


# systool -c fc_host -A port_id

Class = "fc_host"

Class Device = "host0"

port_id = "0x310305"


Device = "host0"


Class Device = "host1"

port_id = "0x25030f"


Device = "host1"


Show the existing multipath configuration in brief


This is an alternate command which can be used to check the loaded configuration for selected values as shown below.

# mpathconf

multipath is enabled

find_multipaths is disabled

user_friendly_names is enabled

dm_multipath module is loaded

multipathd is running

Using user friendly names for multipath

Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging. By default, the name of a multipath device is set to its WWID. Alternately, you can set the user_friendly_names option in the multipath configuration file, which sets the alias to a nodeunique name of the form mpathn


With user_friendly name disabled we will only see the WWID of the multipath connected


# multipath -l

3624a937079ebbf3c903141360001149c dm-2 PURE ,FlashArray

size=1.0T features='0' hwhandler='0' wp=rw

`-+- policy='queue-length 0' prio=1 status=active

|- 13:0:0:249 sdo 8:224 active ready running

|- 11:0:0:249 sdr 65:16 active ready running

|- 11:0:1:249 sdas 66:192 active ready running

`- 13:0:1:249 sdan 66:112 active ready running


Next enable user_friendly name


# /sbin/mpathconf --user_friendly_names y

Check the multipath


# multipath -l

mpathb (3600c0ff000144876065be35a01000000) dm-6 HP ,P2000 G3 FC

size=359G features='1 queue_if_no_path' hwhandler='0' wp=rw

|-+- policy='service-time 0' prio=0 status=active

| `- 2:0:0:1 sdb 8:16 active undef unknown

`-+- policy='service-time 0' prio=0 status=enabled

`- 2:0:1:1 sdd 8:48 active undef unknown


The mapping is done using /etc/multipath/bindings.


how to Check the HBA port name on your Linux box

# ls -l /sys/class/fc_host/

total 0

lrwxrwxrwx 1 root root 0 May 22 18:29 host0 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.2/host0/fc_host/host0

lrwxrwxrwx 1 root root 0 May 22 18:29 host1 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.3/host1/fc_host/host1

You can also check the same using


# lspci -nn |grep -i "Fibre"

04:00.2 Fibre Channel [0c04]: Emulex Corporation OneConnect 10Gb FCoE Initiator (be3) [19a2:0714] (rev 01)

04:00.3 Fibre Channel [0c04]: Emulex Corporation OneConnect 10Gb FCoE Initiator (be3) [19a2:0714] (rev 01)

On some Linux variant you can also grep for HBA


# lspci -nn |grep -i "HBA"

So here I have two HBA


How to Get the HBA WWPN on Linux?

A worldwide port number (WWPN) is unique identifier for each FC port of any Fibre Channel device.


For server, we have a WWPN for each port of the HBA.

For a SAN switch, the WWPN is available for each port in the chassis.

For Storage, each host port has an individual WWPN

Here we have two HBA so each HBA will have unique World Wide Port Name (WWPN) which will be connected to the SAN switch then to the Storage Box.


From my Linux Box


# cat /sys/class/fc_host/host0/port_name

0x50060b0000c2a67c

# cat /sys/class/fc_host/host1/port_name

0x50060b0000c2a67e


These are the WWPN number.


How to Get the HBA WWNN on Linux?

A worldwide node name (WWNN) is a globally unique 64-bit identifier that is assigned to each Fibre Channel node or device.


For servers and hosts, WWNN is unique for each HBA (host bus adapter), and in a case of a server with two HBAs, they have two WWNNs.

For a SAN switch, the WWNN is a common for the chassis.

For storage, the WWNN is common for each controller unit of midrage storage

Since here I have two HBA hence there are two WWNN for each HBA


From my Linux Box


# cat /sys/class/fc_host/host1/node_name

0x50060b0000c2a67f

# cat /sys/class/fc_host/host0/node_name

0x50060b0000c2a67d


How to rescan scsi bus to get the new LUNs on Linux?

If you have new LUNs connected to your Linux box then you will need to rescan the HBA


Assuming you know the WWPN for which the new LUN was added


# echo 1 > /sys/class/fc_host/host2/issue_lip


# systemctl reload multipathd


# multipath -v2

Next validate if the new LUN is visible using


# multipath -ll



How to Blacklist a device in multipath on Linux?

Create a function like below in /etc/multipath.conf where you can give the list of devices which you wish to blacklist


Here I am blacklisting multiple devices which I do not wish to be created as multipath


blacklist {

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"

devnode "^hd[a-z][[0-9]*]"

devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"

}

To blacklist using WWID


You can get the WWID of a device using below command


# /usr/lib/udev/scsi_id -g -u -d /dev/sda

3600c0ff000144876f85ae35a01000000

Here we are blacklisting /dev/sda which is our internal disk and I donot want it to be a multipath


blacklist {

wwid 3600c0ff000144876f85ae35a01000000

}

Perform a multipathd reload to take the changes affect


# systemctl reload multipathd.service

Add blacklist exception for selected disks

Similar to blacklisting we can also add a blacklist exception so that these devices are not blacklisted


Add below function in /etc/multipath.conf with the list of wwid which you wish to add as an exception for blacklist


blacklist_exceptions {

wwid "3600d0230000000000e13955cc3757803"

}

Followed by reload of multipathd


# systemctl reload multipathd.service



How to add path_selector in multipath?


Add your choice of path_selector in /etc/multipath.conf as shown below


defaults {

user_friendly_names yes

failback immediate

path_selector "round-robin 0"

}

Either you add it under default function or you can add a device function if you wish to add to change path selector only for a selected device


devices {

device {

vendor "HP"

product "P2000 G3 FC|P2000G3 FC/iSCSI|P2000 G3 SAS|P2000 G3 iSCSI"

path_grouping_policy "failover"

path_selector "round-robin 0"

}

}

NOTE: You can get the device detail from "multipathd show config"


124 views

Join our newsletter. Get a free Linux account on Cloud.

Get a Free Cloud Server! 

We can use this cloud server to practice Linux commands. Never miss a post!

Thanks for submitting!