Storage Management in Linux
Liju Gopinath - RHCA, RHCSS, RHCDS, RHCVA, RHCSA, RHCE.
Subject Matter Expert - Linux
NAS Vs. SAN
NAS Vs SAN
NAS provides file level access
NAS appears to the client OS as a file server (the client can map network
drives to shares on that server)
SAN provides block level access
Whereas a disk available through a SAN still appears to the client OS as a disk,
visible in disk and volume management utilities (along with client's local
disks), and available to be formatted with a file system and mounted.
Storage Management in Linux
[ Presented by Liju Gopinath ]
Storage Management in Linux
[ Presented by Liju Gopinath ]
Storage Management in Linux
[ Presented by Liju Gopinath ]
Storage Management in Linux
[ Presented by Liju Gopinath ]
iSCSI
iSCSI
iSCSI is Internet SCSI (Small Computer System Interface), an Internet
Protocol (IP)-based storage networking standard for linking data
storage facilities by carrying SCSI commands over IP networks.
We can mount block devices (disks) across an IP network to local
system and then use them like any other block device. iSCSI is a
client-server protocol.
By default iSCSI will listen to the port 3260
Storage Management in Linux
[ Presented by Liju Gopinath ]
iSCSI Target & Initiator
The server-side is referred to as the 'target'
The client-side is referred to as the 'initiator'
Storage Management in Linux
[ Presented by Liju Gopinath ]
Configuring iSCSI server
Install and enable the iSCSI target services.
#
#
#
#
yum -y install scsi-target-utils
service tgtd start
service tgtd stop
chkconfig tgtd on
Storage Management in Linux
[ Presented by Liju Gopinath ]
10
/etc/tgt/targets.conf
<target iqn.2014-01.com.example:vmrh5.target1>
backing-store /dev/VG00/disk1
backing-store /dev/VG00/disk2
backing-store /dev/VG00/disk3
backing-store /dev/VG00/disk4
backing-store /dev/VG00/disk5
</target>
Storage Management in Linux
[ Presented by Liju Gopinath ]
11
Configuring iSCSI client
Install iSCSI client package
# yum y install iscsi-initiator-utils
Start iSCSI services in the client
#
#
#
#
service iscsi start
service iscsid start
chkconfig iscsi on
chkconfig iscsid on
Storage Management in Linux
[ Presented by Liju Gopinath ]
12
Discover the target LUNs
Discover the target LUNs exported by server using following
command.
It will provide iqn name with of LUN associated with given ip
address
# iscsiadm -m discovery -t sendtargets -p 192.168.0.9
Storage Management in Linux
[ Presented by Liju Gopinath ]
13
iSCSI Discovery Configuration
The default iSCSI configuration file is /etc/iscsi/iscsid.conf.
This file contains iSCSI settings used by iscsid and iscsiadm.
Node records in /var/lib/iscsi/nodes:
When logging into a target, iscsiadm uses the settings in this file.
Discovery records in /var/lib/iscsi/send_targets:
When performing discovery to same destination, iscsiadm uses the
settings in this file.
Storage Management in Linux
[ Presented by Liju Gopinath ]
14
Connecting to iSCSI target
To connect iSCSI target, we need to mention iscsi servers ip address
and iqn name.
# iscsiadm -m node -T iqn.2014-01.com.example:vmrh5.target1 p
192.168.0.9 l
To disconnect the iSCSI Target, you will need to log out:
# iscsiadm -m node -T iqn.2014-01.com.example:vmrh5.target1 -p
192.168.0.9 -u
Storage Management in Linux
[ Presented by Liju Gopinath ]
15
Create File System and Mount iSCSI LUN
Create filesystem
# mkfs.ext3 /dev/sdb
/etc/fstab:
/dev/sdb /iscsi_test ext3 _netdev 0 0
Use _netdev parameter while adding entry in/etc/fstab, which is
used to prevent the system from attempting to mount the file
systems until the network, has been enabled on the system.
Storage Management in Linux
[ Presented by Liju Gopinath ]
16
Logical Volume Manager
What is LVM?
Logical volume management provides a higher-level view of the
disk storage on a computer system than the traditional view of disks
and partitions.
This gives the system administrator much more flexibility in
allocating storage to applications and users.
Storage volumes created under the control of the logical volume
manager can be resized and moved around almost at will, although
this may need some upgrading of file system tools.
The LVM is structured in three elements:
Volumes: physical volume, logical volumes and volume groups
Extents: physical and logical extents
Device mapper: the Linux kernel module
Storage Management in Linux
[ Presented by Liju Gopinath ]
18
Is your system ready for LVM2?
Verify whether LVM2 package is installed.
# lsmod | grep dm_mod
dm_mod
81500 11
dm_mirror,dm_log
Load the Device-mapper module if not loaded already.
# modprobe dm_mod
Storage Management in Linux
[ Presented by Liju Gopinath ]
19
Storage Management in Linux
[ Presented by Liju Gopinath ]
20
Storage Management in Linux
[ Presented by Liju Gopinath ]
21
Physical Extents (PE ) and Logical Extents (LE)
Storage Management in Linux
[ Presented by Liju Gopinath ]
22
Each PV consists of a number of fixed-size physical extents (PEs);
similarly, each LV consists of a number of fixed-size logical extents
(LEs).
LEs and PEs are always the same size, the default in LVM 2 is 4 MB.
An LV is created by mapping logical extents to physical extents, so
that references to logical block numbers are resolved to physical
block numbers.
When the logical volume is created a mapping is defined between
logical extents and physical extents,
e.g. logical extent 1 could map onto physical extent 51 of PV01,
data written to the first 4 MB of the logical volume in fact be
written to the 51st extent of PV01.
Storage Management in Linux
[ Presented by Liju Gopinath ]
23
Logical Extend and Physical Extend mapping.
Through these different types of logical-to-physical mappings, LVM
can achieve four important advantages over raw physical partitions:
Logical volumes can be resized while they are mounted and accessible by the
database or file system, removing the downtime associated with adding or
deleting storage from a Linux server.
Data from one (potentially faulty or damaged) physical device may be
relocated to another device that is newer, faster or more resilient, while the
original volume remains online and accessible.
Logical volumes can be constructed by aggregating physical devices to
increase performance (via disk striping) or redundancy (via disk mirroring and
I/O multipathing).
Logical volume snapshots can be created to represent the exact state of the
volume at a certain point-in-time, allowing accurate backups to proceed
simultaneously with regular system operation.
Storage Management in Linux
[ Presented by Liju Gopinath ]
24
Mapping modes (linear/striped)
Linear mapping will assign a range of PE's to an area of an LV in
order e.g.,
LE 1 - 99 map to PV01 and LE 100 - 347 map onto PV02.
Striped mapping will interleave the chunks of the logical extents
across a number of physical volumes eg.,
1st chunk of LE[1] -> PV01[1],
2nd chunk of LE[1] -> PV02[1],
3rd chunk of LE[1] -> PV03[1],
4th chunk of LE[1] -> PV01[2],
Storage Management in Linux
[ Presented by Liju Gopinath ]
25
LVM Metadata
The configuration details of a volume group are referred to as the
metadata.
By default, an identical copy of the metadata is maintained in every
metadata area in every physical volume within the volume group.
LVM volume group metadata is stored as ASCII
By default, the pvcreate command places the physical volume label
in the 2nd 512-byte sector.
This label can optionally be placed in any of the first four sectors,
since the LVM tools that scan for a physical volume label check the
first 4 sectors.
The volume group metadata contains:
Information about how and when it was created
Information about the volume group:
Storage Management in Linux
[ Presented by Liju Gopinath ]
26
Metadata backups and archives are automatically created on every
volume group and logical volume configuration change unless
disabled in the lvm.conf file.
By default, the metadata backup is stored in the /etc/lvm/backup
directory and the metadata archives are stored in the
/etc/lvm/archive directory.
You can manually back up the metadata to the /etc/lvm/backup file
with the vgcfgbackup command.
You can restore metadata with the vgcfgrestore command.
Storage Management in Linux
[ Presented by Liju Gopinath ]
27
LVM snapshot
Snapshot creates exact copy of a logical volume, frozen at some
point in time.
Useful when taking backup.
LVM1 has read-only snapshots.
In LVM2, snapshots are read/write by default.
The snapshot is really nothing more than asking the LVM to give
you a "pointer" to the current state of the filesystem and to write
changes made after the snapshot to a designated area.
LVM snapshots are your basic "copy on write" snapshot solution.
# lvcreate --size 100M --snapshot --name snapshot /dev/VG01/lvol1
Storage Management in Linux
[ Presented by Liju Gopinath ]
28
copy on write
As soon as you create a snapshot, LVM creates a pool of blocks.
This pool also contains a full copy of the LVM metadata of the
volume.
When writes happen to the main volume such as updating an
inode, the block being overwritten is copied to this new pool and
the new block is written to the main volume. This is the 'copy-onwrite'.
Because of this, the more data that gets changed between when a
snapshot was taken and the current state of the main volume, the
more space will get consumed by that snapshot pool.
Storage Management in Linux
[ Presented by Liju Gopinath ]
29
device-mapper
The Device Mapper is a kernel driver that provides a framework for
volume management.
It provides a generic way of creating mapped devices, which may be
used as logical volumes.
It does not specifically know about volume groups or metadata
formats.
In addition to LVM, Device-Mapper multipath and the dmraid
command use the Device Mapper.
The user interface is the dmsetup command.
LVM logical volumes are activated using the Device Mapper.
Each logical volume is translated into a mapped device.
The Device Mapper supports a variety of mapping targets, including
linear mapping, striped mapping, and error mapping.
Storage Management in Linux
[ Presented by Liju Gopinath ]
30
device-mapper files
If VG0 and LV0 are the volume group and logical volume, the
Device mapper would create the following device nodes in the
/dev filesystem:
/dev/mapper/VG0-LV0
/dev/VG0/LV0 (is a link to the above)
/dev/mapper/VG0-LV1
/dev/VG0/LV1 (is a link to the above)
Storage Management in Linux
[ Presented by Liju Gopinath ]
31
The LVM Configuration Files
Central configuration file read by the tools.
/etc/lvm/lvm.conf
Device name filter cache file (configurable).
/etc/lvm/.cache
Directory for automatic volume group metadata backups
(configurable).
/etc/lvm/backup/
Directory for automatic volume group metadata archives
(configurable).
/etc/lvm/archive/
In single-host configuration, lock files to prevent parallel tool runs
from corrupting the metadata; in a cluster, cluster-wide DLM is
used.
/var/lock/lvm/
Storage Management in Linux
[ Presented by Liju Gopinath ]
32
LVM Locking
Non clustered LVM
# grep locking_type /etc/lvm/lvm.conf
locking_type = 1
For CLVM
# grep locking_type /etc/lvm/lvm.conf
locking_type = 3
Storage Management in Linux
[ Presented by Liju Gopinath ]
33
LVM commands
pvcreate
vgcreate
lvcreate
pvdisplay
vgdisplay
lvdisplay
pvscan
vgscan
lvscan
vgextend
lvextend
Storage Management in Linux
[ Presented by Liju Gopinath ]
34
Physical Volume
# pvcreate /dev/sda
/dev/sdb
OR if you are using multipath
# pvcreate /dev/mapper/mpatha /dev/mapper/mpathb
# pvscan
# pvdisplay
Storage Management in Linux
[ Presented by Liju Gopinath ]
35
Volume Group
# vgcreate VG01 /dev/sda /dev/sdb
# vgscan
# vgdisplay
# pvcreate /dev/sdc
# vgextend VG01 /dev/sdc
Storage Management in Linux
[ Presented by Liju Gopinath ]
36
Logical Volume
#
#
#
#
#
#
lvcreate -l 10G -n LOGVOL01 VG01
lvscan
lvdisplay
mkfs.ext3 /dev/VG01/LOGVOL01
lvextend -L100 /dev/VG01/LOGVOL01
lvextend -L+100 /dev/VG01/LOGVOL01
Storage Management in Linux
[ Presented by Liju Gopinath ]
37
pvdisplay
# pvdisplay
--- Physical volume --PV Name
/dev/sda2
VG Name
VG00
PV Size
14.75 GiB / not usable 5.00 MiB
Allocatable
yes (but full)
PE Size
32.00 MiB
Total PE
472
Free PE
0
Allocated PE
472
PV UUID
4sGBhJ-rtRF-2x6A-OTdA-DFS8-0zP9-eZfk2c
Storage Management in Linux
[ Presented by Liju Gopinath ]
38
vgdisplay
# vgdisplay vg01
--- Volume group --VG Name
System ID
Format
Metadata Areas
Metadata Sequence No
VG Access
VG Status
MAX LV
Cur LV
Open LV
Max PV
Cur PV
Act PV
VG Size
PE Size
Total PE
Alloc PE / Size
Free PE / Size
VG UUID
vg01
lvm2
2
1
read/write
resizable
0
0
0
0
2
2
19.99 GiB
4.00 MiB
5118
0 / 0
5118 / 19.99 GiB
e13j0O-b3VG-C4MS-CwZW-lDbw-weQt-Q7G02R
Storage Management in Linux
[ Presented by Liju Gopinath ]
39
lvdisplay
# lvdisplay |more
--- Logical volume --LV Name
/dev/vg02/lvol1
VG Name
vg02
LV UUID
jDmh8L-0pjd-fxbj-yypH-cMRJ-G7x5-Xcdj2Y
LV Write Access
read/write
LV Status
available
# open
0
LV Size
4.00 GiB
Current LE
1024
Segments
1
Allocation
inherit
Read ahead sectors
auto
- currently set to
256
Block device
253:4
Storage Management in Linux
[ Presented by Liju Gopinath ]
40
PVS, VGS, LVS
[root@vmrh5cn01 ~]#
PV
VG
/dev/sda
/dev/sdb
/dev/hda2 VG00
pvs
Fmt
lvm2
lvm2
lvm2
Attr
aaa-
PSize PFree
10.00G 10.00G
10.00G 10.00G
14.88G
0
[root@vmrh5cn01 ~]# vgs
VG
#PV #LV #SN Attr
VSize VFree
VG00
0 wz--n- 14.88G
1
3
0
VG01
2
0
0 wz--n- 19.99G 19.99G
[root@vmrh5cn01 ~]# lvs
LV
VG
Attr
LSize Origin Snap%
home VG00 -wi-ao 2.00G
root VG00 -wi-ao 10.88G
swap VG00 -wi-ao 2.00G
lvol1 VG01 -wi-a- 1.00G
Move Log Copy%
Storage Management in Linux
[ Presented by Liju Gopinath ]
Convert
41
dmsetup
The dmsetup command is a command line wrapper for
communication with the Device Mapper.
For general system information about LVM devices, you may find
the info, ls, status, and deps options of the dmsetup command to
be useful, as shown below.
Storage Management in Linux
[ Presented by Liju Gopinath ]
42
Device Information
[root@vmrh5 ~]# dmsetup info /dev/mapper/VG01-lvol1
Name: VG01-lvol1
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 252, 8
Number of targets: 1
UUID: LVMAbvIw8a9BeN19rTvSIfkT9wpPu7bnhqH78OyjJPRdUOLJwx0dQLwpSeSIVcVmAad
Storage Management in Linux
[ Presented by Liju Gopinath ]
43
[root@vmrh5 ~]# dmsetup ls
mpath2 (252, 5)
mpath1 (252, 4)
VG00-home (252, 1)
mpath0 (252, 3)
VG00-swap (252, 2)
VG00-root (252, 0)
VG01-lvol1 (252, 8)
[root@vmrh5 ~]# dmsetup ls --target linear
VG00-home (252, 1)
VG00-swap (252, 2)
VG00-root (252, 0)
VG01-lvol1 (252, 8)
Storage Management in Linux
[ Presented by Liju Gopinath ]
44
Fibre Channel
Fibre Channel
FC is more likely to be used as a SCSI network. The simplest use of
Fibre Channel is to replace external SCSI-attached storage units on a
single server.
This involves one HBA, some FC-storage units and perhaps an FC
hub for making the cabling simpler.
Storage Management in Linux
[ Presented by Liju Gopinath ]
46
Storage Management in Linux
[ Presented by Liju Gopinath ]
47
Storage Management in Linux
[ Presented by Liju Gopinath ]
48
Storage Management in Linux
[ Presented by Liju Gopinath ]
49
Storage Management in Linux
[ Presented by Liju Gopinath ]
50
Storage Management in Linux
[ Presented by Liju Gopinath ]
51
WWID
The World Wide Identifier (WWID) can be used in reliably
identifying devices.
It is a persistent, system-independent ID that the SCSI Standard
requires from all SCSI devices.
The WWID identifier is guaranteed to be unique for every storage
device, and independent of the path that is used to access the
device.
WWNN (World Wide Node Number)
WWPN (World Wide Port Number)
Storage Management in Linux
[ Presented by Liju Gopinath ]
52
H:B:T:L Values
/sys/class/scsi_host/host/device/targetH:B:T:L/H:B:T:L/
Where:
H is the host number of the HBA
B is the bus number (normally 0)
T is the target number
L is the LUN number
Storage Management in Linux
[ Presented by Liju Gopinath ]
53
Scanning LUN
Scanning new LUNs without reboot.
# echo "- - -" > /sys/class/scsi_host/hostX/scan
# echo "1" > /sys/class/fc_host/hostX/issue_lip
Storage Management in Linux
[ Presented by Liju Gopinath ]
54
device-mapper-multipathing
device-mapper-multipathing
The connection from the server through the HBA to the storage
controller is referred as a path.
When multiple paths exist to a storage device (LUN) on a storage
subsystem, it is referred as multipath connectivity.
Main purpose of multipath connectivity is to provide redundant
access to the storage devices.
Another advantage of multipathing is the increased throughput by
way of load balancing.
Storage Management in Linux
[ Presented by Liju Gopinath ]
56
DM-MP consists of 4 components
DM MP kernel module - Kernel module that is responsible for
making the multipathing decisions in normal and failure situations.
multipath command - User space tool that allows the user with
initial configuration, listing and deletion of multipathed devices.
multipathd daemon - User space daemon that constantly monitors
the paths. It marks a path as failed when it finds the path faulty and
if all the paths in a priority group are faulty then it switches to the
next enable priority group. It keeps checking the failed path, once
the failed path comes alive, based on the failback policy, it can
activate the path. It automatically creates device mapper entries
when a new device comes into existence.
kpartx - User space command that creates device mapper entries
for all the partitions in a multipathed disk/LUN. When the multipath
command is invoked, this command automatically gets invoked. For
DOS based partitions this command need to be run manually.
Storage Management in Linux
[ Presented by Liju Gopinath ]
57
Active/Passive Multipath Configuration
Storage Management in Linux
[ Presented by Liju Gopinath ]
58
Active / Active Multipath Configuration
Storage Management in Linux
[ Presented by Liju Gopinath ]
59
Storage Management in Linux
[ Presented by Liju Gopinath ]
60
Storage Management in Linux
[ Presented by Liju Gopinath ]
61
/etc/multipath.conf
[root@vmrh5 ~]# egrep -v '^#|^$' /etc/multipath.conf
defaults {
find_multipaths yes
user_friendly_names yes
path_grouping_policy
multibus
path_selector
"service-time 0"
}
blacklist {
}
Storage Management in Linux
[ Presented by Liju Gopinath ]
62
# multipath -ll
mpath0 (1IET
00010001) dm-4 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 4:0:0:1 sdb 8:16 active ready running
`- 5:0:0:1 sdf 8:80 active ready running
mpath1 (1IET
00010002) dm-3 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 4:0:0:2 sdc 8:32 active ready running
`- 5:0:0:2 sdg 8:96 active ready running
Here we have two multipath devices, mpath0, and mpath1.
The device mpath1 includes sdc & sdg.
These two sd devices, each correspond to a path.
As shown above the system has two paths to the iSCSI storage so two sd
devices are listed.
The IO policy is sent to round-robin which means that IO is sent
down each of the two paths in a round-robin fashion.
Storage Management in Linux
[ Presented by Liju Gopinath ]
63
Standard output of multipath command
[root@vmrh5 ~]# multipath -ll
mpath2 (1IET_00010003) dm-5 IET,VIRTUAL-DISK
[size=10G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=2][enabled]
\_ 1:0:0:3 sdc 8:32 [active][ready]
\_ 2:0:0:3 sdg 8:96 [active][ready]
mpath1 (1IET_00010002) dm-4 IET,VIRTUAL-DISK
[size=10G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=2][enabled]
\_ 1:0:0:2 sdb 8:16 [active][ready]
\_ 2:0:0:2 sdf 8:80 [active][ready]
mpath0 (1IET_00010001) dm-3 IET,VIRTUAL-DISK
[size=10G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=2][enabled]
\_ 1:0:0:1 sda 8:0
[active][ready]
\_ 2:0:0:1 sde 8:64 [active][ready]
mpath3 (1IET_00010004) dm-6 IET,VIRTUAL-DISK
[size=10G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=2][enabled]
\_ 1:0:0:4 sdd 8:48 [active][ready]
\_ 2:0:0:4 sdh 8:112 [active][ready]
Storage Management in Linux
[ Presented by Liju Gopinath ]
64
mpatha (1IET
00010002) dm-3 IET, VIRTUAL-DISK
------ ----------------- ---- --- ------------|
|
|
|
|------->
|
|
|
|------------------>
|
|
|----------------------->
|
|--------------------------------->
|------------------------------------------------->
Product
Vendor
sysfs name
WWID of the device
User defined Alias name
size=10G
features=0
hwhandler=0
--------------------------------|
|
|----> Hardware Handler, if any
|
|-----------------------------> Features supported
|-----------------------------------------------> Size of the DM device
Path Group:
|-+- round-robin 0 prio=1
active
-------------- ----------|
|
|
|--------------------->
|
|
|------------------------------>
|
|---------------------------------------------->
|--------------------------------------------------->
Path
Path
Path
Path
group state
group priority
selector and repeat count
group level
First path on Path Group 1:
`- 4:0:0:2 sdd 8:32
active ready
-------- --- --------- ----|
|
|
|
|-----------------> Physical Path state
|
|
|
|------------------------> DM Path state
|
|
|--------------------------------> Major, minor numbers
|
|-------------------------------------> Linux device name
|--------------------------> SCSI information: host, channel, scsi_id and LUN
Storage Management in Linux
[ Presented by Liju Gopinath ]
65
Finding scsi id of a disk
[root@vmrh5 ~]# /lib/udev/scsi_id --whitelisted --device=/dev/sdc
1IET
00010002
[root@vmrh5 ~]# /lib/udev/scsi_id --whitelisted --device=/dev/sdg
1IET
00010002
Storage Management in Linux
[ Presented by Liju Gopinath ]
66
Finding HBA details
# systool -c scsi_host -v | egrep -i fw_version | driver_version
| model_name | model_desc
# lspci | grep -i fibre
# ll /sys/class/scsi_host
# ll /sys/class/fc_host/
# cat /proc/scsi/qla**
# systool -m qla2xxx v
# systool -c fc_host v
# cat /proc/scsi/scsi
Storage Management in Linux
[ Presented by Liju Gopinath ]
67
# modinfo qla2xxx
# modinfo lpfc
# lspci | grep -i emulex
# cat /sys/class/scsi_host/host0/fwrev
# cat /sys/class/scsi_host/host0/node_name
# cat /sys/class/scsi_host/host0/port_name
# cat /sys/class/scsi_host/host0/lpfc_drvr_version
# cat /sys/class/scsi_host/host0/serialnum
# cat /sys/class/scsi_host/host0/speed
# cat /sys/class/scsi_host/host0/state
Storage Management in Linux
[ Presented by Liju Gopinath ]
68
Reference
http://sources.redhat.com/lvm2/wiki/MultipathUsageGuide
http://www.tldp.org/HOWTO/LVM-HOWTO/index.html
http://www.ibm.com/developerworks/library/l-lvm2/l-lvm2-pdf.pdf
http://access.redhat.com/knowledge/docs/Red_Hat_Enterprise_Linux/
http://www.redhat.com/magazine/009jul05/features/lvm2/
http://www.tldp.org/HOWTO/LVM-HOWTO/
http://www.sourceware.org/lvm2/
Storage Management in Linux
[ Presented by Liju Gopinath ]
69
