Restore A Missing Physical Volume

Let's find out the step-by-step involved to recover a failed physical volume in Linux.

The setup has got the"testvg" which got "testlv" and corresponding Physical Volume (PV) is on '/dev/sdb' block device as shown below:

[root@RHEL4 archive]# vgs

  VG     #PV #LV #SN Attr   VSize VFree

  VolGroup00   1 2 0 wz--n-  19.88G 32.00M

  testvg   1   1 0 wz--n- 200.00M 0              ← targeted VG

[root@RHEL4 archive]# pvs

  PV     VG     Fmt  Attr PSize   PFree

  /dev/sda2  VolGroup00 lvm2 a- 19.88G 32.00M

  /dev/sdb   testvg lvm2 a-   200.00M 0    ← targeted PV

[root@RHEL4 archive]# lvs

  LV   VG     Attr   LSize Origin Snap%  Move Log Copy%

  LogVol00 VolGroup00 -wi-ao  18.88G

  LogVol01 VolGroup00 -wi-ao 992.00M

  testlv   testvg -wi-a- 200.00M                           ← targeted LV

..... Destroyed the /dev/sdb master table using the below command

[root@RHEL4 ~]# dd if=/dev/zero of=/dev/sdb bs=512 count=2

2+0 records in

2+0 records out

..... Later, when I tried to check for the current PV, LV and VG, it was not found.

[root@RHEL4 ~]# vgmknodes

[root@RHEL4 ~]# vgmknodes -v

Finding all logical volumes

[root@RHEL4 ~]# lvs

  LV   VG     Attr   LSize Origin Snap%  Move Log Copy%

  LogVol00 VolGroup00 -wi-ao  18.88G

  LogVol01 VolGroup00 -wi-ao 992.00M

[root@RHEL4 ~]# vgs

  VG     #PV #LV #SN Attr   VSize VFree

  VolGroup00   1 2 0 wz--n- 19.88G 32.00M

[root@RHEL4 ~]# lvs

  LV   VG     Attr   LSize Origin Snap%  Move Log Copy%

  LogVol00 VolGroup00 -wi-ao  18.88G

  LogVol01 VolGroup00 -wi-ao 992.00M

[root@RHEL4 ~]# pvs

  PV     VG     Fmt  Attr PSize  PFree

  /dev/sda2  VolGroup00 lvm2 a-   19.88G 32.00M

..... Unable to activate this failed VG

[root@RHEL4 ~]# vgchange -ay testvg

  Volume group "testvg" not found

..... Even running e2fsck failed on such device

[root@RHEL4 ~]# !e2fsck

e2fsck -f -y /dev/testvg/testlv

e2fsck 1.35 (28-Feb-2004)

e2fsck: No such file or directory while trying to open /dev/testvg/testlv

The superblock could not be read or does not describe a correct ext2

filesystem.  If the device is valid and it really contains an ext2

filesystem (and not swap or ufs or something else), then the superblock

is corrupt, and you might try running e2fsck with an alternate superblock:

e2fsck -b 8193 <device>

...... Found that the entry to the corresponding VG is still there in the /etc/lvm/archive folder for restoring

[root@RHEL4 ~]# cd /etc/lvm/archive/

[root@RHEL4 archive]# ls

testvg_00000.vg  testvg_00002.vg VolGroup00_00000.vg

testvg_00001.vg  testvg_00003.vg

[root@RHEL4 archive]# cat testvg_00003.vg

# Generated by LVM2: Tue Oct  1 15:06:48 2013

contents = "Text Format Volume Group"

version = 1

description = "Created *before* executing 'vgs'"

creation_host = "RHEL4.4"   # Linux RHEL4.4 2.6.9-42.EL #1 Wed Jul 12 23:15:20 EDT 2006 x86_64

creation_time = 1380654408  # Tue Oct  1 15:06:48 2013

testvg {

     id = "x1TBig-s0D6-7Njv-U2oR-bolr-VZaP-o4Mkw5"

     seqno = 3

     status = ["RESIZEABLE", "READ", "WRITE"]

     extent_size = 8192          # 4 Megabytes

     max_lv = 0

     max_pv = 0

     physical_volumes {

             pv0 {

                     id = "4X3wIH-uuCy-F99r-H2mt-zklh-5ZCR-64lTqQ"

                        device = "/dev/sdb" # Hint only

                     status = ["ALLOCATABLE"]

                     pe_start = 384

                     pe_count = 50   # 200 Megabytes

             }

     }

     logical_volumes {

             testlv {

                     id = "GsaJe0-QCoO-DyeP-mqB3-UaPM-12XQ-4CeVvE"

                     status = ["READ", "WRITE", "VISIBLE"]

                     segment_count = 1

                     segment1 {

                             start_extent = 0

                             extent_count = 50   # 200 Megabytes

                             type = "striped"

                             stripe_count = 1    # linear

                             stripes = [

                                     "pv0", 0

                             ]

                     }

             }

     }

}

..... Trying to activate in-complete VGs

[root@RHEL4 archive]# vgchange -ay testvg --partial

  Partial mode. Incomplete volume groups will be activated read-only.

  Volume group "testvg" not found

..... At this stage, we would need to re-create the missing PV using device reference and UUID details from archive list as shown below.

[root@RHEL4 archive]# pvcreate --uuid "KWfeHu-cP6u-Qpdz-XzDD-iomY-uSKd-2d6V6G" --restorefile /etc/lvm/archive/testvg_00003.vg /dev/sdb

  Couldn't find device with uuid '4X3wIH-uuCy-F99r-H2mt-zklh-5ZCR-64lTqQ'.

  Physical volume "/dev/sdb" successfully created

..... Now, ran vgcfgrestore on the corresponding VG

[root@RHEL4 archive]# vgcfgrestore testvg

  Restored volume group testvg

.... Next, activate the volume group

[root@RHEL4 archive]# vgchange -ay testvg

  1 logical volume(s) in volume group "testvg" now active

..... Check if respective PV, VG and LV are present.

[root@RHEL4 archive]# vgs

  VG     #PV #LV #SN Attr   VSize VFree

  VolGroup00   1 2 0 wz--n-  19.88G 32.00M

  testvg   1   1 0 wz--n- 200.00M 0                  ← here is the restored VG

[root@RHEL4 archive]# pvs

  PV     VG     Fmt  Attr PSize   PFree

  /dev/sda2  VolGroup00 lvm2 a- 19.88G 32.00M

  /dev/sdb   testvg lvm2 a-   200.00M 0      ← restored PV

[root@RHEL4 archive]# lvs

  LV   VG     Attr   LSize Origin Snap%  Move Log Copy%

  LogVol00 VolGroup00 -wi-ao  18.88G

  LogVol01 VolGroup00 -wi-ao 992.00M

  testlv   testvg -wi-a- 200.00M                          ← this is the restored LV

..... Now, the LV can be mounted for further use .

[root@RHEL4 archive]# mount -a

[root@RHEL4 archive]# df

Filesystem       1K-blocks  Used Available Use% Mounted on

/dev/mapper/VolGroup00-LogVol00

                   19481284   2183572 16308120  12% /

/dev/sda1           101086 10275 85592  11% /boot

none                254384     0 254384   0% /dev/shm

/dev/mapper/testvg-testlv

                     198337 10048 178049   6% /test

[root@RHEL4 archive]#

PV has been restored successfully!!!