LVM recovery tale.

Over the weekend I had the worrying experience of losing my LVM settings and potentially all my data… a quick search on the web showed a confusing set of information, much of it for older versions of LVM and therefore rather suspect.

Well, I recovered all my data and it was really quite simple, so I’ve written up what I did in the hope that someone else, in a similar situation, will find it useful. It’s a scary thing, losing the whole hard disk and knowing that, in reality, its all there.

First the situation

I’ve got a small /boot partition as ext2, and a larger one for the root directory (2Gb, also ext2). The rest of the hard disks (nearly 120Gb) are assigned to a volume group called, descriptively, system… (which is SuSE’s idea of a default name).
More accurately, they were supposed to be assigned to it. At first I had just added 60Gb to the volume group, it was my first use of LVM and I was hedging my bets. After 6 months of trouble free operation I decided to add another 60Gb of disk, which I did 3 months ago. Except that, although the physical and volume group managers all agreed that the volume group had 120Gb, the logical volume manager insisted that there was only 60Gb. I’d used Yast2 to create and add the volumes.
I tried every combination of commands I could think of to get the logical volume manager to recognise the additional space but it wouldn’t.
At the time, I was busy, so forgot about it, then last week I realised that I wanted to use the space so settled down to do something about it.

The problem

So, it seemed the best solution would be to remove the second partition that I had added (/dev/hdd1) from the physical volume manager and then add it back.
It wasn’t recognised so wouldn’t be missed, right?
Wrong!
pvremove /dev/hdd1 removed the label from /dev/hdd1 but also from /dev/hda7 (which was the original partition and full of data).
pvscan and pvs reported no physical volumes on the disk.
vgscan and vgs couldn’t find any volume groups.
lvscan and lvs were non-starters obviously.

The rather surreal thing was, the whole system kept on running quite nicely, X Server and KDE desktop and all, but I knew that as soon as I rebooted the system would be toast.

First I tried adding the partition back to the volume group system, but the system couldn’t find the ’system’ group. I tried creating the physical volume again (pvcreate) but that told me that the volume already existed. It became clear that I would need to reboot and hope that the system sorted itself out, flushed the disks, resynced, whatever.

The solution

After rebooting the system wouldn’t come up, which is kind of what I had expected so I had to reboot from the SuSE Rescue disks. So now I had to think about how to recreate the physical volumes, volume group and logical volume and do it with the data intact. (I have daily backups but the thought of restoring the whole system, applications and data, was not too exciting, especially as I knew all the data was there and intact. With a ‘regular’ hard disk partition that had got lost I could scan the disk for potential disk partitions and restore them. But that wouldn’t work with LVM.

On a search through various sites, I found one that mentioned the importance of saving a copy of the volume group parameters to a file using vgcfgbackup. This file could then be used to restore the parameters later, assuming that the underlying physical structure hadn’t changed. Well, the physical layout hadn’t changed but unfortunately I hadn’t created a backup of the volume group parameters (the ‘descriptor area’ to use the technical term) so that didn’t seem to hopeful. I poked around in the /etc directory (I still had the ‘/’ partition remember, as that was on its own ext2 partition) and noticed that there was a /etc/lvm/backup/ directory and a /etc/lvm/archive/ directory. Further investigation and I found that these are automatically created by LVM whenever changes are made to the system.

Unfortunately, all the messing around I had done had created a non-working version of the system file and the archive files didn’t seem to be recent enough. But, I remembered that I had a backup of the system files (going back 6 months in fact) and so I dug out a copy of the /etc/lvm/backup/sysem file and used that.

Here is what I did: First find out the old UID’s of the partitions, this is in the /etc/lvm/backup/system file. They are quite long… make sure you get the UID for the physical volumes.
$pvcreate -u sdSD-2343-SD939-adIda2 /dev/hda6
$pvcreate -u dk33kd-929293nd-adfja298a /dev/hdd1
$vgcreate -v system /dev/hda7 /dev/hdd1
$vgcfgrestore -f /etc/lvm/backup/system system

and lo!, all data present and correct!

In fact, I just rebooted the system and was back where I had started with the additional benefit of an extra 60Gb of disk space, because now I had the extra partition properly included.

[Note: in the lines using pvcreate... above I could have used:
$ pvcreate --restorefile /etc/lvm/backup/system
to automatically find the ID's but I hadn't realized that at the time. Without the UID's then the vgcfgrestore will not find the physical volumes that it needs to recreate the volume group.]

The lesson

Don’t panic!
Keep a safe copy of your /etc/lvm/ files!
Make sure that you have a Rescue disk that understands the LVM system!

Apart from the above disaster, which seems to have sorted itself out very easily, I have had no trouble with the LVM system. At first I was worried that if there was a failure it would lose everything. There is something very comforting about a simple ext2 (or FAT) partition in that I know it can just be hacked at the bit level and rebuilt. Something like LVM, which is logical volumes on top of volume groups on top of physical volumes is impossible to rebuild ‘by hand’ so I’m learning to trust technology a bit.

70 Responses to “LVM recovery tale.”

  1. sol Says:

    THANKS FOR POSTING YOUR LVM recovery tale !!! Following your example saved my data!!!!!!!!!!!!!!! (and my weekend :) I lost my vg and lv after resizing an extended partition containing a physical volume included in the vg. After several repartitions and reiserfsck’s, I was able to recover the lv and suffered very minor reiserfs inconsistancies.

  2. jalal Says:

    I’m really glad it helped you out. :-) Whenever some disaster strikes my computer, I do a search on google for someone who may have solved the same, or similar, problem and I’m always so happy if they’ve taken the trouble to write about it… so it’s good to return the favour.

  3. FluffyArmada Says:

    :) You saved me a lot of trouble too. I made a major ‘booboo’ and used the mkswap command on the wrong partition, but after reading this, I had a better understanding of how to fix it. :-) Thanks a Lot.

  4. Dagnel Says:

    Thanks Thanks and thanks after searching all day it saved my day. This is a real story, this is my story.

  5. seahawk Says:

    Well, after searching for nearly 2 weeks I found the solution. 900 gigs saved. Great story!!!! Thank you, thank you, thank you..

  6. Tim Says:

    ok… what if the logical volume was mounted as /, obviously /etc/lvm/backup (which i know exists) is on the logical volume that i can’t access… any ideas?

  7. jalal Says:

    Hi Tim

    If you have the root directory in a LVM and you can’t access the LVM, then you’ll have to find a backup of the system files… and if you don’t have a backup then it difficult. You may want to take a look at this page: http://tldp.org/HOWTO/LVM-HOWTO/dangerousops.html and the two pages following it as they give a technique for rescuing the volume IDs, but I’ve never tried this, so you’re on your own. Let me know if it helps though, others will find the information useful.

  8. Tenpin Says:

    I had a similar problem but the vg directory under /dev disappeared and would not come back, once I had executed the pvcreate, vgcreate and vgcfgrestore commands I used the ‘vgchange -a y’ and the os recreated those directories under /dev and varied on the volume group.

  9. data recovery Says:

    thank you. the article help me recover data!!

  10. Thomas Says:

    Oh…you are lucky that the backup under a normal file system and revocer your data less than 15mins. I did something like you to extend/delete LV…but somehow missing a command resize2fs <==at last I knew. My case is the root “/” also under the LV…I googleing the web more than 1 week still to find a soultion. Good luck for me…contiune to google.

  11. KraL Says:

    Hi there,

    I was wondering about using LVM, but before doing that, I did a little search on how to recover data incase of physical crash, or hardware changes, etc …

    And that how-to respond totally to my questions : I can now use LVMs without any fear :D

    I have tested several case where, pvs, vgs or lvs where removed. under different FS (Ext & XFS in fact). Rebinding the pv’s UID & vgcfgrestore works like a charm. As mentionned by Tenpin, in my case ‘vgchange -ay’ is needed to mount the lvm back after a vgcfgrestore.

    so, the whole procedure for me is: pvcreate -u ….. vgcfgrestore … vgchange -ay Then mount it :)

    The only thing to care about is to regulary backup /etc/lvm !!

    Thx Jalal for that HowTo, I was about to leave LVM when implemented without RAID (not always possible…)

  12. Daniel Says:

    Hi,

    I admit I’m confused with all this terminology and programs, by force I just started learning about the existence of LVM, so perhaps you can give me a set of recipes for my case. I installed Fedora Core 5 on a VMware machine. Compared with past ocasions I noticed that one of the partitions that the installer was offering me had a device with the name /dev/VolGroup00/LogVol00 instead of the regular /dev/sdax assigned to the root partition. The /boot and swap were /dev/sda1 and /dev/sda2. The installation went OK and the system was running quite well. Unfortunately one day I stopped the VMserver after forgetting to shutdown the guest system. Now I can’t boot, it just hangs with the message “kernel panic”. I was able to boot the system with both the rescue disk and a knoppix CD and see /boot with its content and the existence of /dev/sda2, but the root is invisible or gone. Is there any way to see/recover the root directory? Many thanks in advance.

    Daniel

  13. jalal Says:

    Hi Daniel

    I’m assuming that you went ahead and installed the system on /dev/VolGroup00. And tha tthe kernel panic is because the kernel (on boot) can’t find any of the files it needs to get started.

    If you have access to a backup of the /etc/.lvm directory, then use that as I detailed above, but I guess you don’t. So first make sure that the rescue disk that you’re using has the LVM modules loaded. Check that /dev/sda2 has an id of ‘8e’ (LVM type). Next you should be able to scan the /dev/sda2 partition for old PV blocks (pvscan i think) with -v or -vv to get more detail.

    Beyonf that… I’m not sure. I have a feeling that it must be possible to recover enough of the volume to be able to find the the missing vg ids but I haven’t had to deal with it here.

    Good luck anyway, let us know if you succeed.

  14. daniel Says:

    Hi Jalal,

    The gentoo live-cd appears to have all the lvm tools. I ran lvdisplay and I got what is below. You’re right, I didn’t save anything from the /etc/lvm, so can I use any of the info taken from lvdisplay or vgdisplay or others to recover my root? If so, what should I do? Thanks.

    Daniel

    — Logical volume — LV Name /dev/VolGroup00/LogVol00 VG Name VolGroup00 LV UUID BUx6ZG-QsMx-26hz-5iCv-0YF2-qm4B-ee0Mkw LV Write Access read/write LV Status NOT available LV Size 95.00 GB Current LE 3040 Segments 1 Allocation inherit Read ahead sectors 0

    — Logical volume — LV Name /dev/VolGroup00/LogVol01 VG Name VolGroup00 LV UUID R8uiEo-pCFO-MZd9-VTrr-To12-ZbGv-x5Pmbn LV Write Access read/write LV Status NOT available LV Size 1.94 GB Current LE 62 Segments 1 Allocation inherit Read ahead sectors 0

  15. JB Says:

    What a great find this was. Your help just got me out of a reformat and starting over. Thanks heaps :)

  16. Renate Says:

    On our page you will find how to restore/recover information from your hard disk, cd, dvd, tape drive or flash memory, both for PC and MAC. If the damage done to your Hard disk is physical you will in most cases not be able to restore the data yourself. For this there are specialized companies whom can do the job for you. Come and take a look, free information!

    http://www.informationrestoration.com

  17. sweetfd Says:

    interesting and helpful!

  18. Robin Andersen Says:

    It once did happen to me that one morning suddenly my System refused to start later i discovered that the Disk was crashed. It was still in the warranty period so i called out the system vendor who happened to tell me that he is ready to replace the hard disk but also told me that he was not responsible for any data that was present in the Hard Disk. But i was not ready for the replacement without the data inside the Hard disk because it contained all the files related to my business and many other stuff like the images, logos that were associated with my business, my accounts and all other stuffs. The vendor then referred me to Disk Doctors Labs Disk Recovery where in i recovered all data, and all images and each and evey file in a new disk and it was very much in time which also saved me from further losses.

  19. Julius Smith Says:

    Your LVM recovery tale is a great service – thanks!

    A somewhat related question I’ve been trying to answer is can one simply mount a “foreign” LVM partition in /mnt somehow? In my case, I did a fresh system install, and now I want to copy in selected files from the previous version of the system (which is on a separate LVM disk not recognized by the new system, although fdisk sees it).

    Thanks for any leads, Julius

  20. jalal Says:

    Julius

    As far as I know, no, you can’t. You will need to restore/remount the partitions as they were and copy the files off of them. What you will need is the original LVM layout to find the files again. You could try creating a new volume group and add the old physical volume to it, then try mounting the volume group as a logical volume. I haven’t tried this, but it sounds like it could possibly work. You will need to have the old cfgbackup file (see above).

    Good luck. Let us know if it works out.

  21. Winfried Dobbe Says:

    Thanks so much! This saved me a lot of data and work!! In my case the lvm metadata of only physical volume was gone due to a mistake I made in the bios of the sata controller card. So I did pvcreate -u [id] /dev/sdb Then the ‘vgcreate -v vg00 /dev/sdb /dev/sdc’ returned something like “volume group vg00 already exists”, I ignored that. Then the vgcfgrestore command, no problems there. However ‘vgchange -a y vg00′ returned “/proc/misc: No entry for device-mapper found”. After some googling the following command helped: ‘modprobe dm-mod” Now ‘vgchange -a y vg00′ worked ok and I was able to mount the logical volumes.

  22. John Says:

    THANK YOU! After reading your solution I managed to recover my harddisk.

    I installed a new linux on a new disk and was planning to mount the old one. I never heard of LVM and such and before long I had tried to run vgcreate, pvcreate and lvcreate, rendering my old partition unmountable :( Additionaly I had no backup available of the old system, and was therefore stuck.

    The solution was to copy the old (unmountable) disk with “dd if=/dev/hdb2 of=/backup/olddisk.bak.dd”. I then opened the binary file in a editor, and to my surprise and joy, the partition information was readable in the beginning. I managed to copy/paste a complete backup-file (the one made when I run my first vgcreate) and use it with vgcfgrestore.

    Hope someone else can benefit from this,

    /John

  23. Sexy Sexy Penguins»Blog Archive » LVM - From Failure to Recovery Says:

    [...] http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/ – LVM Recovery Tale [...]

  24. Michael Soland dot com » Catastrophic Hard Drive Failure Says:

    [...] Wrong!  I found to my dismay that when LVM drives are removed, the “logical volume� breaks because it cannot find all of the “physical volumes�: a.k.a. the bad disk on /dev/hdb.  Later that week when I had time to sit down with the bugger, I read an article in Linux Journal related to storage and how to perform recoveries on systems with LVM.  At that moment I realized my design flaw.  If using LVM, always use it on top of a RAID and back-up those good LVM files. [...]

  25. Red Says:

    You just saved my ass, thanks.

    vgcfgrestore -vf /etc/lvm/archive/filename rootvg

    and ‘lo, my volume was retored.

    Many thanks.

  26. data recovery software Says:

    Interesting article on LVM, I referred it to one of my friend and he benefits from it.

  27. Oliver Says:

    Hi,

    I have a really really big problem and hope that you might can help me get my data back. I have a Fedora Core 5 operating system with 2 HDD. One is only for backup (unfortunately I havent made any) and the other one with all my date. HDA1 is the boot partition and HDA2 is set up with LVM. One VolumeGoup and a few Logical Volumes. As my system could not boot bechause of a kernel error I decided to run a Koppix v.5 to get to my files. As I then noticed that I cant mount HDA2 I made a big mistake. I removed the Volume Information (lvremove, vgremove, etc..) and didnt backup the LVM data. What can I do do get back my data from HDA2??? Its really a lot and things that I need! Is there a possibility. lfcfgrecover doesnt do anything because I dont have a backup of LVM Structure. Neiter pvscan, lvscan finds anything.

    Please HELP ME. Thanks a lot!

  28. jalal Says:

    Hi Oliver.

    You should have a backup of the data in /etc/lvm/backup or maybe in /etc/lvm/archive. When you remove the volume group it should only modify one of them and leave a backup… HTH

  29. oliver Says:

    Hi Jalal! Thanks for your fast reply. The problem is, that my / is also in the VolGroup I cannot access. I just made a less -f /dev/hda2 an found some information about my volgroup. So I made a pvcreate -u ID(from less -f /dev/hda2) then a vgcreate VolGroup00 /dev/hda2 but unfortunately I cannot create the lv with an userid.

  30. oliver Says:

    Can I now mount the VolGroup00 and make a vgcfgrestore? How can I mount my VolGoup00 on Fedora running Knoppix. I cannot find anything in /dev/

    thank you!

  31. Rob Says:

    Tried this with reiserfs on LVM…on two different partitions….did not work as advertised….somehow my LVM partition moved…..thus cannot locate journal header or superblock. Not sure what went wrong…..

  32. Rob Says:

    Message received when trying to recover…. No reiserfs metadata found. If you are sure that you had the reiserfs on this partition, then the start of the partition might be changed or all data were wiped out. The start of the partition may get changed by a partitioner if you have used one. Then you probably rebuilt the superblock as there was no one. Zero the block at 64K offset from the start of the partition (a new super block you have just built) and try to move the start of the partition a few cylinders aside and check if debugreiserfs /dev/xxx detects a reiserfs super block. If it does this is likely to be the right super block version.

  33. Sem Says:

    I have a bit different situation with LVM and Suse 10.0. Linux is installed on one old HP server with 2 SCSI drive of 8GB each. I created group system and lvm “local” that I used for backup ISO images. Now I want to remove LVM because I want to make a backup without LVM and then clone/restore entire system to new pc (again with no LVM).

    I have no data on lvm that I should restore on new pc. LVM was configured using Yast2.

    What would you suggest as a best scenario to get rid of LVM without trouble with the rest of linux system?

    Thanks. Sem.

  34. jalal Says:

    Hi Sem

    If you have no data on the lvm, then it should be simple (especially using Yast)

    In Yast LVM admin page, remove the logical volumes from the physical volumes, then remove the physical volumes from the volume group and finally remove the volume group. You will end up with a partition that is flagged as being LVM, you can then go to the partitioner and format that partition as you see fit.

    This will destroy the data in the LVM, so make sure you have nothing in there that you want to keep!

  35. Sem Says:

    Hi Jalal,

    Step one was ok. But when I wanted to remove ph. volume from the group I got an error “You cannot remove the last PV from volume group. Delete Volume Group instead.” So I did delete volume group.

    I found that PV is still type “linux LVM”. I had to change that.

    At the end I rebooted system and it look like everything is fine so far. Thanks for the help. Sem.

  36. chocolate cowgirl Says:

    “Keep a safe copy of your /etc/lvm/ files!” hmmmm good advice unfortunately I am reading this retrospect. My /dev/VolGroup just up and disappeared.

  37. jumroo Says:

    Can the /etc/lvm/backup/{VGNAME} file be used to understand the configuration of the {VGNAME}? I am trying to write a tool that creates a tree-like structure for LVM2?

  38. jalal Says:

    Jumroo: I guess it could. It seems to be a very clearly laid out file. And LVM is an open standard, so you can find the full specifications on the web.

  39. jumroo Says:

    Thanks Jalal. Trying to find the spec. Not on google. probably find it in redhat or someplace. thks, jumroo

  40. Allan Says:

    Thank you greatly for the information. This lead me in the right path to remedy a very big problem for us. Much appreciated that you have provided this to the public.

    Kind Regards, Allan

  41. Guillermo Says:

    THANKS A LOT!!!!!! After a sudden LVM misconfiguration, your post allowed us to recover 4TB of data belonging to the 100 users of the cluster I admin.

    Muchas gracias!!! ;-)

  42. fee Says:

    good

  43. horse Says:

    fair play to ya chief… this really dug me out of a hole… I was attempting to grow a JFS LVM2 disk, it failed, I was able to revert back to my previous lvm config with your tip. I also had to do ‘fsck.jfs -f’ and then ended up having to revert to doing ‘fsck.jfs –omitjournalreplay’ and was able to recover all my data (albeit everything was under lost+found/ but other than having to rename a bunch of dirs I was back up and running in no time :) yay! I was certain that my 1.1TB of data was lost & gone for ever… now, after that scare i’m off to find a proper backup solution. Thank you!

  44. Mojo LA Says:

    My problem with lost logical volumes is slightly different …

    I had a server power supply die last week on a three-year-old Shuttle XPC box running SuSE 9.2.

    Shuttle power supplies are proprietary, so you have to get one from Shuttle. But in the mean time, I wanted to get back on the air.

    Fortunately I had a newer desktop machine, and my thought was to take the boot disk from my server and drop it into my newer Shuttle and get back on the air.

    The disk booted Linux okay, but pvscan could not find ANY LVM physical volumes on the disk, and hence no logical volumes either. So I was dead.

    At that point I decided to take the power supply out of my desktop machine and use it to replace the dead one. Naturally they’re different physical sizes now, so I can’t screw it in.

    The old machine came up and boots just fine, with all the physical and logical volumes in place.

    They’re both Intel processors, and it’s an SATA disk drive.

    Why could the older machine see the volumes but the newer machine couldn’t? Is there a solution to this?

    Cheers, Mojo Morris Jones Monrovia, CA

  45. bills blog » LVM Recovery Says:

    [...] I found a great recovery guide at http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/ that pretty much covers everything I was un-sure about. [...]

  46. LVM Recovery Says:

    Kernel Linux – Ext2 & Ext3 Data Recovery Software supports LVM Partition PARTIALLY

  47. I think I’ll name it Lazurus next time… at The ScatterBrain Chronicles Says:

    [...] When this was complete, I had a booting machine.  But I still was concerned about the loss of data.  In all actuality I was very concerned.  I came back to the office, started researching and game across this and this.  This gave me home that I would be able to rescue the data. [...]

  48. Ryan Says:

    You just saved well over a TB of my data on my debian file server.. I was lost but you shed the most light on what needed to be done.. thanks much

  49. pk stuff » LVM Recovery Tips Says:

    [...] http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/ [...]

  50. Jeff Mercer Says:

    Thanks for the nice little how-to. Like John, I had pretty much my entire disk except for /boot in one volume group. When someone tried doing cryptsetup and specified the wrong partition, they hosed the disk. Fortunately, I had a large external volume and a rescue CD. So I was able to dd the entire hosed device to a file. Than I used a lot of strings, head, tail, cat, grep and cursing to find the automatically created backup lvm configs in that file. Once extracted I had to do some minor corrective editing to put in the close braces, but i was able to get everything back up.

    So, in other words, you rock! Thanks! :)

  51. Joe Nih Says:

    Well, finally I came to this page. It was a disaster for me today, I accidentally issued a GRUB command- setup (hd0)-onto my (whole 5.2TB RAID6) VG, I can not access all the data onit any more, Redhat ES4 LVM2 list it as “raw” disk, no VG/File System……, which I believe overwrited the beginning of the disk, since less -f /dev/sda can see those grub text in the first few pages. I have the VG backup file and will try to recover the data tomorrow. God bless my data!

  52. Joe_Nih Says:

    I can not express my thankfullness to you, man! you are the life saver. I recoverd everything on this 5.2TB. here is what I did: PV is locate on /dev/sdb (whole raid was be used)

    pvcreate -f –uuid “Z4lh8H-G0e8-K8q1-4WB6-faac-39hk-b82MWU” –restorefile /etc/lvm/backup/recover /dev/sdb Physical volume “/dev/sdb” successfully created pvdisplay — Physical volume — PV Name /dev/sda2 VG Name VolGroup00 PV Size 232.72 GB / not usable 0 Allocatable yes PE Size (KByte) 32768 Total PE 7447 Free PE 1 Allocated PE 7446 PV UUID RF27gp-ESK3-LhLv-Y91B-8NOl-6tmG-42Op0X

    — NEW Physical volume — PV Name /dev/sdb VG Name PV Size 4.73 TB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID Z4lh8H-G0e8-K8q1-4WB6-faac-39hk-b82MWU

    vgcreate sbsraid1 /dev/sdb Volume group “sbsraid1″ successfully created vgcfgrestore -f /etc/lvm/backup/recover sbsraid1 Restored volume group sbsraid1

    vgdisplay — Volume group — VG Name sbsraid1 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 4.73 TB PE Size 4.00 MB Total PE 1240049 Alloc PE / Size 1240049 / 4.73 TB Free PE / Size 0 / 0 VG UUID Cl4u2r-MqQe-ws9w-uOT1-K8GO-RO5h-4iimfj

    — Volume group — VG Name VolGroup00 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 4 VG Access read/write VG Status resizable MAX LV 0 Cur LV 3 Open LV 3 Max PV 0 Cur PV 1 Act PV 1 VG Size 232.72 GB PE Size 32.00 MB Total PE 7447 Alloc PE / Size 7446 / 232.69 GB Free PE / Size 1 / 32.00 MB VG UUID LtkJbi-2Lxr-QIBz-ltd5-FH3D-WEuc-FhlymW

    vgchange -ay 1 logical volume(s) in volume group “sbsraid1″ now active 3 logical volume(s) in volume group “VolGroup00″ now active

    change /etc/fstab mount -a everything is recovered

  53. jalal Says:

    Joe, congratulations on saving your data.

    And many thanks from me and those who follow on, for sharing the details of what worked for you.

    cheers

  54. q Says:

    I don’t get it. sudo pvcreate -u /dev/hdd2 Software RAID md superblock detected on /dev/hdd2. Wipe it? [y/n] n Physical volume “/dev/hdd2″ successfully created

    scary enough, then:

    sudo vgcreate -v test /dev/hdd2 Wiping cache of LVM-capable devices Adding physical volume ‘/dev/hdd2′ to volume group ‘test’ /dev/hdd2 not identified as an existing physical volume Unable to add physical volume ‘/dev/hdd2′ to volume group ‘test’.

    Maybe I don’t pass the interpretation test from your site to my shell prompt.

    It’s a foreign drive to this system.

  55. q Says:

    further investigation: sudo pvcreate -u 6oVVgd-WIVK-qB0y-PXk2-Acu5-ZxcI-gMj5JD /dev/hdd2 Software RAID md superblock detected on /dev/hdd2. Wipe it? [y/n] n Physical volume “/dev/hdd2″ successfully created

    pvdisplay — NEW Physical volume — PV Name /dev/hdd2 VG Name PV Size 74.43 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID 6oVVgd-WIVK-qB0y-PXk2-Acu5-ZxcI-gMj5JD

    Then pvdisplay again; no output, no pv found. Weird eh? You can pvcreate then pvdisplay once, then it’s gone!

  56. Daobang Says:

    Hi, all

    I have some questions to ask you, please help me, please send the mail to daobangw@promisechina.com.

    1. How to resume the data from snapshot?

    2. Does LVM support rollback?

    3. If I create a snapshot for LV on PC1, and I copy the snapshot volume data to PC2, how to resume the LV data on PC1 from the snapshot volume data on PC2?

    Thank you very much.

  57. jalal Says:

    Hi q

    Maybe it’s because it’s a RAID drive? Sounds like it’s writing to one drive and trying to read another, or syncing to the ‘un-lvm’ed’ drive.

    Not something I’ve come across though.

  58. jalal Says:

    Hi Daobang

    First, you should read through this: http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html

    To recover the data in a snapshot, I imagine you just mount it and copy it somewhere. If I understand, the snapshot is a complete file system and can be mounted and read like one. But I haven’t tried it, so I may be wrong.

    HTH

  59. Miguel Says:

    Hello,

    This procedure really saved me. I was working on an IBM z990-2084 under zVM 5.1 when I needed to add more disk space. You should seen my face when I rebooted. Linux asked me to enter the root password. I knew then I was in deep problems. I spent a week trying to recover my data when I found your procedure. It’s a lifesaver. Thanks a lot.

    Miguel

  60. Comment on LVM recovery tale. by q Says:

    [...] Comment on LVM recovery tale. by q [...]

  61. Comment on LVM recovery tale. by Jeff Mercer Says:

    [...] This is the cached version of http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/#comment-10062 We are neither affiliated with the authors of this page nor responsible for its content. Comment on LVM recovery tale. by Jeff Mercer [...]

  62. Comment on LVM recovery tale. by Miguel Says:

    [...] This is the cached version of http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/#comment-10702 We are neither affiliated with the authors of this page nor responsible for its content. Comment on LVM recovery tale. by Miguel [...]

  63. back_to_linux Says:

    Thanks, thanks and thanks and Thanks, thanks and thanks and ….. I needed change my hard disk, and some thing gone wrong. No, no , never my fault ;-) ))) Just read and do what you said, more 1 year after, it’s very good thing you exist. Just for people, keep in you mind to never write on your partition to recover it properly all my thanks to your mother to invent you !

  64. Companies Published In The Web Says:

    How to Create Effective Web Publishing Content…

    I frequently tell my coaching clients about the importance of regularly updating web site content. Fresh content keeps customers coming back and gets indexed more frequently by search engines. Novelty spurs attention, whether human or technological. A …

  65. Roy Long Says:

    Jalal, dude you rock! Your instructions saved my team from certain destruction! We were migrating data from a Xiotech SAN to a NetApp filer and something got screwed up along the way. Basically three physical volumes on the Xiotech as a single logical volume migrated to one physical volume on the NetApp – but somehow it kept the original partition information on the new logical volume (this was on a new iSCSi LUN). Anyway, our admin had decided to remove this volume from a volume group as there were some issues booting and it lost the volume information for the new logical volume.

    Well, I read up on LVM for about an hour and a half and it dawned on me that the data should still be on the original physical volumes on the Xiotech SAN – and all I would need to do is roll back to the config before the migration. Thanks to your instructions this process became a lot easier and I had it back and running within an hour.

    We have a research operation (drug discovery) which depends on this server too, so you have my thanks for making the process of finding the correct sequence of commands that much easier.

    Muchos Gracias Amigo!

  66. The Penguinlinux.com Blog » Blog Archive » How to recover from an LVM disaster Says:

    [...] at http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/LVM recovery tale. Over the weekend I had the worrying experience of losing my LVM settings and [...]

  67. the eXternal mind » links for 2008-06-08 Says:

    [...] codeworks gnomedia » Blog Archive » LVM recovery tale. (tags: linux lvm recovery backup) [...]

  68. suse 10 reiserfsck rebuild tree dev sda1 Says:

    [...] lv and … The /boot and swap were /dev/sda1 and /dev/sda2. … November 27th, 2007 at 10:25 am …http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/Serg Iakovlev … is necessary to separate the unstable development tree … after making changes to [...]

  69. marosh Says:

    Many thx for this guide and thx to Mr. John (http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/#comment-2803) I was able to restore my data in 2 hdd, which is connected to HW-RAID 0 FC6 server I found that both hdd has the same info in the begining of partition. So after boot to knoppix I did this (dd if=/dev/”first/second hdd” of=”VolGroup00 bc=100000 count=1) and some editing in gvim and I get file for vgcfgrestore. After that I run fsck.ext3 -y and as I see most of my data is OK.

    THX again and sorry for my english.

  70. xwrs Says:

    I tell pvcreate /dev/sda9 /dev/sda5 Before it i had vg. Then I create new vg in same place. I lost my data. After $pvcreate -u sdSD-2343-SD939-adIda2 /dev/hda6 $pvcreate -u dk33kd-929293nd-adfja298a /dev/hdd1 $vgcreate -v system /dev/hda7 /dev/hdd1 $vgcfgrestore -f /etc/lvm/backup/system system i restore my volumes and data.Thank you very much