Category: Recovery

How to recover file system corruption on 4T LVM using DDrescue on a VM

How to recover file system corruption on 4T LVM on ubuntu using a VM

In this example we will be fixing a xfs filesystem that failed initial xfs_repair

If this happens don’t panic. We can fix most likely fix it.

Steps to do

 Increase vm memory to 32Gig of ram
 Add another disk that is the same size or even slightly larger as the lvm with the corrupted.
 Make sure you use a complete new datastore and that’s not being managed by DRS if your using vmware as this will be temporary.
 reboot the VM


Create new physical volume, volume group and logical volume

 Create a new physical volume so that it can be added to a new volume group
 Fdisk -l (to find the disk)
 Pvcreate /dev/sdx  <- replace x with drive designation
 Create volume group
 Vgcreate recovery /dev/sdx < – use whatever drive designation was used in above step
 Create logical volume
 lvcreate -l 100%FREE -n data1 $VGNAME
 lvcreate -l 100%free -n data1 recovery <- example
 mkfs.xfs /dev/recovery/data1
 Mount the new lvm
 Mount /dev/recovery/data1 /mnt/recovery

Now install ddrescue and make image of the corrupted file system on the new logical volume

 sudo add-apt-repository universe
 sudo apt update
 sudo apt install gddrescue


Make swap size 30gigs – this is needed so when we repair the filesystem it doesn’t time out because it runs out of memory. Which tends to be the problem when trying to repair such large filesystems.

 sudo dd if=/dev/zero of=/swapfile bs=30G count=2 <- move the /swapfile to a location where there enough room
 chmod 0600 /swapfile
 mkswap /swapfile
 swapon /swapfile
 swapon -s (should show the new swapfile)
Sample outputs
 Filename        Type    Size  Used  Priority
 /dev/sda5 partition  3998716  704  -1
 /swapfile file    2097148  0

Create rescue image on new logical volume

 ddrescue -d -r3 $oldfilesyetem imagefile.img loglocationpath.logfile

ddrescue -d -r3 /dev/recovery/data /mnt/recovery/recovery.img /mnt/recovery/recoverylog.logfile

 this will take awhile to run, probably hours

Once the file is created we want to repair it using xfs_repair

 xfs_repair -m 30000 /mnt/recovery/recovery.img
 this will also take awhile to run:

– agno = 29

– agno = 9

– agno = 10

– agno = 11

– agno = 12

– agno = 13

– agno = 14

– 20:02:48: check for inodes claiming duplicate blocks – 88951488 of 88951488 inodes done

Phase 5 – rebuild AG headers and trees…

– 20:02:57: rebuild AG headers and trees – 41 of 41 allocation groups done

– reset superblock…

Phase 6 – check inode connectivity…

– resetting contents of realtime bitmap and summary inodes

– traversing filesystem …

– traversal finished …

– moving disconnected inodes to lost+found …

Phase 7 – verify and correct link counts…


 once complete you should be able to mount the image
 mount recovery.img /mnt/recovery/data1
 if successful it should mount under when you do df -h
 /dev/loop0 3.0T 1.2T 1.9T 38% /mnt/recovery/data1


Written By Nick Tailor