now browsing by category


How to recover file system corruption on 4T LVM using DDrescue on a VM

In this example we will be fixing a xfs filesystem that failed initial xfs_repair

If this happens don‚Äôt panic. We can fix most likely fix it. ūüôā

Steps to do

  • Clone VM to another datastore with lots of room 4T. So have at least 18T free to play with.
  • Increase vm memory to 32Gig of ram
  • Add another disk that is the same size or even slightly larger as the lvm with the corrupted.
    • Make sure you use a complete new datastore and that‚Äôs not being managed by DRS if your using vmware as this will be temporary.
  • Reboot the VM

Create new physical volume, volume group and logical volume

  • Create a new physical volume so that it can be added to a new volume group
    • Fdisk -l (to find the disk)
    • Pvcreate /dev/sdx <–replace x with drive designation
  • Create volume group
    • Vgcreate recovery /dev/sdx <– use whatever drive designation was used in above step
  • Create logical volume
  • lvcreate -l 100%FREE -n data1 $VGNAME
    • lvcreate -l 100%free -n data1 recovery <– example
    • mkfs.xfs¬†/dev/recovery/data1
  • Mount the new lvm
    • Mount /dev/recovery/data1 /mnt/recovery

Now install ddrescue and make image of the corrupted file system on the new logical volume

o    sudo add-apt-repository universeo    sudo apt updateo    sudo apt install gddrescue

Make swap size 30gigs ‚Äď this is needed so when we repair the filesystem it doesn‚Äôt time out because it runs out of memory. Which tends to be the problem when trying to repair such large filesystems.

  • sudo dd if=/dev/zero of=/swapfile bs=30G count=2 <–move the /swapfile to a location where there enough room
  • chmod 0600 /swapfile
  • mkswap /swapfile
  • swapon /swapfile
  • swapon -s (should show the new swapfile)
    Sample outputs

o    Filename                        Type           Size    Used    Priorityo    /dev/sda5                               partition      3998716 704     -1o    /swapfile                               file           2097148 0

Create rescue image on new logical volume

ddrescue -d -r3 $oldfilesyetem imagefile.img loglocationpath.logfile

ddrescue -d -r3 /dev/recovery/data /mnt/recovery/recovery.img /mnt/recovery/recoverylog.logfile

  • this will take awhile to run, probably hours

Once the file is created we want to repair it using xfs_repair

  • xfs_repair -m 30000 /mnt/recovery/recovery.img
    • this will also take awhile to run:Logoutput:
  • – agno = 29
    – agno = 9
    – agno = 10
    – agno = 11
    – agno = 12
    – agno = 13
    – agno = 14
    – 20:02:48: check for inodes claiming duplicate blocks – 88951488 of 88951488 inodes done
    Phase 5 – rebuild AG headers and trees…
    – 20:02:57: rebuild AG headers and trees – 41 of 41 allocation groups done
    – reset superblock…
    Phase 6 – check inode connectivity…
    – resetting contents of realtime bitmap and summary inodes
    – traversing filesystem …
    – traversal finished …
    – moving disconnected inodes to lost+found …
    Phase 7 – verify and correct link counts…

Note: If the above step fails for any reason or does not complete check dmesg as it will probably tell you why it failed.  

  • once complete you should be able to mount the image
  • mount recovery.img /mnt/recovery/data1
    • if successful it should mount under when you do
      df -h
      /dev/loop0 0T  1.2T  1.9T  38% /mnt/recovery/data1

By Nick Tailor