Its tombstone is halfway carved already.

Last night, or I should say few hours ago, I did a very bad thing to that very dying drive — stabbing it in the back when it’s already bleeding, which was thought almost died just about three month ago. This time, no matter what I did, it just hung there after a few noises from read access attempts, and fsck hung on with this message always:


/backup: recovering journal

I unplugged the USB cord while it was busy reading and writing data. It was late and I would just let it do its thing, was going to bed. As usual, I unplug USB mouse, but this time, I pulled the wrong cable. They were stacking together, disk drive on the top port and mouse on the bottom, and you know what happened next.

I frakkingly pulled the wrong one, as soon as the plug out of socket, without even checking the screen, I cried out ‘oh shit, oopsie’ inside. I knew what I just did.

Attempted remounting, re-powering, fscking, nothing could get the drive mount. It just hung there and no physical read access as I could see. I turned off computer and went to bed, mentally prepared myself for the round two after.

After a few hours of sleep, it didn’t go as I hoped. Retried everything I had learned, nothing worked. I was almost giving up, since there actually was nothing important on the disk, but I tried another with Backup superblock:


# dumpe2fs /dev/sdb1 | grep superblock
dumpe2fs 1.42.10 (18-May-2014)
  Primary superblock at 0, Group descriptors at 1-5
  Backup superblock at 32768, Group descriptors at 32769-32773
  Backup superblock at 98304, Group descriptors at 98305-98309
  Backup superblock at 163840, Group descriptors at 163841-163845
[snip]

# fsck -b 32768 /dev/sdb1
fsck from util-linux 2.24.1
e2fsck 1.42.10 (18-May-2014)
Superblock needs_recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
/backup: recovering journal
[*** Still hangs, so I unplugged like the millionth time ***]
Error reading block 30878 (Attempt to read block from filesystem resulted in short read).  Ignore error<y>?

During the course of cursing myself for being an utter idiot, I noticed that block number and all those in dmesg started with exact same starting numbers over and over. However, I honestly didn’t know what to do.

I tried to see if I could somehow disable journal as the last attempt, but didn’t get the useful information at first until I found this post about unreadable journal, I think that’s exactly the same problem I was having.

It did work, I could mount it now:


# debugfs -w -R "feature ^needs_recovery" /dev/sdb1
debugfs 1.42.10 (18-May-2014)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file

# tune2fs -f -O '^has_journal' /dev/sdb1
tune2fs 1.42.10 (18-May-2014)

# mount -t ext2 -o ro /dev/sdb1 /mnt/usb
# [Hooray, it mounted and read normally, or it seemed]

I was not entirely sure, but I though the partition has been converted (back) to ext2, which was fine with me. I didn’t care about the recovery capability as long as the entire drive can still be accessible, losing data because of failures was fine with me.

But not like this hanging issue causing me almost re-formating the drive, while the data was still all there. I could lost everything for nothing even the everything was basically meaning nothing much to me and still is.

Nonetheless, it’s not all good, I noticed this message while trying operate on a file:


[Aug27 10:29] EXT2-fs (sdb1): error: ext2_lookup: deleted inode referenced: 5472723

Apparently, a good fsck was needed:


# fsck -pf /dev/sdb1
fsck from util-linux 2.24.1
/backup: Inode 5472718, i_blocks is 36104, should be 45208.  FIXED.
/backup: Inode 5472686 has illegal block(s).

/backup: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)

# fsck -f /dev/sdb1
fsck from util-linux 2.24.1
e2fsck 1.42.10 (18-May-2014)
Pass 1: Checking inodes, blocks, and sizes
Inode 5472686 has illegal block(s).  Clear<y>? yes
Illegal block #12 (3134239992) in inode 5472686.  CLEARED.
Illegal block #13 (3223730326) in inode 5472686.  CLEARED.
[snip]
Too many illegal blocks in inode 5472686.
[snip for more fixes, a lot more]

/backup: ***** FILE SYSTEM WAS MODIFIED *****
/backup: 1473/10059776 files (81.3% non-contiguous), 5744993/20103331 blocks

After fixing these while realizing how frakked the drive was just by accidental unplug, I decided to convert it back to ext3 by re-enabling the journal:


$ grep sdb /etc/mtab
/dev/sdb1 /mnt/usb ext2 rw 0 0

# tune2fs -j /dev/sdb1
tune2fs 1.42.10 (18-May-2014)
Creating journal inode: done
This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

$ mount /mnt/usb
$ [mounted without any errors]

$ grep sdb /etc/mtab
/dev/sdb1 /mnt/usb ext3 rw,noatime,noexec,nosuid,nodev,user=livibetter 0 0

Back in the business before its last breath.