Ygtjki

I have an external hard drive connected to my Ubuntu laptop via USB. The whole hard drive is LUKS encrypted. Next to the encryption layer sits an LVM volume which I mount to /mnt/es. Once opened, lsblk sees it all as:

sdc 8:32 0 2.7T 0 disk 
└─es (dm-6) 252:6 0 2.7T 0 crypt 
 └─externalstorage-externalstorage (dm-7) 252:7 0 2.7T 0 lvm /mnt/es

Sometimes somehow the hard drive gets "detached" from the volumes, in which case the data becomes inaccessible:

sdc 8:32 0 2.7T 0 disk 
es (dm-6) 252:6 0 2.7T 0 crypt 
└─externalstorage-externalstorage (dm-7) 252:7 0 2.7T 0 lvm

I noted that it typically happens when the laptop goes to sleep.

Now, the problem is that once sdc has [been?] detached, I cannot get rid of the two stale volumes es and externalstorage-externalstorage in order to reuse their names again. The first thing I do is umount /mnt/es which goes fine. But then, even though the volume is not mounted, I cannot remove it:

dmsetup remove --force /dev/mapper/externalstorage-externalstorage
device-mapper: resume ioctl on externalstorage-externalstorage failed: Invalid argument
device-mapper: remove ioctl on externalstorage-externalstorage failed: Device or resource busy
Command failed

The info command shows that the volume is opened:

dmsetup info -c /dev/mapper/externalstorage-externalstorage
Name Maj Min Stat Open Targ Event UUID 
externalstorage-externalstorage 252 7 L--w 1 1 0 LVM-R4bAWzxJ8Cy3MBIjmPps60Rd3cFVyBStxTeKaR6gBHdefTYfJNWhHfA8tzqOBHns

Here is what seems to be holding the volume but it does not tell me much:

fuser -m /dev/mapper/externalstorage-externalstorage
Cannot stat file /proc/5687/fd/4: Stale file handle
Cannot stat file /proc/5687/fd/5: Stale file handle
Cannot stat file /proc/5687/fd/6: Stale file handle
Cannot stat file /proc/5687/fd/7: Stale file handle
Cannot stat file /proc/5687/fd/11: Stale file handle

From LVM point of view it is not accessible either:

pvdisplay
 /dev/externalstorage/externalstorage: read failed after 0 of 4096 at 2981780979712: Input/output error
 /dev/externalstorage/externalstorage: read failed after 0 of 4096 at 2981781037056: Input/output error
 /dev/externalstorage/externalstorage: read failed after 0 of 4096 at 0: Input/output error
 /dev/externalstorage/externalstorage: read failed after 0 of 4096 at 4096: Input/output error

Interestingly, at this point, cryptsetup luksClose es can be repeated any number of times without any visible impact or error message.

So how can I get rid of those stale volumes (apart from rebooting)? And is there any way to prevent the problem happening in the first place, i.e. why does sdc detach now and again?

[Ubuntu 14.04.3 LTS, kernel 3.19.0-42-generic]

Update

vgchange -an yields similar errors as above:

/dev/mapper/es: read failed after 0 of 4096 at 3000590794752: Input/output error
/dev/mapper/es: read failed after 0 of 4096 at 3000590876672: Input/output error
/dev/mapper/es: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/es: read failed after 0 of 4096 at 4096: Input/output error
/dev/externalstorage/externalstorage: read failed after 0 of 4096 at 2981780979712: Input/output error
/dev/externalstorage/externalstorage: read failed after 0 of 4096 at 2981781037056: Input/output error
/dev/externalstorage/externalstorage: read failed after 0 of 4096 at 0: Input/output error
/dev/externalstorage/externalstorage: read failed after 0 of 4096 at 4096: Input/output error

It's more like a guesswork, but it looks like you will need to deactive the lvm before you can remove the crypt mapping, basically working your way back out from the inside:

First, you'll need to umount any stale fs on the lvm:

umount -f /mnt/es

Then deactivate the lvm that you appear to have on top of cryptsetup. Even if you have other vgs, the following should deactivate anything it can, without harming the system if it using any other vg:

vgchange -an

After this, you can use dmsetup to remove the mapping created by cryptsetup(!)

dmsetup remove externalstorage

(I may have mistook some of the names, feel free to correct it)

It seems the basic problem that during sleep, your USB drive is dropping off the bus, and when it comes back, the block system finds that sdb disappeared, though there is an sdc now.

1. Close all users of the device and backup device data as needed.


2. Use umount to unmount any file systems that mounted the device.


3. Remove the device from any md and LVM volume using it. If the device is a member of an LVM Volume group, then it may be necessary to move data off the device using the pvmove command, then use the vgreduce command to remove the physical volume, and (optionally) pvremove to remove the LVM metadata from the disk.


4. If the device uses multipathing, run multipath -l and note all the paths to the device. Afterwards, remove the multipathed device using multipath -f device.


5. Run blockdev –flushbufs device to flush any outstanding I/O to all paths to the device. This is particularly important for raw devices, where there is no umount or vgreduce operation to cause an I/O flush.


6. Remove any reference to the device's path-based name, like /dev/sd, /dev/disk/by-path or the major:minor number, in applications, scripts, or utilities on the system. This is important in ensuring that different devices added in the future will not be mistaken for the current device.


7. Finally, remove each path to the device from the SCSI subsystem. To do so, use the command echo 1 > /sys/block/device-name/device/delete where device-name may be sde, for example.
   Another variation of this operation is echo 1 > /sys/class/scsi_device/h:c:t:l/device/delete, where h is the HBA number, c is the channel on the HBA, t is the SCSI target ID, and l is the LUN.

For more details refer Red Hat documentation!

Also you can use sg_utils script to remove stale volume using rescan-scsi-bus.sh!