5

It appears that after a kernel update happened, and rebooted for, Raspbian will no longer boot. (RPi4) After entering the (correct) encryption key when prompted, the mount fails with error:

Cannot initialize device-mapper. Is dm_mod kernel module loaded?

If I CTRL+C enough I can drop to a BusyBox shell and attempt to cryptsetup manually, but that has the same problem, even after trying modprobe dm_mod, so I don't think that the device mapper kernel module is being included in the initramfs at all.

I can remove the SD card and mount it without issue in a different Debian PC, and am able to read/write the file system, including installing packages via apt and rebuilding the initramfs. The process I'm using to do this is:

# replace /dev/sdf with your SD card block device
# make sure you do that for the two `sed` calls for "/etc/crypttab" too!

# mount
cryptsetup -v luksOpen /dev/sdf2 sdcard
mount /dev/mapper/sdcard /mnt; mount /dev/sdf1 /mnt/boot; mount -o bind /dev /mnt/dev; mount -o bind /dev/pts /mnt/dev/pts; mount -t sysfs none /mnt/sys; mount -t proc  none /mnt/proc

# comment out ld.so.preload
sed -i 's/^/#/g' /mnt/etc/ld.so.preload
# crypttab needs to point to the current physical device during mkinitramfs or cryptsetup won't deploy
sed -i 's/mmcblk0p/sdf/g' /mnt/etc/crypttab

# copy qemu binary
cp /usr/bin/qemu-arm-static /mnt/usr/bin/

# chroot to Raspbian to update and rebuild initramfs
chroot /mnt /bin/bash
    rm -rf /var/tmp/mkinitramfs*
    apt update && apt upgrade && apt dist-upgrade && apt autoremove
    test -L /sbin/fsck.luks || ln -s /sbin/e2fsck /sbin/fsck.luks
    mkinitramfs -o /boot/initramfs.gz -k $(ls -t /lib/modules | tail -1)
    exit

# undo damage
sed -i 's/^#//g' /mnt/etc/ld.so.preload
sed -i 's/sdf/mmcblk0p/g' /mnt/etc/crypttab

# unmount
sync
umount /mnt/{dev/pts,dev,sys,proc,boot} /mnt
cryptsetup -v luksClose sdcard

These instructions have gone through revisions to try and resolve this. Most notably the "crypttab" replacement from "mmcblk0p" to "sdf", which is only there so when mkinitramfs runs within a chroot the encrypted partition is found. Otherwise the process shows "cryptsetup: ERROR: Couldn't resolve device /dev/mmcblk0p2". (Either value give the same error when really trying to boot.)

The boot command line "/boot/cmdline.txt" refers to the encrypted partition correctly:

console=serial0,115200 console=tty1 root=/dev/mapper/sdcard cryptdevice=/dev/mmcblk0p2:sdcard rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet plymouth.ignore-serial-consoles

Kernel version going on is "4.19.97-v8+" though I've tried "4.19.97+", "4.19.97-v7+", and "4.19.97-v7l+" too.

Other than a reinstall, would you have any advice on building a good initramfs to resolve this encryption issue?

Related questions:

Adambean
  • 220
  • 1
  • 10
  • 1
    You could symlink or mknod /dev/mmcblk0p1 to /dev/sdf1 and so on for the other partitions if you wanted to avoid all the sed stuff. Basically you have to figure out how to get the missing module into the initramfs. Most distros have a config file you can edit to list additional modules that should be included, e.g. this one for Ubuntu which might give you some hints. If you can figure that out, regenerating the initramfs as you've done should be all you need to get the system going again. – Malvineous Feb 14 '20 at 14:04
  • 1
    "Other than a reinstall, would you have any advice on building a good initramfs to resolve this encryption issue?" -> It's implied here that you've had this working, but normally Raspbian does not use an initrd of any kind. As far as I'm aware it requires an option set on the kernel command-line, but I don't see one here (mebbe there is a hardcoded default to something in the first partition?). Have you had it working before? – goldilocks Feb 14 '20 at 15:23
  • @Malvineous I didn't think of symlinking the devices to redirect mmcblk to sd. I'll try that right meow :) – Adambean Feb 14 '20 at 17:44
  • @goldilocks yep this system was running very normally until a reboot to apply a kernel update. The boot partition with the necessary configuration to start up Raspbian on the 2nd partition is still all there. – Adambean Feb 14 '20 at 17:44
  • 1
    @goldilocks initrd is deprecated but pretty sure it's impossible to build a modern kernel without initramfs. initrd was a separate file, initramfs is embedded into the kernel image itself. – Malvineous Feb 15 '20 at 02:10
  • 1
    @Malvineous WRT "is depecreciated": Yes, but initrd= is still the kernel parameter used to indicate the path of an external initramfs (which is why I wrote "any kind of" -- anyway, they're normally not compiled in, if you have a normal linux box, go look at the grub2 config, it will be using initrd=/path/to/initramfs). "it's impossible to build a modern kernel without initramfs": Enabling and building them is still an option; I'm on an x64 box with a custom 5.4 kernel built by me, no initramfs. The major reason for using an initrd/ramfs is to facilitate loading of drivers... – goldilocks Feb 15 '20 at 14:35
  • 1
    ...that the kernel might need to mount the root filesystem (at which point it can use /lib/modules; this is also the reason you need one with an encrypted filesystem -- unless you built the encryption drivers into the kernel). These drivers need to cover a range of commonplace hardware (and fs types), and it is not convenient to build them all into the kernel, hence the initramfs. Which also means you can build an initramfs and use it without rebuilding the kernel. – goldilocks Feb 15 '20 at 14:36
  • 1
    ..However, a kernel built for a very specific platform, (such as the Raspberry Pi kernel) doesn't need to bother with that because the specific drivers might as well be built in. Hence the default Pi kernel does NOT use an initramfs (and a stock Raspbian image does not include one), but it is capable of doing so without being recompiled. It's put in the boot filesystem. I'm just unsure whether it may use a default value, although searching online a bit reveals you can indicate an initramfs in /boot/config.txt (as opposed to the cmdline.txt method I mentioned). – goldilocks Feb 15 '20 at 14:36
  • 1
    @Adambean You should check /boot/config.txt to see if there is any reference to an initramfs in there. If not, I think this is a problem. – goldilocks Feb 15 '20 at 16:03
  • 2
    @goldilocks: Sorry you are right, it seems I only got as far in the docs as all kernels must have initramfs (even if it's just empty), but I didn't get to the part where it said you can pass an external one in using the initrd parameter and it will overwrite whatever is included in the built in (and possibly empty) initramfs. Of course now that I think about it it seems obvious, having to patch a precompiled kernel with a user-specific initramfs would be a huge headache! – Malvineous Feb 16 '20 at 09:17
  • Thanks for all this additional information both! I've got a lot to try out here, including the addition of module names to "/etc/initramfs-tools/modules" to force include modules for mkinitramfs. – Adambean Feb 16 '20 at 10:03
  • There is definitely an initramfs reference, as "initramfs initramfs.gz followkernel". "initramfs.gz" exists with a modified timestamp matching when I last ran mkinitramfs. – Adambean Feb 17 '20 at 08:08
  • Unfortunately all of the above has not resolved the issue. – Adambean Feb 17 '20 at 08:29
  • Well, I'm not sure which of the above steps did it, but after all of this and using specifically the "4.19.97-v7l+" kernel, it worked. This was a guess though, as I spotted in an initramfs shell that attempting insmod $(find /usr -type f -name "*dm-mod*") that it returned "insmod: can't insert ......: invalid module format". The only major difference in steps taken I can see is inserting extra kernel modules into "/etc/initramfs-tools/modules" and running update-initramfs -u prior to building, along with the symlinks to mmcblk. Not sure who's answer I should be accepting :) – Adambean Feb 17 '20 at 16:22

2 Answers2

1

Another solution for this problem (besides the comments under the question) which helped in my case on a raspberry 3b+ running on the raspbian OS:

sudo apt-get install --reinstall --yes raspberrypi-bootloader raspberrypi-kernel
sudo reboot

Taken from this thread: https://www.raspberrypi.org/forums/viewtopic.php?t=282802

my cryptsetup command worked afterwards, while having the same error like OP stated: Cannot initialize device-mapper. Is dm_mod kernel module loaded? previously.

ElectRocnic
  • 111
  • 2
0

This happened again today. Fortunately I had kept the script from previously, with the small addition of @ElectRocnic's apt install --reinstall --yes raspberrypi-bootloader raspberrypi-kernel.

VERY IMPORTANT: It mattered to ensure that the correct kernel module was given to mkinitramfs, because apparently this doesn't automatically happen during regular updates. Always look in "/lib/modules" yourself as $(ls -t /lib/modules | tail -1) doesn't always give you the right one. Not doing this will give you an error about "dm_mod" module not being loaded when entering your volume passphrase.

Full script is as follows:

# Replace /dev/sdg with your SD card block device.
# Use `lsblk` to help you find where this is.

Link "mmcblk" block device so chroot later sees the expected device block names.


test -L /dev/mmcblk0p1 || ln -s /dev/sdg1 /dev/mmcblk0p1
test -L /dev/mmcblk0p2 || ln -s /dev/sdg2 /dev/mmcblk0p2


Mount SD card.


cryptsetup -v luksOpen /dev/mmcblk0p2 sdcard
mount /dev/mapper/sdcard /mnt; mount /dev/mmcblk0p1 /mnt/boot; mount -o bind /dev /mnt/dev; mount -o bind /dev/pts /mnt/dev/pts; mount -t sysfs none /mnt/sys; mount -t proc  none /mnt/proc


Comment out ld.so.preload.


sed -i 's/^/#/g' /mnt/etc/ld.so.preload


Copy qemu binary.


cp /usr/bin/qemu-arm-static /mnt/usr/bin/


chroot to Raspbian to update and rebuild initramfs.


chroot /mnt /bin/bash
    rm -rf /var/tmp/mkinitramfs*
    apt update && apt upgrade && apt dist-upgrade && apt autoremove
    apt install --reinstall raspberrypi-bootloader raspberrypi-kernel
    test -L /sbin/fsck.luks || ln -s /sbin/e2fsck /sbin/fsck.luks
    update-initramfs -u
    # IMPORTANT: Replace "5.10.52-v7l+" with the correct kernel from "/lib/modules".
    # Doing $(ls -t /lib/modules | tail -1) here doesn't always give you the right one!!!
    mkinitramfs -o /boot/initramfs.gz 5.10.52-v7l+
    exit


Undo damage to our local recovery system.


sed -i 's/^#//g' /mnt/etc/ld.so.preload


Force flushing write buffers and unmount SD card.


sync
umount /mnt/{dev/pts,dev,sys,proc,boot} /mnt
cryptsetup -v luksClose sdcard


Unlink "mmcblk" block device.


test ! -L /dev/mmcblk0p1 || rm /dev/mmcblk0p1
test ! -L /dev/mmcblk0p2 || rm /dev/mmcblk0p2


All done. Eject the SD card and attempt boot in your Pi.

Did this from another Linux system, worked as normal.

N.B. The step to forcibly reinstall "raspberrypi-bootloader" and "raspberrypi-kernel" takes sodding ages. :) Didn't time it exactly, but it was about half an hour.

Adambean
  • 220
  • 1
  • 10