I’ve been dealing with some really heavy family stuff recently, so when I woke up this morning at 3am and couldn’t get back to sleep, I decided I needed to do something for myself today, despite the fact that it would mean needing to work over the weekend to finish stuff that was supposed to be finished today.
The activity I chose was to attempt to switch the last of my three home computers from Ubuntu to Debian. Yes, really, this is the kind of thing I do for fun.
The other two computers I already switched, my laptops, were easy. I don’t store any important data locally on them, and I have Ansible playbooks that configure nearly all of my customizations, so I was able to simply wipe and reinstall them. However, the remaining computer I set out to tackle this morning was more complicated, because it has a lot of data on it and acts as a server (e.g., backup, home automation, database, DNS) in addition to a desktop. Wiping and reinstalling was not an option.
Things were further complicated by the fact that its root filesystem is a LUKS-encrypted LVM volume, so it has a separate /boot partition by necessity. However, I did have one big thing working in my favor: a 500GB and a 1TB hard HDD were sitting in the computer doing with no data on them, so I could do whatever I wanted with them without losing any data.
When I started working on this today, I didn’t expect to get much further than getting Debian to dual-boot with Ubuntu. However, despite a false start, I made much more progress than expected: I’ve successfully managed to get all of my data and services transitioned over to Debian. Backups, home automation, database, and DNS are all confirmed working. I’m typing this blog posting on Debian right now, and I don’t think there’s any reason I’ll have to switch back to Ubuntu.
There’s a documented process for installing Debian onto a new disk from a running Linux system (as opposed to booting into the installer), so I decided to try that first. I created a LUKS partition filling the 1TB HDD, set up an LVM PV, VG, and LV on it, configured the LV as my new root partition and the 500GB HDD as my boot partition, and followed the documented steps to install Debian on them. Alas, I wasn’t able to get the Debian installation too boot. Grub was trying to find the initrd and vmlinuz for the Debian install on the Ubuntu boot partition, and when I tried to work around that by copying them over to the Ubuntu boot partition I ended up with a completely borked grub.cfg attempting to boot Ubuntu with the Debian kernel. Definitely not what I wanted. So I threw away a couple hours of work and started over.
This time I booted into the Debian installer, figuring it would be able to figure out how to do dual-boot better than I . And indeed, its solution was to install itself as a completely separate operating system in UEFI, yielding separate Ubuntu and Debian installs that I can choose from by opening the BIOS boot menu.
This is the point at which I had the flash of inspiration which allowed me to finish pretty much the whole project in a single day: I realized that rather than needing to copy over all the data from Ubuntu to Debian, I could unlock and mount the encrypted Ubuntu root filesystem from Debian and bind mount a bunch of directories from the old filesystem into the new on and just use them directly. That’s how I ended up with this in my /etc/fstab:
/dev/mapper/ubuntu--vg-root /ubuntu ext4 defaults 0 2
/ubuntu/home/jik /home/jik none bind
/ubuntu/home/macback /home/macback none bind
/ubuntu/home/jik4back /home/jik4back none bind
/ubuntu/d /d none bind
/ubuntu/mnt/backup /mnt/backup none bind
/ubuntu/mnt/mac-backup /mnt/mac-backup none bind
/ubuntu/mnt/share /mnt/share none bindI was delighted to discover that if I added the encrypted Ubuntu filesystem to /etc/crypttab in my Debian install, rebuilt my initrd, and rebooted, then the Ubuntu filesystem would get unlocked automatically on boot; Debian is even smart enough to figure out that I used the same passphrase for both filesystems so it only prompts once.
It’s worth noting that I tweaked the user IDs and group IDs in /etc/passwd, /etc/group, etc. of the various accounts I had to create on Debian so they matched Ubuntu, to avoid file permission issues. It’s also worth noting that I saved myself a ton of work by sharing my home directory between Ubuntu and Debian as shown by the /home/jik mount in the above list.
The next step was to run the Ansible playbooks to reconstruct my customizations and services. This presented a novel challenge. I wanted the Debian install to run on the same IP address as the Ubuntu install so it would be a drop-in replacement for all the other machines that talk to it. I also didn’t want to remove the Ubuntu server from the Ansible inventory until I was absolutely certain I no longer needed to maintain it. So I ended up with two different host names in the inventory pointing at the same IP address, and as a result every time I ran an Ansible playbook it connected to the server twice, once under each name, and the outcome was not pretty. To fix this I replaced my static Ansible inventory with an inventory script which asks the server what its host name is to figure out whether it’s currently booted into Ubuntu or Debian, and removes the other host name from the inventory before returning the inventory to Ansible. Works like a charm. I confess I did spent an inordinate amount of time relearning how inventory scripts need to format their output, since it’s been quite a few years since I last wrote one.
Other changes I had to make to my Ansible code included:
- Review all references to the Ubuntu host-name in the code and figured out how to slot the Debian host name in either in place or (mostly) alongside of each.
- Delete a whole bunch of obsolete code that was hanging around not being used and unlikely to ever be used again, since deleting it was easier than switching it over to the new host name.
- Add a missing a GPG key for one of the APT repositories deployed via Ansible. This goes into the category of “I don’t know this ever worked on any machine.”
- This computer ha an NVIDIA graphics card, for which I was using the proprietary NVIDIA driver on Ubuntu, but I decided for the time being to just stick with the open-source Nouveau driver on the Debian side. My DisplayLink hub seems to be working fine with Nouveau and Xorg, though not with Wayland, so I’ll just be sticking with Xorg for the time being.
- It turns out the UFW profile for allowing web server access to the machine, which is called “Apache” on Ubuntu, is called “WWW Full” on Debian, so I had to conditionalize the Ansible code.
- I use Jailkit for setting up secure backup target environments (
/home/macbackand/home/jik4backare in the above list of mount points are both Jailkit jails). It’s a bit creaky and not actively maintained, but it still works if you hold its hand a little. I had to upgrade from 2.19 to 2.23 and tweak it in a couple different ways to get it to work. - Numerous other minor fixes and clean-ups of the kind that always seem to pop up when you run a Playbook from scratch on a new machine and you haven’t done that in a while.
Once all that was done and all the Playbooks were running cleanly, I had a few other things to do, including:
- Shut down MongoDB and MariaDB, copy their data directories over from the Ubunu disk, and restart them. This was an easily, painless way to get all my databases back up and running quickly.
- Configure Debian to talk to my printer.
- Copy over all the cron jobs from Ubuntu and make sure they’re all working, which most notably involved reinstalling some Python modules and rebuilding some
pipenvvirtualenvs. - Tweak my Synology Drive Client installation so it’s synchronizing into a local directory within each OS rather than my shared home directory, because uses the host name to as part of the client identifier so you can’t sync a shared directory from two different OS installs with different host names, even if they’re never running at the same time.
Because Debian is running on an HDD instead of an SSD, it is noticeably slower, especially when booting and logging in, but it’s plenty fast enough for my day-to-day work, and certainly faster than the Windows computer I’m stuck using for work. 😉 When I’ve been running without issues for a while I’ll overwrite the Ubuntu install with the Debian install and switch back to fully using the SSD to restore maximum performance.
I’m glad to be one step closer to being done with Ubuntu.
