I am often running prerelease versions of Ubuntu, for three reasons: (a) in general I like to help find with finding bugs in prerelease software that I use and benefit from, to give back to the community; (b) I specifically want to find and report bugs that impact me in Ubuntu prereleases to increase the likelihood that they will be fixed before release; and (c) prerelease versions often have new features that I am eager to use and don’t want to wait for the release to get.

Unfortunately, there’s a problem with this: on a somewhat regular basis, broken prerelease packages are released (there’s a reason they tell you not to run prereleases on real systems you’re trying to do real work on!), and my system suddenly becomes mostly or completely unusable. Worse, when new prerelease packages come out, the previous versions of the packages are usually deleted almost immediately from the repository, so you can’t easily downgrade to the working versions of the broken packages by using apt-get install to downgrade to the working version of the package.

I’ve been trying to figure out for a while how to solve this problem, and the barrier I kept running into was where I could find the package files for the old, working versions so that I could downgrade to them. I finally figured out today that they’re available for at least some amount of time on Launchpad, so I wrote a script to help with pulling old packages from there and installing them when I need to downgrade them.

The script reads a fragment from /var/log/dpkg.log that lists the upgrades you want to reverse. It parses that fragment to figure out what the package versions was before the upgrade, fetches those old package versions from Launchpad, and installs them.

You can find the script here.

Bonus content! How to figure out which update out of many broke your system

So, let’s say Ubuntu just released a big batch of prerelease updates, and one of them broke your system, and you just can’t figure out which one it is. Step one is to use the script described above to restore your system to a working state. But now you need to put a hold on the broken package so it won’t get updated again the next time you run an update, and you also presumably want to report the bug to Ubuntu so they can fix it, and when it’s fixed you’ll get a notification from the bug and you can remove the hold from the broken package.

The way you identify the culprit is by upgrading the packages you just downgraded one at a time, or more accurately one group at a time because packages sometimes have interdependencies and can’t be upgraded by themselves, and test between each update until you figure out the one that caused the problem.

Here is a script that’ll pick the first upgradable package that isn’t held, upgrade it and anything else that needs to be upgraded along with it, and then (assuming that succeeds) pull a list out of dpkg.log showing what was updated. Then you can test to see if the upgrade caused the problem you’re worried about. If so, then you can report the problem to Ubuntu with ubuntu-bug and then downgrade those packages again using the script above, and mark them with apt-mark hold so they won’t get upgraded until the bug is fixed.