Lessons learned from setting up a Linux desktop VM on a Windows 11 Home host

Bottom line up-front

For those who don’t care about the story:

• It’s easy to run a Linux VM on Windows 11 Home using VirtualBox.
• However, VirtualBox VMs take a significant, noticeable performance hit on Windows 11 because the operating system’s Hyper-V functionality monopolizes control over hardware virtualization, forcing VirtualBox to do slower, software virtualization.
• Furthermore, I had a lot of problems trying to use the VirtualBox Guest Additions in my VM with Debian. The display in my VM kept hanging and doing other bizarre things when I tried to change the screen resolution in the display settings.
• None of the instructions I found online for turning off Hyper-V in the OS worked with current Windows 11 Home. It may be possible, but I couldn’t figure out how to do it.
• Turning off Secure Boot while attempting to turn off Hyper-V could clear your TPM and cause you to be unable to log in and therefore lock you out of Windows. Be careful!
• To use hardware virtualization when you can’t turn off Hyper-V, you can install Hyper-V Manager and run your VM there instead of in VirtualBox.
• VM performance in Hyper-V is extremely good, much better than VirtualBox’s software virtualization.
• Unlike in VirtualBox, you can’t pass through USB devices into Hyper-V VMs without a huge amount of hackery I did not attempt, which means, e.g., no connecting a WebAuthn security key or a USB headset to a Hyper-V VM.
• Sound output also doesn’t pass through from a Hyper-V VM into the Hyper-V client, but sound can be passed through a Remote Desktop connection, so if you need sound from the VM you can enable Remote Desktop and then connect to the VM with the Windows Remote Desktop Client instead of the Hyper-V client.
• (Tangent: Theoretically you should also be able to pass some USB devices, including WebAuthn security keys, through Remote Desktop Client to a Windows VM, but I didn’t try this since I was working on setting up a Linux VM, not a Windows VM. It doesn’t appear that the Linux RDP server support this.)
• The maximum display resolution for a Hyper-V VM is 1920×1080, even if your physical display is larger than that.
• Windows display scaling interacts with Hyper-V in an annoying way; there are several workarounds. (This problem also applies to VirtualBox.)
• Using an auto-hiding dock inside a Hyper-V Linux VM is annoyingly challenging, but there’s a workaround. (This problem also applies to VirtualBox.)
• You can create a shortcut on your desktop to connect directly to a Hyper-V VM so you don’t have to launch Hyper-V manager every time.

Let’s dig into the details…

What I was trying to accomplish and why

I try to always have a spare laptop in the house, i.e., one more laptop than people who need them, so that when a laptop breaks we have a substitute we can swap in quickly while we’re dealing with the broken one. That’s preferable to having to rush out and buy whatever is available in a nearby store, where selection is limited compared to buying online, and prices are often higher as well. I don’t want to be forced to buy a bad laptop because we need one Now Now Now.

The complicating factor is that I use Linux but everyone else in the family uses Windows (except my wife, who uses a Mac, making her irrelevant to this story so that’s the last time I’ll mention it). The problem to solve is, how do we have a single spare laptop which can quickly be used to run Linux for me or Windows for one of the kids?

My previous attempt to solve this involved buying a laptop with Windows, shrinking the Windows partition down to half the disk, installing Linux into the other half of the disk, and configuring the laptop for dual boot. This worked, kind of, but it was a pain due to Secure Boot and how it interacts with dual boot. It wasn’t really reasonable to expect my kids to have to regularly enter a 48-digit Windows recovery key when trying to reboot the laptop (speaking of which: when is Microsoft going to get a clue and allow the recovery key to be encoded in a QR code which is read by the laptop camera?).

I recently had to replace the spare laptop, so this time I decided to do something different: instead of configuring the laptop for dual boot, I would leave it Windows-only and build out a Linux desktop virtual machine running on top of Windows. Thus, if I ever have to use the spare laptop because mine breaks, I can log into Windows, launch the VM and switch it into full-screen mode, and it’ll be close enough to working on a bare-hardware Linux laptop that I’ll be able to tolerate it and be productive until my actual laptop is repaired.

This is a viable strategy for me because I have a well-defined process and a ton of automation to help me build out new Linux desktops, which I’ve written about previously. Although I have a highly customized Linux desktop working environment, I can rebuild it from scratch in a less than two hours.

Attempt #1: VirtualBox

I use VirtualBox on Linux, and it’s also supported on Windows, so that was an obvious first choice for attempting to set up a Linux VM on a Windows laptop.

(Note: if you’re using VirtualBox for the first time, you may not be aware that in addition to installing the application itself you also need to download and install the VirtualBox extension pack. See the downloads page for links to both. Once you’ve installed the application, double-click on the downloaded extension pack to install it as well. You’ll need to download and install a new extension pack every time you upgrade to a new version of VirtualBox.)

I installed VirtualBox and its extension pack, downloaded the Debian network install ISO, configured a new VirtualBox VM connected to that ISO, installed Debian, and then started working may way through my customization checklist.

I immediately noticed that everything in the Linux VM was running much more slowly than I would have expected. The spare laptop has a AMD Ryzen 7040 with 8 cores, 5.1GHz max, and 16GB of RAM, so it should not be slow. I did a little bit of digging around online and discovered the problem: by default, Windows 11 takes control of the hardware virtualization layer within the OS because it uses it for various security-related tasks. It prevents VirtualBox from being able to access hardware virtualization, so VirtualBox has to fall back on its much slower software virtualization. Functionally speaking, everything works, but it’s visibly slow.

In addition to the speed issue, another issue I encountered in VirtualBox is that attempts to install the VirtualBox Guest Additions and attempts to change the screen resolution often resulted in weird behavior, such as the resize not having any impact, the display server becoming partially unresponsive even though the VM was clearly still running (I could SSH into it), and VirtualBox becoming confused about where my mouse was within the VM, e.g., the display server thought the mouse was in a location different from where I could see the mouse server.

At this point I could have shrugged my shoulders and said to myself, “Meh, this is just for backup, and it’s highly unlikely that I’ll never need to use it since I have two other laptops and a desktop that I can use. This was an interesting experiment, and I’ve proven it’s possible, so I can just live with the slower performance, and I can simply avoid doing the things that are causing the display issues.” In fact, that’s absolutely what I should have said to myself. But I am stubborn, and so what I said to myself instead was, “This is no longer about setting up a backup Linux desktop for me on this Windows laptop. This is me against the machine, now. Getting rid of the speed penalty is an ‘interesting problem’, and I enjoy interesting problems. Onward!”

Trying to take hardware virtualization back from Windows (spoiler: it does not go well)

The same posts I found online explaining why my VirtualBox VM was running so slowly (it’s in software virtualization mode because the OS is monopolizing hardware virtualization) also claim that it is possible to take back hardware virtualization from the OS so VirtualBox can use it.

The virtualization support in Windows is called Hyper-V. Hyper-V isn’t just used to run virtual machines. It is also used by various other pieces of the OS, mostly for security-related purposes. It is, apparently, difficult to get the OS to stop using it. I found several pages online claiming to provide step-by-step instructions for how to do that. All of them failed in one of two ways:

• they were out-of-date and completely non-functional, e.g., instructions saying to disable a specific Windows service from the command line but when the command is run Windows says the specified service doesn’t exist; or
• the provided steps executed as specified but did not have the intended effect, i.e., they did not cause the OS to stop using Hyper-V.

Furthermore, one set of instructions I found went well beyond not working: it completely broke my login, forcing me to go through an extensive recovery process to regain access to my Windows installation so I wouldn’t have to erase and reinstall Windows and rebuild the Linux VM from scratch.

Be very careful about disabling Secure Boot

Those of you who have been using Windows for a long time surely remember that in the Good Old Days, you created a local Windows account when you were configuring a new system, and then, if you wanted, you could log in and connect your Microsoft account to the local Windows account. Then, Microsoft started allowing you the option of logging in directly with your Microsoft account during initial configuration, but the option of using a local account was still available. Now, however, in Windows 11, Microsoft has made it virtually impossible to create a local account: you are forced to configure a Microsoft account when configuring a new machine, unless you are installing Windows from scratch and you build your installer using a tool which knows how to change the installer settings to allow a local account to be created. It’s an incredibly infuriating example of enshittification.

Usually when I’m installing Windows on a new machine that forces me to configure a Microsoft account as part of the initial setup, when I’m done with that I’ll immediately go into Settings or Control Panel and create a local account and switch my day-to-day use over to that account instead of the Microsoft account, or even delete the Microsoft account from the computer entirely. I don’t use any of Microsoft’s online services most of the time, and I don’t want my computer connected to Microsoft if I can avoid it. For whatever, reason I didn’t do that in this case. It turns out that was a mistake.

When you configure a Microsoft account while setting up a Windows computer, it prompts you to set up fingerprint login and then prompts you to add a “Windows PIN” you can log in with instead of your fingerprint. Neither of these are a traditional windows password, and—and this is important—they are both stored in the computer’s Trusted Platform Module (TPM) rather than inside Windows.

One of the pages I found purporting to explain how to force the OS to stop using Hyper-V claimed that if I disabled Secure Boot in my BIOS, then rebooted my computer, then rebooted it a second time, the OS would no longer be using Hyper-V, and then I could turn Secure Boot back on and proceed with using VirtualBox in hardware acceleration mode.

Except it turns out that when you disable Secure Boot, it clears your TPM, and suddenly you can’t log into Windows because your fingerprint and Windows PIN logins have both been erased from the PIN. They are gone, gone, gone, so even if you turn Secure Boot back on you won’t be able to log in.

Here’s what I ended up doing:

• I used the instructions I found in the first reply here to essentially hack into Windows and create a new, local admin account. The steps I followed were slightly different from what’s documented there:
  • Power-cycle the laptop three times. The first two times, when the spinning “Windows is loading” cursor appears, hold down the power button until the computer shuts off. The third time you power on it reboots into recovery mode.
  • You are prompted for your Windows recovery code at this point to unlock the hard disk since it’s encrypted by default with BitLocker (good thing I wrote down the recovery code when I installed Windows!).
  • Click “Advanced options” on the “Automatic Repair” screen.
  • Click “Troubleshoot” on the “Choose an option” screen.
  • Click “Advanced options” on the “Troubleshoot” screen.
  • Click “Command Prompt” on the “Advanced options” screen.
  • Run these commands, which replace the “Utility Manager” that’s available during boot (which is what opens when you click the little person icon on the boot screen) with a command prompt:
    c:
cd windows\system32
ren utilman.exe utilman.exe.bak
copy cmd.exe utilman.exe
    Note that the first line here, “c:“, is actually a command, to switch from the recovery mode drive to your real Windows drive. Don’t forget that part!
  • Exit from the command prompt and click “Continue” to boot Windows.
  • One you’re on the login screen, hit any key to make the login prompt appear, then click the little person icon, which now opens a command prompt because of the steps you did above.
  • In the command prompt, use these commands to create a new local admin account:
    net user /add [username] [password]
net localgroup administrators [username] /add
  • I don’t think the new account appears on the login screen right away, so I think you need to reboot to be able to see it and log in with it.
• The instructions at the page linked above claimed that the new admin account would be able to reset the PIN or password on my Microsoft account. That wasn’t the case. The local admin account was not allowed to change anything about the Microsoft account except switch it between normal and administrator user. So I had to delete the Microsoft account and recreate it from the admin account. However, before doing that…
• I copied the Linux VirtualBox VM files that I had started creating from my Microsoft account in C:\Users to the home directory of the new admin user, so I wouldn’t have to rebuild it from scratch after deleting and recreating the Microsoft account.
• After recovering in this way, I left the local admin account around for any future recoveries that might be necessary.
• I rebooted into recovery mode and opened a command prompt again as described above so I could put back Utility Manager:
  c:
cd windows\system32
del utilman.exe
ren utilman.exe.bak utilman.exe

In their infinite wisdom, Microsoft has decided that people should be completely locked out of their Windows and lose all their data if they turn off Secure Boot, reboot, turn it back on, and reboot again. This is patently ridiculous. To recover from this, they will have had to save their recovery key when they installed Windows; remember where they saved it; and be able to either figure out and execute the convoluted instructions above, or have it occur to them to bring the laptop to someone proficient enough with technology to do it for them, probably for a fee.

Microsoft’s answer to this is, “Well, if you back up your laptop into OneDrive than this isn’t a problem,” but as far as I know OneDrive backups don’t include applications that weren’t installed through the Microsoft app store, and furthermore, most people probably use more storage on their computer than they get for free from OneDrive, so this is just a backdoor method of forcing people to pay Microsoft a subscription fee for more OneDrive space.

It’s evil.

Attempt #2: Install Hyper-V Manager and convert VirtualBox VM to Hyper-V VM

If I can’t get Windows to release its death-grip on Hyper-V, then the next best thing is to run the VM under Hyper-V, which means running it in the Windows Hyper-V Manager rather than in VirtualBox. The catch is that Hyper-V manager and all the pieces necessary to use it don’t come installed in Windows 11 Home by default. Fortunately, you can add them for free, as documented here (archived here and here in case the original goes away).

After getting Hyper-V Manager installed, a followed the instructions here for converting the VirtualBox VDI file to a Hyper-V VHD file, attached it to a newly created Hyper-V VM, and tried to boot. It wouldn’t recognize the disk as bootable. I tried booting from a Debian ISO and using rescue mode to reinstall grub, but it still wouldn’t boot after I did that. I eventually gave up and ended up reinstalling Debian from scratch, but a pop-up I saw during the Debian install clued me into why I couldn’t get the disk converted from VDI to boot: the VirtualBox VM had been running in legacy BIOS mode instead of UEFI mode, whereas the VM I created in Hyper-V manager was running in UEFI mode. I think if I had created the Hyper-V VM in “Generation 1” mode instead of “Generation 2” mode the Debian disk would have booted fine, but it’s possible that I still would have had to use Debian rescue boot to reinstall grub and/or rebuild the initramfs.

In any case, I was able to install Debian into a Hyper-V Manager VM with no trouble, and both the performance penalty and the display issues I had been encountering with VirtualBox were gone. I did run into a few other issues, though, which brings us to…

Hyper-V Linux VM tips and tricks

Sound

When you use the Hyper-V Manager app to connect to the display of a running Hyper-V VM, sound being generated in the VM is not passed from the VM through to the app, so if you’re doing something in the VM that required sound, you’re out of look.

Or, more accurately, you would be out of luck if there weren’t a workaround. At least in recent versions of GNOME (your desktop environment may vary), the Remote Desktop server that allows you to connect remotely to your local display knows how to pass sound back through the Remote Desktop connection to the Remote Desktop client. If you configure desktop sharing (under “System” in the GNOME Settings app), make sure that your VM is configured to allow access on port 3389 (which, for good security, is not open to the global, public internet!), and then connect to your VM’s display using the Windows Remote Desktop client instead of Hyper-V Manager, you can hear the sound just fine. The performance is pretty good too, though there’s no denying that there is a slight visible delay in screen updates.

USB Devices

VirtualBox lets you connect any physical USB device, such as a WebAuthn security key or a headset, to a VM in pass-through mode, making the VM think the device is plugged into it. Hyper-V Manager does not support this. There are some pages online which describe incredibly convoluted ways to make it work; I did not try them.

While theoretically the Windows Remote Desktop Client knows how to pass through some USB devices to a remote system, the Linux remove desktop server (at least for GNOME) doesn’t support this on the server side, so if you absolutely need to use USB devices inside your VM, you’re going to need to stick with VirtualBox despite the speed penalty.

Screen resolution

While the Display Settings app in GNOME inside the VM allows you to select screen resolutions higher than 1920×1080, if you pick anything higher than that it’ll revert to the previous setting, since 1920×1080 is the maximum supported screen resolution for Hyper-V VMs. This is somewhat unfortunate if your physical screen resolution is higher than 1920×1080; in that case you can’t use the extra screen real estate inside the VM. When you switch Hyper-V Manager into full-screen mode the VM screen is centered on your physical screen and the surplus screen real estate is empty.

Windows display scaling

If you have a extremely high-resolution display, then you may have display scaling configured in your settings. For example, my spare laptop has a 13.5″ screen with a resolution of 2256×1504, and when I use Windows 11 without scaling (i.e., scaling factor of 100%), everything is really small and difficult to read.

However, the problem with using a larger scaling factor is that when you open a VM inside Hyper-V Manager, the window displaying the VM is scaled by that scaling factor just like everything else. This is a problem if you’ve got the resolution of the VM display cranked up, because that means with scaling it won’t all fit on your physical screen. That’s obviously not palatable. There are three workarounds:

1. Use full-screen mode in Hyper-V Manager. VM display scaling doesn’t happen in full-screen mode.
2. Connect to the VM using Remote Desktop Client as described above and use its full-screen mode, which also isn’t scaled.
3. Instead of using display scaling, change your desktop icon size (right click on the desktop, select “View”, select “Large Icons”); text size (in the Settings app, select “Accessibility” from the menu on the list and then “Text Size” in the Accessibility settings); and taskbar height, icon size, and button width (install and open Windhawk, install the “Taskbar height and icon size” mod in Windhawk, play with its settings until you like what you see). This isn’t a perfect solution because not everything obeys the text size setting, but you may find this is preferable to being forced to use full-screen mode to see your whole VM screen.

This problem also occurs with VirtualBox. The three workarounds listed above should work for VirtualBox as well, but there’s an additional workaround you can use for VirtualBox which doesn’t work with Hyper-V Manager:

1. Go to C:\Program Files\Oracle\VirtualBox.
2. Right click on VirtualBox.exe and select “Properties”.
3. Select the “Compatibility” tab.
4. Click “Change high DPI settings”.
5. Under “High DPI scaling override” enable “Override high DPI scaling behavior. Scaling performed by: Application”.
6. OK out of the properties dialog.

Problems opening auto-hiding GNOME dock

If you use the dash-to-dock extension in GNOME because you prefer the dock to the default GNOME behavior, and you have the dock configured to auto-hide, then you may find it challenging to get the hidden dock to open in your VM. This is because by default the dock requires “pressure” against the edge of the screen to open the dock, which means continuing to push the mouse pointer against the edge of the screen when it’s already there, which you can’t do when the “screen” is a window that’s part of a bigger screen.

To fix this you can change the dash to dock extension configuration in the VM not to require pressure:

1. Right click on the 3×3 dots icon at the bottom of the dock and select “Dash to Dock Settings”.
2. Click the little gear icon next to ‘Intelligent autohide”.
3. Uncheck the “Push to show” checkbox.

After these steps the dock will appear if you move the mouse slowly across the left edge of the screen.

Desktop shortcut

VirtualBox lets you save a shortcut to your desktop for a specific virtual machine, so you can launch the VM directly rather than needing to go through the VirtualBox app. I had to search a bit to find out how to do this. I found the answer here. In a nutshell.

1. Right click on the desktop, select new shortcut.
2. Use vmconnect.exe localhost "VM Name" as the location for the shortcut (including the quotation marks).
3. Open the shortcut properties.
4. Under “Advanced…”, enable “Run as administrator”.

In conclusion…

I hope you found some of this helpful. Share and enjoy!