TL;DR: use device-mapper snapshot target

Recently I’ve decided to set up my 9800X3D + 9070XT gaming PC I assembled more than 1 year ago as a server with GPU passthrough in a dedicated VM for offline “dangerous” gaming on top of the physical Debian host, alongside the physical Windows host that I would only boot into for “legitimate” gaming.

The PC has three drives:

  • Samsung 990 PRO with heatsink 2TB as physical Windows system drive and for online games and IO-bound offline games
  • Geil P4S 4TB as physical Windows data drive and for offline games
  • HYVX4 2TB as physical Debian system drive

As I sometimes would prefer play games from my collection from the physical Windows drives directly in the VM than rebooting into the physical Windows, especially when there’s workload running on the Debian system, I prefer if the physical Windows drives are readable in the VM.

The simplest idea to pass through both of these disks surely have performance benefits, but if viruses and trojans went wild in the VM, while the system residing on QCOW2 could be rolled back, the physcial drives passed through then cannot be fixed, things just go boom. Besides, I don’t want the physical Windows drives accidentally modified even not under those circumstances, so writable physical is no-go.

The next simplest idea is to make the drives read-only, on the block device level under Linux. This is quite easy to do with a simple udev rules file:

> cat /etc/udev/rules.d/99-windows-ro.rules
SUBSYSTEM=="block", ACTION=="add", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX_1|GeIL_P4S_4TB_XXXXXXXXXXX_1", RUN+="/sbin/blockdev --setro /dev/%k"

The character | is used to match multiple drives, and each IDs are obtained through simple udevadm info:

> sudo udevadm info /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX
......
E: ID_SERIAL=Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX_1
......

Note a explicit command with RUN+= is used instead of ATTR{ro}="1", as the latter file is simply read-only, and writing 0 to there does not work.

These rules put the drive into read-only state once after boot (albeit revertable through explicit commands). The read-only state could be verified from the ro sysfs virtual file:

> ls -l /dev/disk/by-id/nvme-{HYV2TBX4_2280__XXXXXXXXXXXXXXX,GeIL_P4S_4TB_XXXXXXXXXXX,Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX}
lrwxrwxrwx 1 root root 13 Jul 28 17:17 /dev/disk/by-id/nvme-GeIL_P4S_4TB_XXXXXXXXXXX -> ../../nvme1n1
lrwxrwxrwx 1 root root 13 Jul 28 17:17 /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXX -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Jul 28 17:17 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX -> ../../nvme2n1

> cat /sys/block/nvme*n1/ro
0
1
1

One can write to these drives and confirm all writes would fail.

The read-only drives then can be assigned to the VM without worrying about they getting written, either by writing the path explicitly in virt-manager GUI, or with a disk node similar to the following one in XML / virsh edit:

<disk type='block' device='disk'>
    <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
    <source dev='/dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX'/>
    <target dev='vdc' bus='virtio'/>
    <readonly/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
</disk>

Note that the readonly state must be marked explicitly (also could through virt-manager GUI), otherwise the VM manager could not open the corresponding block device. The VM itself and the system running inside therefore knows the fact that the drive is read-only.

While Windows could mount such read-only drive, and content could be indexed and read, many games don’t like their executable residing in a read-only folder. If one would want to play such game from the read-only drive, the only possible workaround is to copy the whole thing into the writable VM drive.

If only we could fake the writability of such read-only drive and make the VM believe they can write something onto it. And it’s natural to come up the idea to lay some writable block layer on top of such read-only block, to get a writable block.

Luckily the device-mapper system in Linux is quite powerful and provides many useful “target”s, among them the dm-snapshot provides just what we need. The idea is pretty similay to overlayfs albeit on the block device level: it combines a optionally read-only low layer, a optionally persistent copy-on-write layer, to get a read-write block device, that starts as a merely “clone” of the low layer, and writes would only hit the CoW layer, not the bottm layer.

To assemble such dm, a command looks like the following:

sudo dmsetup create winData --table '0 8001573552 snapshot /dev/disk/by-id/nvme-GeIL_P4S_4TB_XXXXXXXXXXX /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part5 P 128'

The arguments might look frightening and hard to understand at the first glance, but it’s pretty structural:

  • dmsetup create to create a dm
  • winData is the name of the to-be-created dm, a symlink /dev/mapper/winData would be created to point to the actual numbered device (usually dm0)
  • --table begins the dm table declaration as a single long string as the following argumnent, and in it:
    • 0 marks the start sector of first target
    • 8001573552 marks the end sector of first target; the two combined means the assembled dm block device would have its 0 to 8001573552 - 1 sector “mapped” to the currrently declared target, with details coming next to it; so the size is also 8001573552 (end-begin), which could be obtained by cat /sys/block/nvmeXn1/size
    • snapshot sets the current dm target to dm-snapshot, and later table arguments should define its target-specific setting
    • /dev/disk/by-id/nvme-GeIL_P4S_4TB_XXXXXXXXXXX is the bottom layer, no write would hit it; in this case it’s the 4TB physical drive
    • /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part5 is the Copy-on-Write layer, all writes to the assembled dm would only hit this layer, with the whole chunk copied to there first before being modified; in this case it’s a 128GB physical partition, but it could also be a LVM LV, a loopback file, a RAM disk, or any block you can spare
    • P tells the CoW layer is persistent; i.e. on re-assembling after physical reboot, the assembled dm should still look the same on binary perspective; I chose the data drive as persistent as maybe sometimes I would store games here rather than the VM system drive, and I don’t want it gone away right after a physical reboot; if the CoW layer shall be dropped, it could easily be done by blkdiscard -f on the CoW layer without the dm assembled.
    • 128 is the count of sectors as chunk size, with a 512 sector size this means 512 byte/sector * 128 sector = 65536 byte = 64 KiB chunk size, which should be a friendly IO size for SSD, and reduces the metadata; the dm-snapshot documentation wrote 16 in its example for lvm snapshot, so you can choose freely

For a non-persistent case, the command is similar except the persistent mark is N

sudo dmsetup create winSystem --table '0 3907029168 snapshot /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part4 N 128'

After a full assembling, the lsblk output would look like the following:

NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
nvme2n1       259:0    0  1.8T  1 disk
├─winSystem   254:0    0  1.8T  0 dm
├─nvme2n1p1   259:1    0  100M  1 part
└─nvme2n1p2   259:2    0  1.8T  1 part
nvme1n1       259:3    0  3.7T  1 disk
├─winData     254:1    0  3.7T  0 dm
├─nvme1n1p1   259:4    0   16M  1 part
└─nvme1n1p2   259:5    0  3.7T  1 part
nvme0n1       259:6    0  1.9T  0 disk
├─......
├─nvme0n1p4   259:10   0   32G  0 part
│ └─winSystem 254:0    0  1.8T  0 dm
├─nvme0n1p5   259:11   0  128G  0 part
│ └─winData   254:1    0  3.7T  0 dm
└─......

Note the assmebled block device does not have their partitton identified, which is by design, and for a VM access this is pretty OK. If you want the partitions however, this can be fixed with a manual kpartx call, e.g.

sudo kpartx -a /dev/mapper/winSystem

Then the lsblk output would look like the following:

NAME             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
nvme2n1          259:0    0  1.8T  1 disk
├─winSystem      254:0    0  1.8T  0 dm
│ ├─winSystem1   254:2    0  100M  0 part
│ └─winSystem2   254:3    0  1.8T  0 part
├─nvme2n1p1      259:1    0  100M  1 part
└─nvme2n1p2      259:2    0  1.8T  1 part
nvme1n1          259:3    0  3.7T  1 disk
├─winData        254:1    0  3.7T  0 dm
├─nvme1n1p1      259:4    0   16M  1 part
└─nvme1n1p2      259:5    0  3.7T  1 part
nvme0n1          259:6    0  1.9T  0 disk
├─...
├─nvme0n1p4      259:10   0   32G  0 part
│ └─winSystem    254:0    0  1.8T  0 dm
│   ├─winSystem1 254:2    0  100M  0 part
│   └─winSystem2 254:3    0  1.8T  0 part
├─nvme0n1p5      259:11   0  128G  0 part
│ └─winData      254:1    0  3.7T  0 dm
└─...

In this case, the writability of the dm-snapshot can be verified under Linux by simply mounting the NTFS partition with ntfs-3g, and writting something into it. And after umounting it you could mount the lower read-only NTFS partition to confirm no write hit it.

The VM can then use such pseudo writable drive, e.g.

<disk type='block' device='disk'>
    <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
    <source dev='/dev/mapper/winSystem'/>
    <target dev='vdc' bus='virtio'/>
    <address type='pci' domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
</disk>
<disk type='block' device='disk'>
    <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
    <source dev='/dev/mapper/winData'/>
    <target dev='vdd' bus='virtio'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
</disk>

Some systemd service units could also be used if the dm-snapshot blocks shall be ready upon boot:

> cat /etc/systemd/system/dmsetup-winSystem.service
[Unit]
Description=Setup winSystem device with dmsetup
Requires=dev-disk-by\x2did-nvme\x2dSamsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX.device
After=dev-disk-by\x2did-nvme\x2dSamsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX.device

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/dmsetup create winSystem --table '0 3907029168 snapshot /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part4 N 128'

[Install]
WantedBy=local-fs.target

> cat /etc/systemd/system/dmsetup-winData.service
[Unit]
Description=Setup winData device with dmsetup
Requires=dev-disk-by\x2did-nvme\x2dGeIL_P4S_4TB_XXXXXXXXXXX.device
After=dev-disk-by\x2did-nvme\x2dGeIL_P4S_4TB_XXXXXXXXXXX.device

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/dmsetup create winData --table '0 8001573552 snapshot /dev/disk/by-id/nvme-GeIL_P4S_4TB_XXXXXXXXXXX /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part5 P 128'

[Install]
WantedBy=local-fs.target

The Windows physical drives therefore, while read-only, appears writable in the Windwos VM, and upon physical Debian host reboot, changes made into the physical Windows system snapshot are gone, but made into the physical Windows data snapshot still persist but could also be throw away with blkdiscard, both physical drives always unmodified.