Trying to move a VM running on 32bit debian to 64bit nixos. Before braking everything I'd like to create an image of the VM in qemu to try it. I only have access via ssh. Here's a log about what I'm trying to do.
Root partition of this VM is about 8GB big filled up with 6GB of data with ext4.
Let's try a very simple approach by just doing a dd image of the whole disk and convert it to a qcow2 image. It is not really clear whether we should use `qemu-img convert` or `qemu-img dd` but the manpage tells me that convert is for images so I guess dd would be more correct:
ssh root@vm 'qemu-img dd -f raw -O qcow2 bs=64k if=/dev/vdb of=-'
Turns out that qemu-img isn't able to write to pipes but there is a way to do this with nbdcopy(1) from libnbd[1].
[1] https://gitlab.com/nbdkit/libnbd
The run would then be:
size=$(ssh root@vm 'blockdev --getsize64 /dev/vdb')
qemu-img create -f qcow2 vm.qcow2 $size
ssh root@vm 'dd bs=64k if=/dev/vdb' | pv -s $size | \
nbdcopy -- - [ qemu-nbd -f qcow2 vm.qcow2 ]
Let's do the same with the other disk which only has swap but just so that we are complete:
mv vm.qcow2 vm-vdb.qcow2
size=$(ssh root@vm 'blockdev --getsize64 /dev/vda')
qemu-img create -f qcow2 vm-vda.qcow2 $size
ssh root@vm 'dd bs=64k if=/dev/vda' | pv -s $size | \
nbdcopy -- - [ qemu-nbd -f qcow2 vm-vda.qcow2 ]
So all combined together we can do this:
nix-shell -p pv libnbd qemu-utils
for disk in vda vdb; do
size=$(ssh root@vm "blockdev --getsize64 /dev/$disk")
qemu-img create -f qcow2 vm-$disk.qcow2 $size
ssh root@vm "dd bs=64k if=/dev/$disk" | pv -s $size | \
nbdcopy -- - [ qemu-nbd -f qcow2 vm-$disk.qcow2 ]
done
We can now convert the disk images to compress them a little just for fun and to save some space.
for disk in vda vdb; do
qemu-img convert -f qcow2 -O qcow2 -c -o compression_type=zstd \
vm-$disk.qcow2 vm-$disk-compressed.qcow2
mv vm-$disk-compressed.qcow2 vm-$disk.qcow2
qemu-img snapshot -c init vm-$disk.qcow2
done
This made our images from using 8.9GiB to using 2.9GiB.
Let's try to get this thing to boot with qemu:
qemu-system-x86_64 --enable-kvm -name vm -m 741 \
-smp 2,sockets=2,cores=1,threads=1 \
-drive file=vm-vda.qcow2,id=vm-vda,if=none \
-device virtio-blk-pci,bootindex=1,drive=vm-vda \
-drive file=vm-vdb.qcow2,id=vm-vdb,if=none \
-device virtio-blk-pci,bootindex=2,drive=vm-vdb
After doing a init=/bin/sh trick to reset the root password to something I haven't forgotten we're in. Let's try to get a serial or shell so that I can copy-paste.
-display curses
And let's try to get our network drive to be the same as in the original so that we can use it without changing anything on image:
-netdev user,id=eth0,net=<network/cicd>,host=<ip_of_host>,dns=<dns_entry>,ipv6-net=<network/cicd>,ipv6-host=<ip_of_host>
-device virtio-net-pci,mac=<mac_of_nic>,netdev=eth0
This seems to work. We'd like to connect via ssh so that we can work the same way as on the real remote machine so we need to add
hostfwd=tcp:127.0.0.1:2222-<ip_of_guest>:22
to the netdev. Now we can connect with `ssh root@localhost -p 2222`. I also added this to my .ssh/config for faster logins
Host vm
User root
HostName localhost
Port 2222
Just remember to remove the known host again later with
ssh-keygen -R '[localhost]:2222'
The full command I use now for qemu is:
qemu-system-x86_64 --enable-kvm -name vm -m 741 \
-smp 2,sockets=2,cores=1,threads=1 \
-drive file=vm-vda.qcow2,id=vm-vda,if=none \
-device virtio-blk-pci,bootindex=1,drive=vm-vda \
-drive file=vm-vdb.qcow2,id=vm-vdb,if=none \
-device virtio-blk-pci,bootindex=2,drive=vm-vdb \
-netdev user,id=eth0,net=<network/cicd>,host=<ip_of_host>,dns=<dns_entry>,ipv6-net=<network/cicd>,ipv6-host=<ip_of_host> \
-device virtio-net-pci,mac=<mac_of_nic>,netdev=eth0 \
-display curses
Here I did a lot of stuff assuming it would be easier to at first install 32bit NixOS and then upgrade to 64bit. But it is way easier to at first install a 64bit kernel and boot that so that we can run 64bit binaries.
dpkg --add-architecture amd64
apt update
apt install linux-image-4.15.0-213-generic:amd64
After a reboot we are on a 64bit kernel and can continue.
Let's install nix on the guest
curl -L https://nixos.org/nix/install | sh -s -- --daemon
and free some space
rm /var/log/auth.log.*
apt clean
apt autoremove
[ remove a lot of compilers with aptitude ]
Now we need to get a minimal nixos running and also make sure that we don't lock ourselves out when we switch to it.
nix-channel --add https://nixos.org/channels/nixos-unstable nixpkgs
nix-channel --update
nix-env -uA --always nixpkgs.nix nixpkgs.cacert
nix-collect-garbage -d
We need to fix our PATH without an interactive shell.
for a in /nix/var/nix/profiles/default/bin/nix*; do ln -s $a /usr/local/bin/; done
To generate a config we need to build nixos-install-utils on our own machine:
nix-env -iA nixpkgs.nixos-install-tools
and finally can generate our config:
LC_CTYPE=C nixos-generate-config
We're going to have to run this again on the real vm so that we get the correct output but for now we can use this to create our minimal system.
We also need to decide what files we want to keep when we're lustrating[2].
[2] https://nixos.org/manual/nixos/stable/#sec-installing-from-other-distro
Before we start, we're going to create a snapshot of our current VM image so that we can roll back if we broke something.
Now we can try to get a working configuration that we can boot and then ssh into it. The NixOS configuration won't be scope of this document. But which files to keep when lustrating should be.
Maybe it makes sense to create a snapshot after we've copied a system so that we don't have to copy everything every time after reverting:
system=/nix/store/system.nix
nix-copy-closure --to vm $system
ssh vm "nix-env --set --profile /nix/var/nix/profiles/system $system"
qemu-img snapshot -c warm vm-vda.qcow2
qemu-img snapshot -c warm vm-vdb.qcow2
> /etc/NIXOS
for a in /etc/ssh/ssh_host* /etc/motd /root/.ssh/authorized_keys \
/home/*/.ssh/{authorized_keys,known_hosts}; do
echo $a >> /etc/NIXOS_LUSTRATE
done
/nix/var/nix/profiles/system/bin/switch-to-configuration boot
reboot
When the disk images were growing too large I compressed them. Note that snapshots will get lost then:
for disk in vda vdb; do
qemu-img convert -f qcow2 -O qcow2 -c -o compression_type=zstd \
vm-$disk.qcow2 vm-$disk-compressed.qcow2
mv vm-$disk-compressed.qcow2 vm-$disk.qcow2
qemu-img snapshot -c init vm-$disk.qcow2
done
After everything is done I did some steps to clean up the VM:
rm /nix/var/nix/profiles/per-user/root/*
for user in <list of users>; do
chown $user:users /home/$user/.ssh
chmod 700 /home/$user/.ssh
done
chmod 700 /root/.ssh
nix-collect-garbage -d
nix-store --optimize
# add note about /old-root to motd
# go through /old-root and delete old system
I can now say that with these steps to train everything in a local VM I was able to upgrade the remote VM without locking myself out. I did make sure that my network configuration depended on the mac address of the NIC.