Trying to move a VM running on 32bit debian to 64bit nixos. Before braking everything I'd like to create an image of the VM in qemu to try it. I only have access via ssh. Here's a log about what I'm trying to do. Root partition of this VM is about 8GB big filled up with 6GB of data with ext4. Let's try a very simple approach by just doing a dd image of the whole disk and convert it to a qcow2 image. It is not really clear whether we should use `qemu-img convert` or `qemu-img dd` but the manpage tells me that convert is for images so I guess dd would be more correct: ssh root@vm 'qemu-img dd -f raw -O qcow2 bs=64k if=/dev/vdb of=-' Turns out that qemu-img isn't able to write to pipes but there is a way to do this with nbdcopy(1) from libnbd[1]. [1] https://gitlab.com/nbdkit/libnbd The run would then be: size=$(ssh root@vm 'blockdev --getsize64 /dev/vdb') qemu-img create -f qcow2 vm.qcow2 $size ssh root@vm 'dd bs=64k if=/dev/vdb' | pv -s $size | \ nbdcopy -- - [ qemu-nbd -f qcow2 vm.qcow2 ] Let's do the same with the other disk which only has swap but just so that we are complete: mv vm.qcow2 vm-vdb.qcow2 size=$(ssh root@vm 'blockdev --getsize64 /dev/vda') qemu-img create -f qcow2 vm-vda.qcow2 $size ssh root@vm 'dd bs=64k if=/dev/vda' | pv -s $size | \ nbdcopy -- - [ qemu-nbd -f qcow2 vm-vda.qcow2 ] So all combined together we can do this: nix-shell -p pv libnbd qemu-utils for disk in vda vdb; do size=$(ssh root@vm "blockdev --getsize64 /dev/$disk") qemu-img create -f qcow2 vm-$disk.qcow2 $size ssh root@vm "dd bs=64k if=/dev/$disk" | pv -s $size | \ nbdcopy -- - [ qemu-nbd -f qcow2 vm-$disk.qcow2 ] done We can now convert the disk images to compress them a little just for fun and to save some space. for disk in vda vdb; do qemu-img convert -f qcow2 -O qcow2 -c -o compression_type=zstd \ vm-$disk.qcow2 vm-$disk-compressed.qcow2 mv vm-$disk-compressed.qcow2 vm-$disk.qcow2 qemu-img snapshot -c init vm-$disk.qcow2 done This made our images from using 8.9GiB to using 2.9GiB. Let's try to get this thing to boot with qemu: qemu-system-x86_64 --enable-kvm -name vm -m 741 \ -smp 2,sockets=2,cores=1,threads=1 \ -drive file=vm-vda.qcow2,id=vm-vda,if=none \ -device virtio-blk-pci,bootindex=1,drive=vm-vda \ -drive file=vm-vdb.qcow2,id=vm-vdb,if=none \ -device virtio-blk-pci,bootindex=2,drive=vm-vdb After doing a init=/bin/sh trick to reset the root password to something I haven't forgotten we're in. Let's try to get a serial or shell so that I can copy-paste. -display curses And let's try to get our network drive to be the same as in the original so that we can use it without changing anything on image: -netdev user,id=eth0,net=<network/cicd>,host=<ip_of_host>,dns=<dns_entry>,ipv6-net=<network/cicd>,ipv6-host=<ip_of_host> -device virtio-net-pci,mac=<mac_of_nic>,netdev=eth0 This seems to work. We'd like to connect via ssh so that we can work the same way as on the real remote machine so we need to add hostfwd=tcp:127.0.0.1:2222-<ip_of_guest>:22 to the netdev. Now we can connect with `ssh root@localhost -p 2222`. I also added this to my .ssh/config for faster logins Host vm User root HostName localhost Port 2222 Just remember to remove the known host again later with ssh-keygen -R '[localhost]:2222' The full command I use now for qemu is: qemu-system-x86_64 --enable-kvm -name vm -m 741 \ -smp 2,sockets=2,cores=1,threads=1 \ -drive file=vm-vda.qcow2,id=vm-vda,if=none \ -device virtio-blk-pci,bootindex=1,drive=vm-vda \ -drive file=vm-vdb.qcow2,id=vm-vdb,if=none \ -device virtio-blk-pci,bootindex=2,drive=vm-vdb \ -netdev user,id=eth0,net=<network/cicd>,host=<ip_of_host>,dns=<dns_entry>,ipv6-net=<network/cicd>,ipv6-host=<ip_of_host> \ -device virtio-net-pci,mac=<mac_of_nic>,netdev=eth0 \ -display curses Here I did a lot of stuff assuming it would be easier to at first install 32bit NixOS and then upgrade to 64bit. But it is way easier to at first install a 64bit kernel and boot that so that we can run 64bit binaries. dpkg --add-architecture amd64 apt update apt install linux-image-4.15.0-213-generic:amd64 After a reboot we are on a 64bit kernel and can continue. Let's install nix on the guest curl -L https://nixos.org/nix/install | sh -s -- --daemon and free some space rm /var/log/auth.log.* apt clean apt autoremove [ remove a lot of compilers with aptitude ] Now we need to get a minimal nixos running and also make sure that we don't lock ourselves out when we switch to it. nix-channel --add https://nixos.org/channels/nixos-unstable nixpkgs nix-channel --update nix-env -uA --always nixpkgs.nix nixpkgs.cacert nix-collect-garbage -d We need to fix our PATH without an interactive shell. for a in /nix/var/nix/profiles/default/bin/nix*; do ln -s $a /usr/local/bin/; done To generate a config we need to build nixos-install-utils on our own machine: nix-env -iA nixpkgs.nixos-install-tools and finally can generate our config: LC_CTYPE=C nixos-generate-config We're going to have to run this again on the real vm so that we get the correct output but for now we can use this to create our minimal system. We also need to decide what files we want to keep when we're lustrating[2]. [2] https://nixos.org/manual/nixos/stable/#sec-installing-from-other-distro Before we start, we're going to create a snapshot of our current VM image so that we can roll back if we broke something. Now we can try to get a working configuration that we can boot and then ssh into it. The NixOS configuration won't be scope of this document. But which files to keep when lustrating should be. Maybe it makes sense to create a snapshot after we've copied a system so that we don't have to copy everything every time after reverting: system=/nix/store/system.nix nix-copy-closure --to vm $system ssh vm "nix-env --set --profile /nix/var/nix/profiles/system $system" qemu-img snapshot -c warm vm-vda.qcow2 qemu-img snapshot -c warm vm-vdb.qcow2 > /etc/NIXOS for a in /etc/ssh/ssh_host* /etc/motd /root/.ssh/authorized_keys \ /home/*/.ssh/{authorized_keys,known_hosts}; do echo $a >> /etc/NIXOS_LUSTRATE done /nix/var/nix/profiles/system/bin/switch-to-configuration boot reboot When the disk images were growing too large I compressed them. Note that snapshots will get lost then: for disk in vda vdb; do qemu-img convert -f qcow2 -O qcow2 -c -o compression_type=zstd \ vm-$disk.qcow2 vm-$disk-compressed.qcow2 mv vm-$disk-compressed.qcow2 vm-$disk.qcow2 qemu-img snapshot -c init vm-$disk.qcow2 done After everything is done I did some steps to clean up the VM: rm /nix/var/nix/profiles/per-user/root/* for user in <list of users>; do chown $user:users /home/$user/.ssh chmod 700 /home/$user/.ssh done chmod 700 /root/.ssh nix-collect-garbage -d nix-store --optimize # add note about /old-root to motd # go through /old-root and delete old system I can now say that with these steps to train everything in a local VM I was able to upgrade the remote VM without locking myself out. I did make sure that my network configuration depended on the mac address of the NIC.