Blog Inforeseau.com – Alexandre Blanc Cyber Security

Proxmox – ZFS – Dead drive on active VM, recover from replicated disk

So you run a proxmox cluster, and you have 3 nodes.

Your VMs are all replicated, using ZFS replication, and in HA mode.

Each ZFS volume runs on a single drive, because we don’t have too much money, and it’s an home setup, OR, your RAID back end went nuts and you lost a full physical volume.

Issue is, your VM did not migrate through HA to another node, because obviously, the PVE node was not down, as only 1 of the ZFS drives died.

Then, I had major power outage, and a second one, ending after auto reboot, with some VMs on one node, some on another, and, this specific VM (my nextcloud), with a 1TB virtual disk, that was on the dead physical disk.

At this stage, I have a failed VM, on an active node, without valid disk for the VM, BUT, thanks to the cluster, I still see with « zfs list » on the other nodes, that the replicated copies are still here !

So here are the steps, finally easier and faster than expected :

Move the VM manually in another node of the cluster, and this is very simple, as SSH with a shell on any PVE nodes :

mv /etc/pve/nodes/pve1/qemu-server/106.conf /etc/pve/nodes/pve6/qemu-server/

Now, you did it, refresh the web interface, and the VM is now back on the node that has a valid disk for your VM ! …but it won’t start because it was in failed state at HA level. Not an issue, on the VM, menu « more », manage HA, and set it to « disabled », validate. Then again, in the menu, set it to « started ».

VM should start again, using the replica of the disk !

Now, on the node that had the failed disk, I have SATA drives, I just hot unplugged the dead unit, and put a new disk.

Initiate the new disk as GPT, and we’ll need to add it to ZFS. Issue is, volume already exists. So we need to go to « datacenter » then « storage », double click on the needed ZFS Pool, and uncheck the node that had the failed drive.

Validate, and go add the disk on the node, in ZFS, with the same ZFS Pool name.
Now you have valid new ZFS volume.

Make sure your run the replication from the currently running node, to this empty volume !!! So has you have the data replicated again.

I was a bit stressed, even if I have backup on another machine at filesystem level with backuppc of my VM content, it’s always much much much more easy and quick to get the existing VM back on track.

I hope it will help you. I realized at least how easy and simple it is to move a VM that is not running from one host to another.

Have a great day, and go to hell power outage and dead drives ! (Yes I have UPS, surge protection… still).