ESXiArgs Ransomware (2023) – How to recover a VM with no snapshots
This past February, thousands of ESXi servers were affected by a new ransomware attack (called ESXiARGS) that targeted the servers through a security flaw (tracked as CVE-2021-21972, CVE-2021-21973 and CVE-2021-21974), which is caused by a heap overflow issue in the OpenSLP service. The systems targeted appeared to be ESXi hypervisors in version 6.x and prior to 6.7.
In this article we will go over the steps to recover VMs that DO NOT have snapshots (-delta or -sesparse files), only a flat VMDK file. At this point, it does not seem like there is a process that works for everyone to successfully recover full VMs or their data if these had snapshots (-delta) files. Some people seem to have been able to recover VMs that had -sesparse snapshots.
I’m going to assume that you already have stopped the attack and regained control on your ESXi Hypervisor, and only concentrate on recovering the virtual machines.
These steps have worked for some people, but not for everyone. The probability of success depends on how much of your VM disk has been encrypted, which depends on how long the encryption script ran and/or if it failed.
The first thing to do, before anything else, create backup of the VM files:
Either create a copy of the actual VM folder, or
create a backup folder inside the VM folder and copy all files there.
Remove unnecessary files
The following files won’t be necessary for this process, and can safely be deleted. You should still have a copy of all of them in the backup you created previously, just in case.
.vmx – you restore it from the .vmx~ file.
You can remove them and recover the vmx file using the following commands:
There should now be two new files, a temp-flat.vmdk file and a temp.vmdk file. The original and new -flat.vmdk files should have the exact same size.
Edit the temp.vmdk file:
On line 9, replace temp-flat.vmdk with the name of the original -flat file, and
If your disk WAS NOT thin provisioned, remove or comment out line 19 ddb.thinProvisioned. If it was, leave the line as is.
Save the file.
# Disk DescriptorFile
# Extent description
RW 209715201 VMFS "WebServer_2-flat.vmdk"
# The Disk Data Base
ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "13054"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "cf65c0c190124ab5571025d1fffffffe"
#ddb.thinProvisioned = "1"
ddb.uuid = "60 00 C2 94 74 7b 9f 6f-36 7e b3 ce d1 90 72 2f"
ddb.virtualHWVersion = "14"
At this point you should have something similar to this:
[root@esxihost:]$ ls -la
drwxr-xr-x 1 root root 1120 Feb 26 18:01 .
drwxr-xr-t 1 root root 2100 Feb 4 15:56 ..
drwxr-xr-x 1 root root 3500 Feb 20 16:31 BKUP
-rw------- 1 root root 107374182912 Feb 3 10:18 VMNAME-flat.vmdk (ORIGINAL)
-rw------- 1 root root 1045 Feb 3 10:18 VMNAME.vmdk (Recreated & Edited)
-rwx------ 1 root root 3449 Dec 5 17:12 VMNAME.vmx (Recovered from .vmx~)
Let’s now test these .vmdk and -flat.vmdk files with a VM. To do this, you’ll have to do the following:
Create a new VM
Create a new VM folder (outside of the original VM folder) called “Test”.
You can do this either via the shell (SSH) or the Datastore Browser.
Copy the following files into this folder:
Create a new VM through the UI:
Same or similar CPU and RAM (I don’t think it makes a big difference).
Remove the default Hard Disk, and add a new hard disk, selecting Existing hard disk. Then browse to the “Test” folder created earlier, and select the .vmdk file in there.
Under “CD/DVD Drive 1”, select Datastore ISO File and browse to a linux live CD. In my case I used Kali Linux Live CD, which worked. Make sure “Connect at power on” is selected.
Start the VM.
Recover Partition with Testdisk
Boot the Live CD – in my case with graphical interface – to try to recover your partition and re-install Grub. These steps may vary depending on what partitions you had on your VM.
Open a terminal and use testdisk to recover your partition table. I followed this guide to recover my Linux partition. DO NOT RESTART as indicated in this guide.
Install testdisk if it is not installed (it comes with Kali Linux).
Select the No Log option and proceed.
Highlight the VM disk (usually /dev/sda), and select Proceed.
Select the partition type (Intel/PC).
Select Analyse and press enter.
Select Quick Search and press enter.
Highlight your partition if found (in my case Linux – it might be called different depending on your OS/setup) and press enter.
If the partition table looks good, choose the Write option, and confirm it in the next step by pressing Y.
When done, use Quit to exit testdisk.
Use fdisk -l to list and check your disk and partition.
DO NOT RESTART.
Re-install Grub with the following commands:
root@kali-live:~# mount /dev/sda1 /mnt
root@kali-live:~# mount --bind /dev /mnt/dev
root@kali-live:~# mount --bind /dev/pts /mnt/dev/pts
root@kali-live:~# mount --bind /proc /mnt/proc
root@kali-live:~# mount --bind /sys /mnt/sys
root@kali-live:~# chroot /mnt /bin/bash
root@kali-live:~# grub-install /dev/sda
Some guides I found online state to only use chroot /mnt when using the chroot command. This did not work for me and I was getting the following errors: chroot: failed to run command '/bin/bash' or chroot: failed to run command '/bin/zsh' – This is because if you don’t specify the path for bash, Kali Linux will attempt to chroot using the local shell environment, which cannot work. You need to specify the path and shell used by the OS you’re VM was using.
Some guides I found online state to run update-grub after the grub-install command. I did not do this.
Exit the chroot and reboot the machine.
Boot the VM
It should boot up the VM’s operating system from the -flat.vmdk file. You should be able to login and browse the file structure.
If this worked, the base VMDK and Flat file should be good! ✅ 🎉
Hopefully someone manages to find a way to recover Virtual Machines that have snapshots. If that happens, I’ll write about it!
Additional Information / Sources
The following are useful links for additional information: