ESXiARGS Ransomware
LinuxGuidesTechnology

ESXiArgs Ransomware (2023) – How to recover a VM with no snapshots

This past February, thousands of ESXi servers were affected by a new ransomware attack (called ESXiARGS) that targeted the servers through a security flaw (tracked as CVE-2021-21972, CVE-2021-21973 and CVE-2021-21974), which is caused by a heap overflow issue in the OpenSLP service. The systems targeted appeared to be ESXi hypervisors in version 6.x and prior to 6.7.

In this article we will go over the steps to recover VMs that DO NOT have snapshots (-delta or -sesparse files), only a flat VMDK file. At this point, it does not seem like there is a process that works for everyone to successfully recover full VMs or their data if these had snapshots (-delta) files. Some people seem to have been able to recover VMs that had -sesparse snapshots.

I’m going to assume that you already have stopped the attack and regained control on your ESXi Hypervisor, and only concentrate on recovering the virtual machines.

Recovery Prerequisites

Disclaimer

These steps have worked for some people, but not for everyone. The probability of success depends on how much of your VM disk has been encrypted, which depends on how long the encryption script ran and/or if it failed.

Backup

The first thing to do, before anything else, create backup of the VM files:

  • Either create a copy of the actual VM folder, or
  • create a backup folder inside the VM folder and copy all files there.

Remove unnecessary files

The following files won’t be necessary for this process, and can safely be deleted. You should still have a copy of all of them in the backup you created previously, just in case.

  • .args
  • .vmxf
  • .vmsd
  • .vmsn
  • .vmem
  • .nvram
  • .vmx – you restore it from the .vmx~ file.

You can remove them and recover the vmx file using the following commands:

[root@esxihost:]$ mv VMNAME.vmx~ VMNAME.vmx
[root@esxihost:]$ rm *.args *.vmfx *.vmsd *.vmsn *.vmem *.nvram

Now your folder should look a little bit cleaner.

ESXiArgs – VM Recovery

Recreate VMDK file

  1. Note down the original size of the -flat file using the command ls -la:
[root@esxihost:]$ ls -la
total 92925984
drwxr-xr-x    1 root     root          1120 Feb 26 18:01 .
drwxr-xr-t    1 root     root          2100 Feb  4 15:56 ..
drwxr-xr-x    1 root     root          3500 Feb 20 16:31 BKUP
-rw-------    1 root     root  107374182912 Feb  3 10:18 VMNAME-flat.vmdk
-rw-------    1 root     root          1045 Feb  3 10:18 VMNAME.vmdk
-rwx------    1 root     root          3449 Dec  5 17:12 VMNAME.vmx
  1. Delete the existing .vmdk file by typing rm -rf xxx.vmdk:
[root@esxihost:]$ rm VMNAME.vmdk
  1. Create a new “temp” disk (-flat.vmdk and .vmdk) using the vmkfstools command. Make sure the size you specify is exactly the same as the size of the original -flat file when running ls -la:
[root@esxihost:]$ vmkfstools -c 107374182912 -d thin temp.vmdk
Create: 100% done.
  1. There should now be two new files, a temp-flat.vmdk file and a temp.vmdk file. The original and new -flat.vmdk files should have the exact same size.
  2. Edit the temp.vmdk file:
    • On line 9, replace temp-flat.vmdk with the name of the original -flat file, and
    • If your disk WAS NOT thin provisioned, remove or comment out line 19 ddb.thinProvisioned. If it was, leave the line as is.
    • Save the file.
# Disk DescriptorFile                                                                                      
version=1                                                                              
encoding="UTF-8"                                                                       
CID=fffffffe                                                                           
parentCID=ffffffff                                                                     
createType="vmfs"                                                                      
                                                                                       
# Extent description                                                                   
RW 209715201 VMFS "WebServer_2-flat.vmdk"                                              
                                                                                       
# The Disk Data Base                                                                   
#DDB                                                                                   
                                                                                       
ddb.adapterType = "lsilogic"                                                           
ddb.geometry.cylinders = "13054"                                                       
ddb.geometry.heads = "255"                                                             
ddb.geometry.sectors = "63"                                                            
ddb.longContentID = "cf65c0c190124ab5571025d1fffffffe"                                 
#ddb.thinProvisioned = "1"                                                             
ddb.uuid = "60 00 C2 94 74 7b 9f 6f-36 7e b3 ce d1 90 72 2f"                           
ddb.virtualHWVersion = "14"

At this point you should have something similar to this:

[root@esxihost:]$ ls -la
total 92925984
drwxr-xr-x    1 root     root          1120 Feb 26 18:01 .
drwxr-xr-t    1 root     root          2100 Feb  4 15:56 ..
drwxr-xr-x    1 root     root          3500 Feb 20 16:31 BKUP
-rw-------    1 root     root  107374182912 Feb  3 10:18 VMNAME-flat.vmdk  (ORIGINAL)
-rw-------    1 root     root          1045 Feb  3 10:18 VMNAME.vmdk       (Recreated & Edited)
-rwx------    1 root     root          3449 Dec  5 17:12 VMNAME.vmx        (Recovered from .vmx~)

Let’s now test these .vmdk and -flat.vmdk files with a VM. To do this, you’ll have to do the following:

Create a new VM

  1. Create a new VM folder (outside of the original VM folder) called “Test”.
    • You can do this either via the shell (SSH) or the Datastore Browser.
  2. Copy the following files into this folder:
    • VMNAME.vmdk
    • VMNAME-flat.vmdk
  3. Create a new VM through the UI:
    • Same or similar CPU and RAM (I don’t think it makes a big difference).
    • Remove the default Hard Disk, and add a new hard disk, selecting Existing hard disk. Then browse to the “Test” folder created earlier, and select the .vmdk file in there.
    • Under “CD/DVD Drive 1”, select Datastore ISO File and browse to a linux live CD. In my case I used Kali Linux Live CD, which worked. Make sure “Connect at power on” is selected.
    • Start the VM.
ESXi - Create new VM
Create a new VM but replace the default disk with the disk of the encrypted VM.

Recover Partition with Testdisk

  1. Boot the Live CD – in my case with graphical interface – to try to recover your partition and re-install Grub. These steps may vary depending on what partitions you had on your VM.
  2. Open a terminal and use testdisk to recover your partition table. I followed this guide to recover my Linux partition. DO NOT RESTART as indicated in this guide.
    • Install testdisk if it is not installed (it comes with Kali Linux).
    • Run testdisk.
    • Select the No Log option and proceed.
    • Highlight the VM disk (usually /dev/sda), and select Proceed.
    • Select the partition type (Intel/PC).
    • Select Analyse and press enter.
    • Select Quick Search and press enter.
    • Highlight your partition if found (in my case Linux – it might be called different depending on your OS/setup) and press enter.
    • If the partition table looks good, choose the Write option, and confirm it in the next step by pressing Y.
    • When done, use Quit to exit testdisk.
    • Use fdisk -l to list and check your disk and partition.
    • DO NOT RESTART.

Re-Install GRUB

  1. Re-install Grub with the following commands:
root@kali-live:~# mount /dev/sda1 /mnt
root@kali-live:~# mount --bind /dev /mnt/dev
root@kali-live:~# mount --bind /dev/pts /mnt/dev/pts
root@kali-live:~# mount --bind /proc /mnt/proc
root@kali-live:~# mount --bind /sys /mnt/sys
root@kali-live:~# chroot /mnt /bin/bash
root@kali-live:~# grub-install /dev/sda

Some guides I found online state to only use chroot /mnt when using the chroot command. This did not work for me and I was getting the following errors: chroot: failed to run command '/bin/bash' or chroot: failed to run command '/bin/zsh' – This is because if you don’t specify the path for bash, Kali Linux will attempt to chroot using the local shell environment, which cannot work. You need to specify the path and shell used by the OS you’re VM was using.

Some guides I found online state to run update-grub after the grub-install command. I did not do this.

  1. Exit the chroot and reboot the machine.

Boot the VM

It should boot up the VM’s operating system from the -flat.vmdk file. You should be able to login and browse the file structure.

If this worked, the base VMDK and Flat file should be good! ✅ ?

Hopefully someone manages to find a way to recover Virtual Machines that have snapshots. If that happens, I’ll write about it!

Additional Information / Sources

The following are useful links for additional information: