Ubuntu 18.04 Crashing on vmWare ESXi Server and Carrots

Internal scenario recently. Some Ubuntu Server guest VMs running on top of vmWare ESXi server. Everything running fine, no issues, then the Ubuntu guest VMs will just randomly become utterly unresponsive.

Nothing happening before, the only thing on the Ubuntu guest logs are just a crap ton of carrot jibberish:

"^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@"

Luckily I came across this post related to Mastodon that described the exact same event:

Related #1: Join Mastodon

And releated info from vmWare:

Related #2: vmWare 2151480

Same environment:

  • Linux VM is running kernel >= 4.8
  • HW version of VM is >=13
  • ESXi version is 6.5

After applying the recommended work arounds and subsequently updating the environment this appears to have been stabilized.

Resolution:

This is a known issue affecting VMware ESXi 6.5.
This issue is resolved in VMware ESXi 6.5 Update 1, available at VMware Downloads.
To work around this issue if you do not want to upgrade, use any one of these options.
  • Add the vmxnet3.rev.30 = FALSE parameter in the vmx file of virtual machine:
    1. Power off the virtual machine.
    2. Edit the vmx file and add the below parameter:

      vmxnet3.rev.30 = FALSE

    3. Power on the virtual machine.
  • If you do not want to power off the virtual machine, disable the receive data ring for each vmxnet3 vNIC on the VM by running this command:

    ethtool -G ethX rx-mini 0

    Note: Replace ethX with virtual machine interface name.