Can't add network in VMWare notes

While attempting to add an additional network in my VMWare environment, I was returned the popular "Call "HostNetworkSystem.UpdateNetworkConfig" for object "networkSystem" on ESXi "10.200.180.5" failed. Operation failed, diagnostics report: Error interacting with configuration file /etc/vmware/esx.conf: Write failed during Unlock. This is likely due to a full or read-only filesystem. Original Error was: Error interacting with configuration file /etc/vmware/esx.conf: Unable to write to file /etc/vmware/esx.conf.FcuxNQ while saving /etc/vmware/esx.conf operation aborted. It is likely this was caused by a Full Disk." error.

All documentation points to MAINSYS reaching a 32MB limit, likely a result of some run away IPMI errors or logging.

You can check volume usage by running "df -h" at the CLI, but to do so remotely, you need to enable remote tech support.

From the VSPhere client go to Configuration tab - Security Profile - Properties - Remote Tech Support (SSH) - Options button.  From there you can choose to Start Automatically, Start and Stop with host, or Start and Stop Manually.

When started, you can SSH to the host and run from CLI.

You can run "df -h" or "vdf -h" to identify issues.  Running vdf -h showed MAINSYS full at 32MB.

You can "ls -l" around to find offending files, in my case sel.raw and sel files unusually and relatively large.

cd /var/log/ipmi/0


-rw-r--r-- 1 root root 5800 Oct 25 19:55 fru
-rw-r--r-- 1 root root 17727 Oct 25 19:54 sdr_content.raw
-rw-r--r-- 1 root root 37 Oct 25 19:55 sdr_header.raw
-rw-r--r-- 1 root root 7208960 Oct 6 08:08 sel
-rw-r--r-- 1 root root 24600538 Oct 27 12:30 sel.raw
-rw-r--r-- 1 root root 35 Oct 27 12:30 sel_header.raw
-rw-r--r-- 1 root root 882 Oct 25 20:30 sensor_hysteresis.raw
-rw-r--r-- 1 root root 2676 Oct 25 20:30 sensor_readings.raw
-rw-r--r-- 1 root root 2192 Oct 25 20:30 sensor_threshold.raw

I won't comment on modifying IPMI settings or the safety of doing so, but you can remove the files via:

rm sel
rm sel.raw

/etc/init.d/sfcbd-watchdog restart


Additionally, on Cisco UCS systems, you can SSH to CIMC and investigate sel logging and clear them via:

Server# scope sel
Server /sel # show entries
Server /sel # clear