Friday, June 30, 2017

Failed to mount Cisco Prime Collaboration Deployments export as NFS store to the ESXi host.

Running Cisco PCD 11.5.3 to migrate / upgrade CUCM and IM&P from version 8 to 11, and from physical to new virtual C240 chassis.  After installing PCD on one of the chassis and adding the CUCM cluster to inventory, I had trouble adding the new C240 ESXi hosts to inventory.

After entering the correct ESXi interface address and double checking the root password, I was consistently returned an error that says:

"Failed to mount Cisco Prime Collaboration Deployments export as NFS store to the ESXi Host.  Please look at the exception details in PCD logs and check the ESXi logs for further details of the exceptions reported on PCD."

There were no obvious events in VMware corresponding to trying to add the hosts to inventory in PCD.  The PCD logs did have entries showing Java exceptions but there was no verbose comments to indicate what raised the errors, nor could I find any Cisco documentation regarding deciphering the logs.

There are lots of discussions on the Internet and Cisco's support forums regarding the common causes of this problem.

  1. Your VMware license type may be unsupported.  If you have the Cisco UC Virtualization Hypervisor (appears as "Hypervisor Edition" in vSphere Client) license installed, you can remove it temporarily and use the Evaluation Mode license.  I had no license installed yet and was in Evaluation Mode.
  2. Network issues like firewalls between PCD and the host or poor DNS implementations may cause this.  In my case, PCD was running on the same host I was trying to import with no firewall or ACLs between the PCD and ESXi networks.  DNS resolution worked forward and reverse everywhere.
  3. Vmware being in lockdown mode may cause this.  This was easy to confirm from the VMware and ESXi configurations and I went so far as tring to set "utils os secure permissive" in the event something was overly restrictive in PCD's OS.

Ultimately my problem was an issue with an OVERLY COMPLEX PASSWORD on the ESXi root user account.

I didn't want to change the root user password as it was terribly complex and few knew it.  Adding a new user with a password comprised of just alpha characters allowed me to complete the ESXi import almost instantly though.

One challenge is ESXi 6 now enforces complex passwords by default, so this is easier said than done.

To support less complex password in ESXi 6 you can modify the security setting string under "Advanced Settings | Security".  The default ESXi 6 string is "retry=3 min=disabled,disabled,disabled,7,7"  From VMware's site "With this setting, passwords with one or two character classes and pass phases are not allowed, because the first three items are disabled. Passwords from three- and four-character classes require seven characters."

First I modified that string on the two chassis that would be running the enterprise. The much less restrictive string found in ESXi 5 is "retry=3 min=8,8,8,7,6".

Then I added a new user through my vSphere client that I would use for PCD purposes.  I planned to then delete it when the project was complete.  Here I added the "pcd" user, and because I already modified the ESXi security string I could use a simple password like "MyPassword".

Then I added the Administrator role to the pcd user so that PCD could actually manage the host.  This provides more privileges than is actually required but I did not want to troubleshoot permissions during the project and would be removing this account later regardless.

Adding the ESXi hosts to PCD inventory then worked instantly using the new "pcd" user and password.

In hindsight the PCD logs did reference:

2017-06-29 15:24:32,863 ERROR [pool-3-thread-14] db.DBEntity.hexStringToByteArray - Exception parsing int
java.lang.NumberFormatException: For input string: "y2"

I imagine if PCD is looking at the ESXi inventory password as a string of hex characters then using extended characters might break the process.  This is just conjecture though.