Today I’ve spent almost 8 hours in troubleshooting a SSO issue. And it turned out to be so small issue and easily assumed or shall I say “Ignored”. We were upgrading vCenter5.0 to vCenter5.1.
We choose simple installation.
What is simple installation ? In my words put all eggs in single basket
It installs all the below component on same server
1. Install SSO
2. Upgrade Inventory Service (Only in case 5.0) or install Inventory service (for 4.x)
3. Upgrade vCenter
We clicked next – next and next..it was really simple.
But then we tried logging to vCenter, it didn’t worked.
We realized we need to add identity source. So we add as shown below
UPDATED: Reuse Session will if you’re running SSO service under a domain account
NOTE When you use the authentication type Password for an identity source, you must update the identity source details whenever the password changes for the super administrator user. You update the password on the Edit Identity Source dialog box. Everyone is going to forget this and at least in India no one stays any company for more than three years. VMware recommends using a special service user to ensure that the password does not expire and lock out or disable the user account.
But Still it didn’t worked
We tweaked identity source but still it didn’t worked.
I opened vCenter logs and found this
“[UserDirectorySso] AcquireToken InvalidCredentialsException: Authentication failed: Authentication failed”
Authentication is failing but why?
Came across this thread ->http://communities.vmware.com/thread/418759. But No help.
It was not related to us.
Restarted everything again. We had taken a snapshot of vCenter, We reverted back. I was literally praying for snapshot to get reverted successfully. It did, thanks to VMware engineering.
It was like circular dependencies. If snapshot revert operations fails, vcenter is gone.
Lessons learn: vCenter snapshot is not necessarily is reliable way to revert back to original vCenter settings
UPDATE: Do not use vCenter to revert back snapshot, use the VI Client directly to login to ESXi and also it is recommended to take a offline snapshot of vCenter.
Started again now…no simple but steps by step this time.
1. Installed SSO. SSO didn’t detected active directory and therefore failed to add the identity source
2. Upgrade inventory services –All okay
3. Now upgraded vCenter. First warning message “users are not matching as per vCenter database and will be removed”
Got another error or should i say warning.
4. Then came another error
“Error 29113. Wrong input – either a command line argument is wrong, a file cannot be found or the spec file doesn’t contain the required information, or the clocks on the two systems are not synchronized. Check vm_ssoreg.log in system temporary folder for details.
This was funniest. It directed us to this KB. This KB says if your certificate has expired you will get this error.
Quite wrong. But there was hidden hint here. Why will certificate of vSphere 5.0 will expire.
Then someone of System administrator came to us and told his Windows OS is losing time.
That was the moment. It turns out it was time issue. Servers where not in synch and there was gap of 8 mins between AD and vCenter. Time difference should be less than 5 mins and I ended up spending 8 hours.
Helpful KB: 2033880
Read my another vCenter SSO experience here