VMware Cloud Foundation (VCF) delivers a modern, automated infrastructure. A vital, but frequently neglected, element of its operation is enabling password rotations. The svc-nsx-vc-fqdn service account within the vCenter Server SSO domain and SDDC Manager. This account is critical for NSX or LCM functionality and integration between solutions, and its failure – typically due to an expired or incorrect password – can cause significant disruption to your VCF environment. This post will highlight the importance of svc-nsx-vc-fqdn, explain the impact of a failed account, and provide proactive measures to ensure its ongoing health and prevent costly outages.
Error Message – “NSX is integrated with vLCM feature of vSphere system. Cannot disable two way authentication and service account creation functionality”
Verifying Service Account with CURL
When the SDDC Manager Password Management UI displays an invalid or outdated timestamp for the svc-nsx-vc-fqdn service account, it indicates a potential synchronization issue between SDDC Manager and the underlying vCenter SSO domain. This can lead to authentication problems and network connectivity issues within your VMware Cloud Foundation environment. As illustrated below, the ‘Last Modified Date’ is incorrectly displayed as 11/10/23 1:03PM. Let’s examine this issue in more detail.
Addressing SDDC Manager UI Timestamp Discrepancies
To troubleshoot this, administrators can use the curl command to directly query the VCF SDDC API and verify the existence and timestamp of the svc-nsx-vc-fqdn
- SSH Into SDDC Manager – ( Prepare your command using your SSO account credentials. While ‘[email protected]‘ may be a default account, you must retrieve the current, rotated password using the
lookup_passwordutility within SDDC Manager. - Elevate to root ” su – “
- Run the Bearer Token curl to retrieve a access token
TOKEN=$(curl -d '{"username" : <sso_username>, "password" : "<sso_password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
Once we have gotten the Bearer Token we are now able to execute the next CURL command with syntax below:
curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_ppSo, now that we have execute the CURL command above you will get a detailed JSON API back which has all the creation times and ensuring that the SSO Service account still exists and has not disappeared even though the SDDC Manager UI is showing — or incorrect timestamp of last time it was active / updated / modified.
{
"creationTime" : 1699646593061, #This is the timestamp representing when the object was created
"credentialType" : "SSO" #This is indicating that it is apart of the SSO Domain
"entityId" : "17bd9679-a83b-485c-9b97-ac079827224d", # Unique identifier which is stored in the VCF SDDC Database.
"entityType" : "VCENTER", #Displays that this SSO object is related to vCenter
"id" : "b69362ad-c797-4689-b763-f1fc4aed5dff",
"modificationTime" : 1699646593061, #Timestamp showing when last time this object was modified
"serviceId" : "0dd05d81-6eaa-442b-9d47-fb1134ffdbe6", #This is the ID between NSX Managers and vCenter
"serviceType" : "NSXT_MANAGER",
"targetType" : "VCENTER",
"username" : "[email protected]" # Username of service account which is integrating the two solutions
}Now that you have executed the curl command, you should see output similar to the example JSON I’m providing above, where I’m commenting on the specific API fields of interest and explaining what they mean.
As demonstrated in the JSON outputs for other credentials, the ‘secrets’ field typically contains the encoded password within the API. However, for this specific SSO Account, this field is absent, allowing us to manually set the password on both sides of vCenter and NSX and SDDC and re-establish the connection.
Remediation of Service Account
After verifying that the SSO accounts still exist within SDDC, if you attempt to rotate them using the SDDC Password Manager UI, the initial rotation will often stall at approximately 50% and present a retry or cancel popup. Waiting the 15 minutes suggested in some KB articles and then retrying will result in the process reaching 100%, followed by a rapid failure. This is due to timing differences and synchronization processes between vCenter, NSX and SDDC.
First, lets set a basic password within vCenter SSO for the “[email protected]” Set it to VMware1!VMware1! , after you hit save wait few seconds
After the password has been updated manually via vCenter Users and Groups for the “[email protected]”.
Then log into your NSX Manager – > System -> Fabric -> Compute Managers -> Edit on vCenter Object and click edit on the FQDN | IP Address, and input your svc sso account and password.
- Note: Ensure that ‘Create Service Account’ is enabled and ‘Enable Trust’ is also selected. If you encounter an error during the connection process, toggle the ‘Create Service Account’ option, enter the password, save the configuration, and then re-enable the Service Account.
Finally, lets go back into the SDDC Manager -> Password Management -> Filter based on vCenter
Enter the remediated password you previously set within vCenter and NSX. Executing this operation should result in a successful completion task within a few minutes.
Now that two-way communication between vCenter and NSX has been re-established, you can resume standard operations, including enabling vLCM for baseline-controlled patching or image-based deployments.
Beyond the technical steps, this highlights the importance of treating credential management as a proactive, ongoing process. Regular audits, automated password rotation, and diligent verification using methods like the curl command we’ve explored, aren’t merely reactive fixes—they’re essential components of a robust security posture and a resilient infrastructure. Prioritizing these practices ensures that your VMware Cloud Foundation environment remains a reliable and secure foundation for your critical applications and workloads.
