Tonight’s topic – I want to share with you a recent headache I encountered while working with my VMware Cloud Foundation SDDC Manager 5.x and NSX Password Rotation for Audit account! I was in the middle of a routine password rotation service when suddenly, my task got stuck, leaving me scratching my head in frustration. I couldn’t believe how one little hiccup could bring my whole operation to a screeching halt. In this blog post, I will walk you through the issue I faced, how I troubleshooted it, and ultimately resolved it. So grab a cup of your favorite drink, sit back, and let’s dive into this tech challenge together!
Lets Begin!
- Take Snapshot (Uncheck Memory) of SDDC Manager
- SSH into SDDC Manager Appliance
- Elevate to Root ( su – )
Now we will start digging in the Postgres Database, we will try to find the culprit of what is holding up the lifecycle management of VCF.
The command below will display any locked tasks that are running or are stuck
psql --host=localhost -U postgres -d platform -c "select * from lock"My Issue – NSX Audit Password got stuck rotating and caused a halt in all operations, example below
{“serviceIdentifier”:”LCM”,”operationIdentifier”:”NSX_AUDIT”,”description”:”Resource of type NSX locked by service (ID: LCM) and operation (ID: NSX_AUDIT)”,”pollingInterval”:0,”expirationTime”:0}
Now that we have our locks displayed, for mine there was 2 locks I had to delete – eample below
psql --host=localhost -U postgres -d platform -c "delete from lock where id='ba4e6ff4-689a-4905-92ff-635cb7403698'";
psql --host=localhost -U postgres -d platform -c "delete from lock where id='ID_FROM_RESOURCE_NAME'";Next, we will remove the second lock from the database:
psql --host=localhost -U postgres -d platform -c "delete from lock where id='6bd393ba-ad8f-4e1a-a6c3-0695c4556c29'";
psql --host=localhost -U postgres -d platform -c "delete from lock where id='ID_FROM_RESOURCE_NAME'";
Now we have a healthy and happy SDDC Manager!
As well our password options are no longer blocked out!
Reboot and remove snapshot after you are done, ensure you have all working services before snapshot is removed or a good backup!