How To Update Service Control Manager

Recently we've had some weird problems on 1 of our customers vCenter Servers. For starters the vMotion and Storage vMotion features weren't working anymore considering of fourth dimension-outs. Which is weird and something I've never seen earlier. So nosotros started troubleshooting the VCSA server and noticed that it couldn't retrieve the installed licenses (VMware vSphere Enterprise Plus) from the production ESXi hosts anymore.

Going to the "Licensed Features" tab in the vSphere Client (VCSA version half-dozen.0 GA) usually gives you a nice overview of what vSphere license is installed, but this time it was just empty. Going to the ESXi host straight yous could yet meet that the license was present and activated. Nosotros besides noticed that the License module in the vSphere client was also providing us with a timeout.

Once nosotros pigeon into the log files from the license service in "/var/log/vmware/cis-license/license.log" we noticed some Security Token Service STS service, SSO service and web-client service issues in regards to certificates. Which got me thinking and looking at the certificates for this vCenter Server Appliance. Below you can find some snippets of logs which might be interesting for you lot to match your trouble to the one I was having:

license . log :

2019-05 -13T13 :49:10.674Z Timer -3 WARN core . management . maint . service . AssetInventoryMaintainerTimerTaskImpl Maintanance of the asset inventory failed .

com . vmware . cis . license . server . common . provider . ClientStubProviderException : com . vmware . vim . vmomi . client . exception . SslException : com . vmware . vim . vmomi . core . exception . CertificateValidationException : Server certificate chain not verified

Caused by : javax . net . ssl . SSLPeerUnverifiedException : peer not authenticated

at sunday . security . ssl . SSLSessionImpl . getPeerCertificates ( Unknown Source )

at com . vmware . vim . vmomi . client . http . impl . ThumbprintTrustManager $HostnameVerifier . verify ( ThumbprintTrustManager . java :296)

. . . 44 more

2019-05 -13T13 :22:43.443Z pool -3 -thread -1 WARN mutual . vmomi . authn . impl . SsoAuthenticatorImpl STS signing certificates are missing or empty

2019-05 -13T13 :22:43.601Z puddle -three -thread -ane Fault server . common . sso . impl . SsoAdminProviderImpl Refetch STS certificates failed

You can use the following cli cmdlets to check your certificate stores and the certificates that are in them:

/ usr / lib / vmware -vmafd / bin / vecs -cli entry list -- store MACHINE_SSL_CERT -- text | less

/ usr / lib / vmware -vmafd / bin / vecs -cli entry list -- shop machine -- text | less

/ usr / lib / vmware -vmafd / bin / vecs -cli entry list -- store vpxd -- text | less

/ usr / lib / vmware -vmafd / bin / vecs -cli entry listing -- store vsphere -webclient -- text | less

All certificates checked out simply guess what, the "MACHINE_SSL_CERT" didn't. Turns out information technology was expired. Funny affair though is that this particular vCenter Appliance should'nt even be working anymore because one time the certificate is expired, about of the fourth dimension it won't even start all of the vCenter services once you reboot information technology. In our example somehow it did.

So we went ahead and fired upwards the "certificate-manager" tool which can be found in "/usr/lib/vmware-vmca/bin/certificate-manager", picked option 3 to supervene upon the the Motorcar SSL with a VMCA certificate (which is a cocky-signed certificate only that's fine for this environment), entered the information which was present in the current document such as hostnames and IP-address information and accepted all changes.

Certificate-director tool on the vCenter Server Appliance

Once you lot accepted the change it is proposing it will update the certificates in the locations it is needed and finish and start all services. Easy. Our certificate-managing director nevertheless decided it was time to throw an error:

ERROR certificate -manager Fault while starting services , please run across log for more details

certificate -manager Error while replacing Automobile SSL Cert , please run across / var / log / vmware / vmcad / document -director . log for more data .

Once nosotros checked that log we saw that the certificate-managing director tooling couldn't start the "vmware-eam" service, see the below log snippet which can be constitute in "/var/log/vmware/vmcad/certificate-director.log":

two

seven

eleven

eighteen

thirty

Waiting for VMware ESX Agent Manager . . . . . . .

WARNING : VMware ESX Agent Manager may have failed to starting time .

Concluding login : Mon May 13 thirteen:22:44 UTC 2019 on console

Stderr =

2019-05 -13T13 :47:40.139Z {

"resolution" : null ,

"item" : [

{

"args" : [

"Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "

] ,

"id" : "install.ciscommon.command.errinvoke" ,

"localized" : "An fault occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'outset']\nStderr: '" ,

"translatable" : "An error occurred while invoking external control : '%(0)s'"

}

] ,

"componentKey" : null ,

"problemId" : zilch

}

ERROR : root : Unable to showtime service vmware -eam , Exception : {

"resolution" : null ,

"particular" : [

{

"args" : [

"vmware-eam"

] ,

"id" : "install.ciscommon.service.failstart" ,

"localized" : "An error occurred while starting service 'vmware-eam'" ,

"translatable" : "An fault occurred while starting service '%(0)due south'"

}

] ,

"componentKey" : cypher ,

"problemId" : null

}

Certain enough we were hitting a bug in our vCenter Server Appliance. This issues prevented the EAM service from starting after a vCenter reboot. This bug basically deletes the "eam.properties" file in the "/etc/vmware-eam/" directory. This file is crucial for the service to start and know what to practice. Since this file was missing in our environment, the "vmware-eam" service was broken. This VMware KB explains how to ready this. Which basically means that you have to download the attachment called "Recreate_eam.properties.sh" and run it. This script recreates the eam.properties file and so that your "vmware-eam" service tin can start again. Please not that you can only run this when you lot run the EAM service on the vCenter Server you are working on. The steps to run this script are described beneath:

two

half-dozen

vii

nine

Step one:

Download the script and upload it to your vCenter Server

Step ii:

Create a fill-in from the current eam . properties file ( if present ) . Don't forget to create a VM snapshot either

Footstep 3: Determine the host ID :

cat / etc / vmware / install -defaults / sca . hostid

Step 4: Decide the vCenter Server appliance hostname

hostname -f

Footstep five: Set permissions on the Recreate_eam . properties . sh file

chmod 777 Recreate_eam . properties . sh

Footstep 6: Run the Recreate_eam . properties . sh file :

. / Recreate_eam . backdrop . sh and enter the required information

Step 7: Check the / eam . properties for the right hostname and host ID which you lot nerveless before

Step 8: Get-go the "vmware-eam" service .

service -control vmware -eam start

Step 9: re -run the certificate manager with your previously entered information

/ usr / lib / vmware -vmca / bin / certificate -managing director and select Choice 3

In our state of affairs this most fixed our bug. We were forced to break the certificate-manager procedure in the middle where it starts starting the services over again after it updated the "MACHINE_SSL_CERT" in the places information technology has to. You can do this by just pressing CTRL+C on the right time in the procedure. To observe this correct time you can open up another putty session to the VMware vCenter server and using the following command:

tail -f / var / log / vmware / vmcad / certificate -manager . log

Just press CTRL+C when the post-obit log entries pass past:

three

six

xiii

sixteen

eighteen

xix

twenty

2019-05 -13T14 :xv:06.607Z INFO certificate -director Running control : - service -command -- stop -- ignore -- all

2019-05 -13T14 :fifteen:06.608Z INFO certificate -director delight encounter service -control . log for service status

INFO : root : Service : vmware -psc -client , Activity : stop

INFO : root : Service : vmware -syslog -health , Action : stop

INFO : root : Service : vmware -vsan -health , Activeness : end

INFO : root : Service : applmgmt , Action : stop

INFO : root : Service : vmware -eam , Action : end

INFO : root : Service : vmware -mbcs , Activity : cease

INFO : root : Service : vmware -netdumper , Activity : finish

INFO : root : Service : vmware -perfcharts , Activity : end

INFO : root : Service : vmware -rbd -watchdog , Activity : stop

INFO : root : Service : vmware -sps , Action : terminate

INFO : root : Service : vmware -vapi -endpoint , Action : stop

INFO : root : Service : vmware -vdcs , Action : stop

INFO : root : Service : vmware -vpx -workflow , Activeness : cease

INFO : root : Service : vmware -vsm , Activeness : stop

INFO : root : Service : vsphere -customer , Activity : stop

INFO : root : Service : vmware -vpxd , Activeness : terminate

INFO : root : Service : vmware -cis -license , Action : stop

INFO : root : Service : vmware -invsvc , Activity : stop

INFO : root : Service : vmware -vpostgres , Action : cease

INFO : root : Service : vmware -syslog , Activity : stop

INFO : root : Service : vmware -sca , Activeness : stop

INFO : root : Service : vmware -vws , Action : stop

INFO : root : Service : vmware -cm , Activeness : cease

INFO : root : Service : vmware -rhttpproxy , Activity : stop

INFO : root : Service : vmware -stsd , Action : finish

INFO : root : Service : vmware -sts -idmd , Action : finish

INFO : root : Service : vmcad , Action : end

INFO : root : Service : vmdird , Activity : stop

INFO : root : Service : vmafdd , Activity : stop

2019-05 -13T14 :15:52.728Z INFO certificate -director Command executed successfully

2019-05 -13T14 :15:52.728Z INFO certificate -manager all services stopped successfully .

2019-05 -13T14 :15:52.728Z INFO certificate -manager None

2019-05 -13T14 :fifteen:52.729Z INFO document -director Running control : - service -control -- starting time -- all

2019-05 -13T14 :xv:52.729Z INFO document -manager please run across service -control . log for service condition

One time you are at this point just start the services yourself with:

service -control -- first -- all

This should start all the services nicely. After this signal we had our VMware vCenter Server Appliance working again with a new fresh "MACHINE_SSL_CERT" certificate. Every bit a final bank check you tin execute the following control and verify the expiration date:

/ usr / lib / vmware -vmafd / bin / vecs -cli entry list -- store MACHINE_SSL_CERT -- text | less

There you have it. I figured it would exist like shooting fish in a barrel plenty and set this rapidly, turned out we were facing a bug in the "vmware-eam" service. I promise this mail service helps when you are finding the aforementioned issues we found.

Source: https://vcloudvision.com/2019/05/13/how-to-fix-an-expired-vcsa-machine-ssl-certificate-with-a-bugged-vmware-eam-service/

Posted by: cotnerfolearribled.blogspot.com