• Twitter

Scenario: Have you ever been in that situation where you want to remediate a host for some sort of baseline and you can’t due to a pending reboot on all of your hosts,  all hosts in a particular cluster, or just some hosts?  This situation can be particularly frustrating especially if you run TrendMicro DeepSecurity in your environment.

If this is the case for you then you know there are several tedious tasks that you must execute in order to do this properly:

  1. Shut down the TrendMicro DeepSecurity Virtual Appliance.
  2. Place the host into maintenance mode to evacuate all of the guests except for theDSVA.
    • The DSVA shouldn’t be moved for several reasons: DRS Rules to prevent one host being protected by multiple DSVAs and reduction of DSVA activation or even possible redeployment.
  3. Reboot the ESXi host and wait until it is connected to the vCenter Server.
  4. If you have MegaRaid, Offloaded DVS, or LACP issues you may have to rescan your HBA’s in order for your datastores to appear.
  5. Exit Maintenance Mode.
  6. Reload the DSVA if you suffer from step #4.
  7. Start the DSVA
  8. Do steps #1 to #7 over and over again until this process is complete.

What Now?: Let’s face it you are too busy and important to have to do this for one ESXi host much less 50 or 100!  Now that our steps are broken down let’s examine how to automate this daunting task so we do something else now that we have the proper logic.

Let’s Get To Work!: During my journey I had to do this per cluster hence our code without any cmd-lets in the script block looks like the following.  Note I sorted based on name for good measure.

Now that we are executing code that will begin the shutdown of the DSVA it is important that you update line #3 below with your DSVA nomenclature.  This is obviously an important step and if you miss it – well then – I guess you should have paid better attention.

Following shutting down the DSVA we will place the host into maintenance mode, and because our DSVA host is shutdown all VM’s except for our DSVA will be evacuated to some other host somewhere in our cluster.

Now for the part that is no fun to watch and can put you on edge if you aren’t paying attention to your clock — the VMHost reboot.  In this code block we do a couple of things to put our mind at rest during the Restart-VMHost:

  1. We wait for the host state to change from Maintenance to NotResponding. This is the logic that will let us know that the host has rebooted and not just that the reboot signal was sent.
  2. Once the state of the host has changed to NotResponding we will wait for the state of the host to change back again to Maintenance so we know that we are now connected to the VC.
  3. If you are like me you want to know how long it’s been so we place a poor man’s timer into both our Do While statements.  Once you have gone through a couple of host reboots you will know the rhythm of the reboots and may tend not to review the shell nearly as often.

If you are in spot where you know you will have to scan your HBA’s after the host is connected and before you leave maintenance mode.  If this part is hit/miss for you there is opportunity additional logic using the Count() function.  If your hosts are responsive and you never have to do this then omitting the next few lines are OK as well.

Next we quite simply exit maintenance mode.

Again, if your DSVA’s are stored on iSCSI HBA’s and your datastores are not present once the host has reconnected to the VC you will notice that your DSVA will be in an invalid state and must be reloaded.  The same will be true for any VM in this case that was registered to your host.

These next few lines of code are hand both inside this script block and for any time you need to restore the state of any VM when they appear to be inaccessible, and it sure beats having to execute the steps listed in VMware KB 2172 (http://goo.gl/ZKwLfK).

Obviously, if no VM’s are invalid nor inaccessible the code will keep trucking along.

Lastly, it’s time to fire our DSVA online and move on to the next host in our cluster.

 Closing Thoughts: Sometimes we have to do silly tasks to complete more important tasks, and sometimes those silly tasks are really silly because you have to work with application like TrendMicro DeepSecurity where the application logic is built on your hosts running 24/7/365.  All of this is OK because now we all know we can use PowerCLI to automate this headache and time waster of a task.


About the author

@VirtualizedCJ CJ is a virtualization professional that loves automation -- maybe a little to much.

Categories: Scripts, Trend Micro

One Response so far.

  1. […] some code that you may have already used and updating it to suit your needs can be found in the Reboot Automation script that we walked through […]

Leave a Reply to Upgrading ESXi: When is it worth it? : Virtual Jalapeno