Hypervisor Monitoring

1. Overview
   1. What are ESX and CIM?
   2. Why VMware ESX or CIM?
   3. Important First Steps
   4. How it Works
   5. Scripts Used
   6. What is Being Monitored
   7. Understanding the Results
2. Setup
3. Scheduling
4. Document Revision History

Overview

Virtualization technology enables MSPs to partition one physical server into several virtual servers or machines.  Each virtual machine can interact independently with other devices, applications, data and users as though it were a separate physical resource. Different virtual machines can run different operating systems and multiple applications while sharing the resources of a single physical computer. Additionally, because each virtual machine is isolated from other virtualized machines, if one crashes, it does not affect the others.

Hypervisor software makes virtualization possible. This software, also known as a virtualization manager, sits between the hardware and the operating system, and decouples the operating system and applications from the hardware. The hypervisor assigns the amount of access that the operating systems and applications have with the processor and other hardware resources, such as memory and disk input/output.

Businesses use virtualization technologies to save money, energy, time and complexity. MSPs often utilize free 3rd party monitoring software to monitor the hypervisor (if at all) and rely on the manufacturer's management software to provisioning and operations. If RMM software is being utilized, it is on the virtual machine not the virtual host.

By creating a monitoring and management system within LabTech, LabTech is able to provide a consolidated, single interface management system for all supported hypervisors by utilizing plug-ins in the LabTech Control Center. Additionally, LabTech is able to utilize the existing monitoring interface, extended to alert on the virtual machines as well as the virtual hosts themselves.

Image may be NSFW. Clik here to view.

NOTE: Ignite™ is required in order to set up and use Hypervisor monitoring.

What are ESX and CIM?

ESX is a specific type of virtualization management. It refers to the monitoring, troubleshooting and configuration of virtual machines running virtualization software from VMware.

CIM is short for Common Information Model.  It allows for the interchange of management information between management systems and applications. For more detailed information on CIM and how VMware uses CIM:

• CIM FAQ by DMTF
• CIM Overview  by DMTF
• vSphere Client Hardware Health Monitoring by VMware

Why VMware ESX or CIM?

• Determines root cause analysis for poor VMware performance
• Determines root cause analysis for VMware failures
• Eliminates the need to set up alerts through vSphere Management Console

Important First Steps

There is a known issue within database views related to the following ESX/CIM internal monitors. Partner will see the following ESX monitors disabled, and will receive an error if you click the Build and View Query button on the monitor:

• ESX – Host Memory Usage
• ESX – VM CPU Usage per Core
• ESX – VM Memory Usage
• ESX – VM Memory Ballooned

In order to resolve the views please do the following:  

1. Open the ESX Views.sql file with notepad.exe. This file can be downloaded from the bottom of this documentation page.
2. Copy the entire contents.
3. Open the LabTech Control Center on the LabTech server and select Tools > MySQL Prompt.
4. Paste the contents of the .sql file into MySQL Prompt window.
5. Press [Enter] to execute the pasted query.
   Figure 1: Query Results
   Image may be NSFW.
   Clik here to view. 
6. You should see 'Query OK, 0 rows affected' as the result.
7. Type 'quit' and press [Enter] to exit MySQL Prompt.
8. You can now open these monitors and click the Build and View Query button and you will not see any errors.

How it Works

A script will run on an agent that is designated as the probe and scan the network devices table to determine if the network device has been designated as a Virtual Host.  Once a device has been found that has been designated as a virtual host, the script will determine what hypervisor is being used and its version. Then, the script will then call one of four main functions:

• GetHostHardware: This will populate the database with the found hardware devices for the host (e.g., network card, hard drive, memory, etc.).
• GetVMConfig: This will populate the database with the virtual machine configuration.
• GetHostCounters: This will populate the database with the performance counters for the host system. This information will be what is used to trigger alerts from the monitors. There are five types of performance counters which will be gathered: system, network, disk, memory and CPU.
• GetVMCounters: This will populate the database with the performance counters for the virtual machine. This information will be what is used to trigger alerts from the monitors. There are five types of performance counters which will be gathered: system, network, disk, memory and CPU.

There will be two sets of monitors available, host and VM. The host monitors will be used to check the health of the overall virtual environment for key items, such as disk latency, network over utilization, CPU and memory over-utilization which affect all virtual machines. The other monitor sets will investigate the individual virtual machine's health. The same counters for the host are queried, except these are VM specific. Industry accepted best practices and thresholds will be used for all monitors.

Scripts Used

There are 15 scripts used for virtual host collection. They are:

GetHostHardware: This script collects all the hardware information about a virtual host, like CPU cores, memory size, and PCI device information. This script MUST BE RUN BEFORE any of the other virtual host collection scripts, so that the hardware tables are populated with information that the other collection scripts rely on. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect the hardware information.

GetVMConfig: This script collects the hardware configuration for each virtual machine instance on a VMware host, like CPU cores, memory size, network hardware information, and storage disk information. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect the virtual machine hardware configuration information.

GetHostCounter: This script collects performance counters for ALL categories (CPU, Disk, Memory, Network, System) on the host. Since this script collects on all categories this should be run once to get an initial collection. The following host counter scripts should then be scheduled to collect on each performance category. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect all performance counters information.

GetHostCounterCPU: This script collects performance counters for CPU on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the CPU performance counters information.

GetHostCounterDisk: This script collects performance counters for Disk on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the Disk performance counters information.

GetHostCounterMemory: This script collects performance counters for Memory on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the Memory performance counters information.

GetHostCounterNetwork: This script collects performance counters for Network on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the Network performance counters information.

GetHostCounterSystem: This script collects performance counters for System on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the System performance counters information.

GetVMCounters: This script collects performance counters for ALL categories (CPU, Disk, Memory, Network, System) for each virtual machine on each host. Since this script collects on all categories this should be run once to get an initial collection. The following VM counter scripts should then be scheduled to collect on each performance category. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect all performance counters information for each virtual machine on the host.

GetVMCountersCPU: This script collects performance counters for CPU for each virtual machine on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the CPU performance counters information for each virtual machine on the host.

GetVMCountersDisk: This script collects performance counters for Disk for each virtual machine on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the Disk performance counters information for each virtual machine on the host.

GetVMCountersMemory: This script collects performance counters for Memory for each virtual machine on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the Memory performance counters information for each virtual machine on the host.

GetVMCountersNetwork: This script collects performance counters for Network for each virtual machine on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the Network performance counters information for each virtual machine on the host.

GetVMCountersSystem: This script collects performance counters for System for each virtual machine on the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date VMwarecollector.exe, and all dll's needed to run the collector executable, are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect only the System performance counters information for each virtual machine on the host.

CIM Collect: This script collects extended hardware information based on the classes listed in the database for the host. The script checks to ensure that the agent it is running on is designated as a probe. The script will then ensure that the up-to-date GetCIM.exe, and all dll's needed to run the executable are downloaded to the probe agent. The script then looks for network devices in the same location as the probe agent, to collect against, that have the 'VMware host' extra data field selected. The script will then loop through those network devices and collect the extended hardware information on the host, looping through all the classes.

What is Being Monitored

The following table displays each monitor with what is being monitored, the name of the script that is being used for collection and whether the monitor is hardware based or performance counter based.

Understanding the Results

The returned values are defined by the DMTF monitoring standard for HealthStates. LabTech ESX monitoring does not define these values, it is only extracting the CIM values and alerts on the results. Additionally, these values are hardware manufacturer independent. At this time, the monitor only returns the integer value and if the value does not equal 5 (OK) an alert will be generated. Refer to the following table for possible CIM HealthStates values:

Image may be NSFW. Clik here to view.

NOTE: Please be aware that these are not real time monitors. The data is gathered via the collection scripts and then the monitors need to run against the data. You can modify your script schedule; however, it is recommended that you do not schedule these scripts more frequently than once an hour as this could impact your system performance.

Table 2: Error Descriptions

Setup

Image may be NSFW. Clik here to view.

NOTE: If the monitors are not part of your base installation, you can download the monitor set from the Marketplace.  If you need download assistance, please refer to the Marketplace documentation.

Once downloaded (if necessary), each Location's Passwords tab will store the credentials to log in and collect the VMware performance counters and configuration. 

1. From the Control Center, expand Clients > Client and double-click on the appropriate Location.
2. Click on the Passwords tab.
3. Right-click on the window and select Add Entry.
   Figure 1: Passwords - Add Entry
   Image may be NSFW.
   Clik here to view.
4. Enter the following:
   • Title:  VMWare
   • User Name:  root
   • Password:  enter your VMware root password
   • Repeat:  Re-enter your VMware root password

Image may be NSFW. Clik here to view.

NOTE: The title MUST always be 'VMWare' to allow the collection scripts to populate the user name and password parameters correctly.

5. Click OK, and then Save.
6. Ensure that there is a probe enabled at the location with ESX hosts.
7. Once the probe has detected the host, expand Clients > Client > Location > Network and then double-click the device to open.
   Figure 2: Network Device
   Image may be NSFW.
   Clik here to view.
8. Select the vmware host checkbox in the Comments and Information section and click Save Additional Information.
9. Close the window.
10. Setup is complete.

Scheduling

• The scripts should be scheduled to be run on the _System Automation > Probe Machines with VMWare Credentials group.
• Get HostHardware should be scheduled to run once per day.
• Get HostCounter should be scheduled to run once per day.
• Each HostCounter sub-script for Disk, Network, etc. should be scheduled to run per your monitoring policy.
• GetVMConfig should be scheduled to run once per day.
• GetVMCounter should be scheduled to run once per day.
• Each VMCounter sub-script for Disk, Network, etc. should be scheduled to run per your monitoring policy.

Document Revision History

Image may be NSFW.
Clik here to view.