Category Archives: Virtual Servers

Tossing Out Unresponsive Guests

Ever have a guest you wish you could throw out of your house?  The unresponsive lump of flesh that sits in your favorite chair, eating your food, using up your soap, smelling up the facilities, while contributing nothing to the household economy?  I know I have, but unfortunately social decorum prevents me from acting on my base desire to toss the sponger out into the rain.

Fortunately, we are not bound my the same laws of etiquette in the IT world.  When your VM Guest cops and attitude with you, you can kill it.  Unfortunately, VMWare does not document all of the ways in which you can put an end to pestilential guests.  Fortunately, we have a Platinum support contract, and I was thus able to learn how to send a proper kill signal to a guest.  Here is the drill:

  • Verify that ESX “thinks” your guest is still running:
    • vmware-cmd "<pathToVmxFile>/<vmxfile>.vmx" getstate
    • Should return:
      getstate() = on
  • Scan the running VM Guest process names to find the VM “World ID” of the hung guest:
    • cat /proc/vmware/vm/*/names
    • Output returned should be similar to this:
      vmid=1105 pid=-1 cfgFile="/vmfs/volumes/46f13af6-27b2dd91-6873-00145e6d4c2c/hyperion11/hyperion11.vmx" uuid="50 26 41 af 30 dd cb 48-2d 9a ed 48 6e 6d db 6c" displayName="hyperion11"
    • Take note of the “vmid” above (in this case, “1105”.  This is the vmid of the “world”, or “cartel” of you guest OS.
  • Send a “kill signal 9” to the discovered world ID:
    • /usr/lib/vmware/bin/vmkload_app -k 9 <vmid>
    • Output should be similar to this:
      08 09:40:43.189: Sending signal '9' to world 1104.
  • Verify that the guest is not running:
    • vmware-cmd "<pathToVmxFile>/<vmxfile>.vmx" getstate
    • Returns:
      getstate() = off
  • You should then be able to start the guest normally.

VMWare "BootRun" Service hangs on ESX Guests

I have been getting intermittant errors on Server 2003 systems that we have deployed on our ESX servers:
“The Virtual BootRun Service failed to start”.

Apparently this service is called during deployment of a VM Guest from a template, and is not needed after deployment:
http://www.vmware.com/community/thread.jspa?threadID=6007&messageID=39465

Ass I had to do was run:

c:\WINDOWS\vmware_imc\bootrun -unregserver

Migrating Symantec AntiVirus management servers

Well, it has been a fun week of migrating our Symantec AntiVirus servers from old, dying Dell 5th-gen PowerEdge servers onto bleeding-edge ESX virtual machines. Here are some of the highlights:

Firewall changes:
In moving the servers, we had to assign new IP addresses in our protected 102.0 subnet. Thus, I had to research the firewall exceptions required for access to the servers. It seems the two required ports are:
TCP port 2967 (Inboud) – for Symantec AntiVirus service (RtVscan.exe), for AV definition push updates, and client monitoring
UDP port 38293 (Inbound) – for Intel PDS service (pds.exe), allows retrieval of AV policy settings
(initial rules were not correct, resulting in clients falling out of the mangement cycle)

LiveUpdate changes:
I have been wanting to change the address of our internal LiveUpdate server for awhile… we are now using http://liveupdate.uvm.edu as the primary distribution server, with http://norton1.uvm.edu, http://norton2.uvm.edu, and http://liveupdate.symantecliveupdate.com as backups. “liveupdate.uvm.edu” is a round-robin record that alternates between norton1 and norton2. We are considering a load balancing implementation instead, but this probably is unnecessary given the presence of “backup servers” in the liveupdate.hst file distributed to clients.
The only real problem here was that many of the file types in the LiveUpdate download directory were not of recognized “MIME Types” (i.e. they were not html, xml, zip, txt, audio/video, or MS Office files). I had to add the following extensions to the IIS configs before clients could successfully retrieve updates:
.x00, .ieg, .m25, .ia64ap, .x86, .lin
Once these MIME types were added and I had run an “iisreset”, LiveUpdate started to function normally.

Reporting Services
Migration of reporting services is a total PIA. I am trying to migrate the back-end database to an external SQL 2005 server from SQL 2000 in addition to re-installing the Reporting Services binaries on the new Norton2 server. Here are the steps taken so far:

  • detach the SymReport database from the old server, copy the files to the new server and attach
  • change ownership of the database back to its original setting
  • change the compatibility level of the database to “9.0” (SQL 2005)
  • install the new SQL native client on the SAV hosts
  • launch the Reporting services installer setup.exe. Note: do not launch from the autorun setup menu on the SAV CD! You must use the reporting services setup.exe or the advanced install options that we need will not be available.
  • supply the credentials necessary to connect to the new SQL 2005 DB. Also, specify alternative credentials for the db user, datasource name, and db name. Use the DB name that was imported into the SQL 2005 server, and get the username that was previously used from the DB security tabs.
  • After install, the reporting server should smoothly reconnect to the existing DB. You can check that this is happening in the SQL activity monitor pane.

Unfortunately, ran into some problems with the Reporting Agent on the primary SAV server (it is running a remote agent). The agent slowly hogs up all the memory on the box and is creating a CPU-bound condition (very bad news on an ESX host). I has no success trying to troubleshoot the situation, and I was not having fun… Using sysinternal tools I was able to watch the ReportingAgentLauncher thrash the heck out of some temp files that it was creating, but it never did anything with these files. I believe there must have been some bad configuration information being fed to the SAV server from the reporting database, and that this was creating a loop. So untimately I fixed the situation with the following “solution”:

  • Uninstall reporting services
  • Reinstall with a new database (thus abandoning old report data)

Voila… reporting services are running normall, we have our first production SQL 2005 database, and our second set of production ESX guests.

VMWare Virtual Infrastructure 3 and IBM Director

Here are some links and notes regarding configuring VI 3 for use with IBM Director 5.10:

Good IBM Redbook on VI 3 and Director 5.10:
http://www.redbooks.ibm.com/redbooks/SG247190/wwhelp/wwhimpl/java/html/wwhelp.htm

More official guide to using ESX 2.5 with Director:
ftp://ftp.software.ibm.com/pc/pccbbs/pc_servers_pdf/managingvmware.pdf

Several management methods are available for ESX hosts. The Redbook focuses on using the Director Agent for Linux. The apparent reason for this are:

  • Support for hardware alerting though the agent is becomming available with ESX3. This was not the case with ESX2.5, so the agent did not do much for you.
  • “Level 0” agents are managed though SSH. Under ESX 3, root login is not allowed via SSH, The linux agent relies on root access, so this agentless management method no longer works.
  • SNMP is the only other management method, and who wants to go there?