Migrating Symantec AntiVirus management servers

Well, it has been a fun week of migrating our Symantec AntiVirus servers from old, dying Dell 5th-gen PowerEdge servers onto bleeding-edge ESX virtual machines. Here are some of the highlights:

Firewall changes:
In moving the servers, we had to assign new IP addresses in our protected 102.0 subnet. Thus, I had to research the firewall exceptions required for access to the servers. It seems the two required ports are:
TCP port 2967 (Inboud) – for Symantec AntiVirus service (RtVscan.exe), for AV definition push updates, and client monitoring
UDP port 38293 (Inbound) – for Intel PDS service (pds.exe), allows retrieval of AV policy settings
(initial rules were not correct, resulting in clients falling out of the mangement cycle)

LiveUpdate changes:
I have been wanting to change the address of our internal LiveUpdate server for awhile… we are now using http://liveupdate.uvm.edu as the primary distribution server, with http://norton1.uvm.edu, http://norton2.uvm.edu, and http://liveupdate.symantecliveupdate.com as backups. “liveupdate.uvm.edu” is a round-robin record that alternates between norton1 and norton2. We are considering a load balancing implementation instead, but this probably is unnecessary given the presence of “backup servers” in the liveupdate.hst file distributed to clients.
The only real problem here was that many of the file types in the LiveUpdate download directory were not of recognized “MIME Types” (i.e. they were not html, xml, zip, txt, audio/video, or MS Office files). I had to add the following extensions to the IIS configs before clients could successfully retrieve updates:
.x00, .ieg, .m25, .ia64ap, .x86, .lin
Once these MIME types were added and I had run an “iisreset”, LiveUpdate started to function normally.

Reporting Services
Migration of reporting services is a total PIA. I am trying to migrate the back-end database to an external SQL 2005 server from SQL 2000 in addition to re-installing the Reporting Services binaries on the new Norton2 server. Here are the steps taken so far:

  • detach the SymReport database from the old server, copy the files to the new server and attach
  • change ownership of the database back to its original setting
  • change the compatibility level of the database to “9.0” (SQL 2005)
  • install the new SQL native client on the SAV hosts
  • launch the Reporting services installer setup.exe. Note: do not launch from the autorun setup menu on the SAV CD! You must use the reporting services setup.exe or the advanced install options that we need will not be available.
  • supply the credentials necessary to connect to the new SQL 2005 DB. Also, specify alternative credentials for the db user, datasource name, and db name. Use the DB name that was imported into the SQL 2005 server, and get the username that was previously used from the DB security tabs.
  • After install, the reporting server should smoothly reconnect to the existing DB. You can check that this is happening in the SQL activity monitor pane.

Unfortunately, ran into some problems with the Reporting Agent on the primary SAV server (it is running a remote agent). The agent slowly hogs up all the memory on the box and is creating a CPU-bound condition (very bad news on an ESX host). I has no success trying to troubleshoot the situation, and I was not having fun… Using sysinternal tools I was able to watch the ReportingAgentLauncher thrash the heck out of some temp files that it was creating, but it never did anything with these files. I believe there must have been some bad configuration information being fed to the SAV server from the reporting database, and that this was creating a loop. So untimately I fixed the situation with the following “solution”:

  • Uninstall reporting services
  • Reinstall with a new database (thus abandoning old report data)

Voila… reporting services are running normall, we have our first production SQL 2005 database, and our second set of production ESX guests.

SQL 2005 setup on 2k3EE R2 cluster

Here are a few notes from my initial setup of SQL Server 2005, Standard Edition on our new IBM LS20 blades.

Our blades are equiped with 2x dual core AMD Opteron processors, two Qlogic FC HBAs, two Broadcom 57xx Ethernet ports, and 8Gb of RAM. We thusly are using Server 2003 64-bit edition for the install.

Server install and setup:

  • Installed OS from the BladeCenter Management Module, mounting Server 2003 ISO file from a neighboring blade’s web browser. It is slow and ugly, but works.
  • On first boot, we have no functioning network or FC HBA. I downloaded the latest Broadcom application ISO from IBM, mounted via managment module, and ran the installer. Also ran the “Broadcom Advanced Services” installer, then configured an Active/Active NIC team with VLANs for public networking (720) and the cluster heartbeat (4000).
  • On the heartbeat network, we disable File and Print services, as well as Client for Microsoft Networks. Be sure to disable NetBIOS over TCP as well. LAN speed duplex does not appear to be configurable with this driver, but the switch ports have been set to negotiate for 1Gbps speed only, and PortFast is on to reduce port blocking time on initial NIC connect.
  • Once networking is functional, I am able to transfer Qlogic HBA drivers to the server, and install them in the Device Manager control panel.
  • We also transfer the DS4000 Storage Manager for Windows installer to the server (obtained from IBM.com), and then perform a client install of the software (this installs the RDAC multi-path I/O driver onto the server).
  • Configure LUNs on the DS4800 storage server, configure LUN masking, and configure Zoning on the Brocade switches. SAN volumes are now available on the server.

Cluster Configuration:

  • Setup Microsoft Distributed Transaction Coordinator as a cluster resource as well. Strictly follow instructions in MS KB301600. Created 1Gb lun for this task – drive letter “T”
  • Setup a cluster group for MSSQL server, add physical disk resources for MDF and LDF files (database and transaction logs) (Drives M: and L:)

SQL Setup:

  • Ensure that you are logged on to only one node of the cluster – setup will fail if there are active RDP sessions on both cluster nodes.
  • From http://support.microsoft.com/?kbid=916760, create “Servers” and “Tools” directories for CD1 and CD2 contents, respectively.
  • Run SQL Setup from the primary node in the cluster, source files extracted to our primary file server using the base path \\\software\sql2005server.
  • Start with an installation of Workstation Components only… need to run this install on each node separately
  • Run SP1 patch on both nodes to update SQL Setup Support files.
  • Install all remaining SQL components (Database, Analysis services, Reporting, Notification, Integration). Select option to create failover cluster for Database and Analysis services, select drive M: as the destination for data files.
  • Configure service accounts for every service
  • Set server to “mixed” authentication mode for support of some of our apps
  • Keep default collation settings (this uses SQL Collations for the Database service and Windows Collations (Latin_CS_AS) for Analysis Services).
  • NOTE:Reporting Services will have to be configured after SQL install
  • enabled automatic error reporting but NOT usage statistics.

Next Steps:

  • http://msdn2.microsoft.com/en-us/library/ms171338.aspx – Review use of clustering with Notification Services
  • Review Reporting Services deployment options:
    • Separate Reporting Services Instance for each application that uses it?
    • Central Reporting Services web cluster? (Would require 2x additional Windows 2003 EE installs!)

VMWare Virtual Infrastructure 3 and IBM Director

Here are some links and notes regarding configuring VI 3 for use with IBM Director 5.10:

Good IBM Redbook on VI 3 and Director 5.10:

More official guide to using ESX 2.5 with Director:

Several management methods are available for ESX hosts. The Redbook focuses on using the Director Agent for Linux. The apparent reason for this are:

  • Support for hardware alerting though the agent is becomming available with ESX3. This was not the case with ESX2.5, so the agent did not do much for you.
  • “Level 0” agents are managed though SSH. Under ESX 3, root login is not allowed via SSH, The linux agent relies on root access, so this agentless management method no longer works.
  • SNMP is the only other management method, and who wants to go there?

WDS – booting from UFD (USB Flash Drive)

Booting to WDS can be a bit pokey over lower-bandwidth connections. Even over GbE, WDS boot images take significantly longer to load than the original RIS menu system. Bummer!

I expedite loading of the “WDS Discover” boot image (this is the image that allows loading of WDS server images) I decided to try creating a bootable “WDS Discover” image on UFD. The Vista Beta 2 BDD (and included AIK) include instructions on generating WinPE-bootable UFDs. However, there are two problems:

  • The instructions do not work… the resuling UFD is not bootable!
  • The image generated results only in a standard WinPE environment, not a WDS deployment GUI

Here are the missing bits fo informaiton:
When formatting a UFD for WinPE, you must set the formatted partition as “active”. To do this, plug your UFD into a Vista OS:
select disk 1
select partition 1
format fs=ntfs override quick

Failing to make the UFD partition “active” will result in an unbootable UFD. Note that the Vista build 5472 AIK includes these updated instructions.

In order to make a UFD-bootable WDS Discover image, you need to use the “WDSUtil” CLI (with /new-discoverimage or /new-captureimage options) or WDS MMC GUI.
Here are the steps:

  1. Use ImageX to extract the Windows Vista SETUP image from the boot.wim that ships with the Vista DVD (e.g. “imagex /apply [path to boot.wim>] 2 [path to apply image]”)
  2. Apply custom drivers (e.g. “peimg /inf: [path to driver] [path to applied image “windows” directory]”)
  3. Capture the custom image (e.g. “imagex /boot /compress maximum /capture [path to applied image] [wim file name] [image name]”)
  4. Import the image into WDS using the GUI or CLI
  5. Use WDSUtil with /new-CaptureImage or /new-DiscoverImage and /Image:[image Name] to create a new capture or discovery image based on the custom WIM generated above.

You can now write your image to CD (using “oscdimg”) or to UFD (using the BDD/AIK instructions).

NOTE! If you expect to be able to capture using the WDS Capture image, you must first run “sysprep” on the OS in question. The WDS Image Capture Wizard will not work without sysprep!

Resetting corrupted performance counters

Performance counters on SYSIMG1 became corrupted recently. The MS KB is full of strange and dated advice on the subject. Here is the most recent article I could find:

It is full of tedious and laborious steps to manually repair the performance counter object mappings. Towards the bottom of the article, we learn that on Server 2003, the following console command will restore corrupted counters, thus obviating the need for any manual config steps:

This will re-read all of the perfXXX.ini files in the system32 directory and rebuild the counter data. If you have any apps running that do not house their .ini files in “system32” you will need to load those manually.

In any event, the above command fixed the problem on SYSIMG1 quite nicely. Thanks, MS!

RIS – adding "text mode" storage drivers

We had some fun adding Intel SATA drivers for the new Intel chipsets onto our RIS server. These drivers are required in order to deploy RIPrep-generated images onto the new Gateway M285 Tablet PCs.

Here were the key steps, as outlined in this TechNet Article:

  • In the “flat” image for the platform to be imaged (in our case, the flat “Tablet PC 2005” image) create a $OEM$ directory with a “textmode” subdirectory.
  • Copy the SATA storage driver into the textmode folder. Make sure that the entry in the [disk] section of the TXTSETUP.OEM file in this directory does not contain a “relative path” on the end of entry. (i.e. the line should end with a simple “\”).
  • Copy the exact text of the driver name from the [scsi] section into the [MassStorageDrivers] section of the Flat image’s i386\templates\ntstndrd.sif file. Also add the following lines:
    #Added this
    "Intel(R) 82801GBM SATA AHCI Controller (Mobile  ICH7M)"="OEM"
  • Also in the [unattended] section, set the OemPreinstall option to “yes”

Some interesting things that we discovered in making this work:

  1. PNP NIC drivers that are accessed during the initial NDIS driver load can be added to ANY flat image on the server. A Tablet PC will access drivers from the “XP SP2” Flat image. If you have more than one version of the same driver in different folders, only the first alphabetically listed driver will be made avaialble to RIS net boot clients.
  2. Text-mode setup drivers (which generally are mass storage drivers) are loaded when setup.exe starts in earnest, after the NIC load process. These drivers must be available in the $OEM$ directory of the platform for which you are loading an image (RIPrep or Flat). If you are loading an RIPrep image for a Tablet PC, you must have appropriate drivers available in the Tablet PC flat image.

WDS testing – firewall requirements

I have installed an instance of WDS on our current production RIS server. No worries, RIS functionality is still present, fully unadulterated.

However, I seem to be having problems getting a PXE boot client in my office to load. PXE boot works, and the WIM image gets loaded onto the workstation. However, I get a “unable to communicate with the WDS server” error after the GUI loads.

Some Ethereal packet captures show that the client is sending a port map request to the server, and the server is telling the client to connect to port “5040”. This value is not in our range of pre-allocated RPC ports, so I assumed that this port is hard-coded into the WDS service.

Sure enough, a quick registry search reveals that this is a parameter of the WDSServer service:
The default value was “5040”. I changed it to be within our range of excepted RPC ports, then ran:
net stop WDSServer
net start WDSServer

Lo and behold, I can now boot to WDS and install Vista. Cool. Now they just need to get the bugs out and give us the RTM WIM files. We will be ready!

Note: Net booting the install WIM is kinda pokey. I think I will try making a bootable USB drive next. I think this might be our best option for mass distribution.