Since our original deployment of Microsoft Operations Manager 2005 (now System Center Operations Manager 2007 R2), we have struggled with handling of alerting during maintenance windows for our servers. As many a SCOM admin can tell you, your management server can do a whole lot of squawking if you fail to set maintenance mode before a server reboot.
In the past, we configured notification blackout windows during scheduled Windows Update times. This worked, sort of… While we did not get paged during system reboots, we none-the-less had a lot of alerts that needed to be closed out every time a system rebooted. Why? Because suppressing notification does not suppress alerting. What we really needed to do was to put systems into maintenance mode before the Windows Update triggered reboot. This always seemed to difficult to implement, so we never did it. Scripting of Maintenance Mode requires that any account attempting to start MM have "admin" rights on the RMS. We did not want to go there with distributed scripts, and so we were at an impasse. The other big disadvantage to this approach was that it suppressed notifications for all systems in the infrastructure during scheduled maintenance windows, not just the ones that needed a reboot at that particular time. If there were no Windows Update-induced reboots that evening, too bad.
But now, thanks to Group Policy Client-Side Extensions, PowerShell, various improvements in SCOM 2007 R2 Maintenance Mode, and the fine work of blogger and SCOM developer Derek Harkin, we have a workable solution to this problem.
- GP Client-Side Extensions with item-level targeting (requires that CSE update be applied to any Server 2003 systems under management, and also use of a Vista-later system running a current Group Policy Management Console)
- Derek Harkin’s Maintenance Mode Management Pack for SCOM 2007 R2:
- Our additional VBScript "WUTriggerMaintMode.vbs" to create events triggered by Windows Updates
- "eventtriggers.exe" on Server 2003 (should be included with the OS)
- "schtasks.exe" on Server 2008 (definately included with the OS)
- Windows Scripting Host to run VBScripts on all managed targets (should be there by default…)
- PowerShell on the RMS (required to install SCOM 2007 R2 anyway…)
How it Works:
- Group Policy Preferences run every day to refresh the scripts and Scheduled Tasks on all managed systems. "MaintModeTrigger2008.xml", "WUTriggerMaintMode.vbs" and "MaintMode.vbs" are distributed to all systems, and tasks are created which perform the following task:
Watch the "SYSTEM" event log for event ID 22 from source "WindowsUpdateClient". When this event is seen, run the "MaintMode.vbs script with the arguments "ON 20M"
- The MaintMode.vbs script will create a WSH entry in the Application event log which will state that the system needs to enter Maintenance Mode for 20 minutes.
- A trigger set by the Maintenance Mode management pack will detect this event, and run a PowerShell script on the Root Management Server (RMS) that will put the desired system into Maintenance Mode.
- Configure your Windows Update policy to delay at five minutes after update application before initiating system reboot.
- Install the Maintenance Mode management pack following the included directions.
- Create a Group Policy object which is linked to your managed servers for the purpose of configuring Windows Update-triggered Maintenance Mode settings.
- Copy the MaintMode.vbs script and, our "WUTriggerMaintMode.vbs", and custom XML file to all managed Windows servers using Group Policy Preferences. I copied my scripts to “%SystemDrive%\local\scripts” on all managed system, but you could just plan to run the scripts off of a trusted, highly-available network share.
NOTE: If you will be copying the files from a network share to a local directory using GP Preferences, then you must grant “Domain Computers” read/execute/traverse rights to the parent directory that contains your files. If you will be executing the files directly from the file server, you need grant “Domain computers” rights to only the files themselves.
- Create a Scheduled Task preference targeted to Server 2003 and Server 2003 R2 systems. This task will use cscript.exe to run "WUTriggerMaintMode.vbs". I configured this task to run daily, but you could adjust the interval to suit your needs (weekly, monthly, on next reboot, run once, etc.)
- Create a Scheduled Task preference targeted to Server 2008 and Server 2008 R2 systems. This taks will run schtasks.exe to create a scheduled task from the "MaintModeTrigger2008.xml" task specification file. I used the command line:
schtasks.exe /Create /RU SYSTEM /TN "WUTriggerMaintMode" /XML [pathToFile]\MaintModeTrigger2008.xml /F
Note: I was unable to use the "Vista and later" version of the Scheduled Task preference, which is exposed when running GPMC on a Windows 7 client. I am unclear why this did not work, but just to be safe you probably will need to use the "legacy" Scheduled Task tool.
Again, you will need to set a schedule for this task that suits your environment.
- Force a Group Policy update on one of each OS type that you are managing to ensure that the GP Preferences are being distributed.
- Run the Scheduled Tasks that are created by the the policy to make sure that they work (at least on Server 2003, you can create synthetic events in the SYSTEM event log using "EVENTCREATE.exe". You can set of the event triggers using this tool.).
- Enjoy the silence during your next system maintenance.
the file which creates a Server 2008 scheduled task that will run the MaintMode.vbs script after Windows Update signals that the server will be rebooted (an Event ID 22).
2009-12-04T13:05:53.1637625 CAMPUS\admin-jgm PT30M true <QueryList><Query Id="0" Path="System"><Select Path="System">*[System[Provider[@Name='Microsoft-Windows-WindowsUpdateClient'] and EventID=22]]</Select></Query></QueryList> HighestAvailable S-1-5-18 true false IgnoreNew false true true false false true true false false false PT1H 7 cscript.exe %SystemDrive%\local\scripts\MaintMode.vbs ON 20M %systemroot%\system32
Uses "eventtriggers.exe" on Server 2003 systems to create an event trigger (and corresponding Scheduled Task) which will watch the system event log for Windows Update-triggered reboots (event ID 22) and run MaintMode.vbs when this happens. By the way, I know that this is crumby code… it could/shold be updated to provide better error handling and return codes.
' WUTriggerMaintMode.vbs - Trigger Maintenance Mode based on pending Windows ' Update-triggered reboot, as seen in the System event log Option Explicit Dim bSetTrig Dim fLog Dim oShell, oExec, oExec2, oFile Dim sLine, sSub, sSysDrive Set oShell = WScript.CreateObject("WScript.Shell") sSysDrive = oShell.ExpandEnvironmentStrings("%SystemDrive%") ' Create Log File Set oFile = CreateObject("Scripting.FileSystemObject") Set fLog = oFile.CreateTextFile(sSysDrive & "\local\scripts\WUTriggerMaintMo" _ & "de.log", true) ' Query for existing triggers: Set oExec = oShell.Exec("eventtriggers.exe /query") ' If Existing triggers previously created by this script are present, ' delete them: Do While Not oExec.StdOut.AtEndOfStream sLine = oExec.StdOut.ReadLine fLog.WriteLine(sLine) if InStr(sLine,"Windows Update - Reboot") then fLog.WriteLine("Existing trigger detected") sSub = Left(LTrim(sLine), 1) fLog.WriteLine("Trigger ID is: " & sSub) set oExec2 = oShell.Exec("eventtriggers.exe /delete /tid " & sSub) Do while oExec2.Status = 0 WScript.sleep 100 Loop fLog.WriteLine("Deletion of Event trigger attempted.") end if Loop ' Create latest event trigger: set oExec2 = oShell.Exec("eventtriggers.exe /create /tr ""Windows Update - R" _ & "eboot Impending"" /l SYSTEM /eid 22 /tk ""cscript.exe %SystemDrive%\l" _ & "ocal\scripts\MaintMode.vbs ON 20M"" /ru ""System"" ") Do while oExec2.Status = 0 WScript.sleep 100 Loop fLog.WriteLine("Creation of event trigger attempted.")