Posts Tagged ‘OpsMgr’

Improving Notifications in System Center Operations Manager 2012

Anyone who depends on System Center Operations Manager 2012 (or any earlier version of SCOM, back to MOM) likely has noticed that notifications are a bit of a weak spot in the product.

To address this, we have use the “command channel” to improve the quality of messages coming out of SCOM.  Building on the backs of giants, we implemented a script that takes an AlertID from SCOM, and generated nicely formatted email and alpha-numeric pager messages with relevant alert details.

More recently, we have identified the need to generate follow-up notifications when an initial alert does not get addressed.  I went back to our original script, and updated it to use a new, custom Alert ResolutionState (“Notified”), and I have added logic to update the Alert CustomField1 and CustomField2 with data that is useful in determining whether or not an alert should get a new notification, and how many times follow-up notifications have been sent.

Heart-felt appreciation goes out to Tao Yang for his awesome work on his “SCOMEnhancedEmailNotification.ps1″ script, which served as the core for my work here.

Here is my version… I don’t have a lot of time to explain it, but hopefully the comments give you enough to go on. Apologies for the rather bad munging of quotation marks… wordpress hates me this month. If you want to use this code, search for ampersand-quot-semicolon, replace with actual quotation marks.

#=====================================================================================================
# AUTHOR:	J. Greg Mackinnon, Adapted from 1.1 release by Tao Yang 
# DATE:		2013-05-21
# Name:		SCOMEnhancedEmailNotification.PS1
# Version:	3.0
# COMMENT:	SCOM Enhanced Email notification which includes detailed alert information
# Update:	2.0 - 2012-06-30	- Major revision for compatibility with SCOM 2012
#								- Cmdlets updated to use 2012 names
#								- "Notified" Resolution Status logic removed
#								- Snapin Loading and PSDrive Mappings removed (replaced with Module load)
#								- HTML Email reformatted for readability
#								- Added '-format' parameter to allow for alphanumeric pager support
#								- Added '-diag' boolean parameter to create options AlertID-based diagnostic logs
# Update:   2.2 - 2013-05-16    - Added logic to update "CustomField1" alert data to reflect that notification has been sent for new alerts.
#								- Added logic to update "CustomField2" alert data to reflect the repeat count for new alert notification sends.
#								- Added support for specifying alerts with resolution state "acknowledged"
#                               - Did some minor adjustments to improve execution time and reduce memory overhead.
# Update:	3.0 - 2013-05-20	- Updated to reduce volume of PowerShell instance spawned by SCOM.  Added "mailTo" and "pageTo" paramerters to allow sending of both short
#                                         and long messages from a single script instance.
#								- Converted portions of script to subroutine-like functions to allow repetition (buildHeaders, buildPage, buildMail)
#								- Restored "Notified" resolution state logic.
#								- Renamed several variables for my own sanity.
#								- Added article lookup updates from Tao Yang 2.0 script.
# Usage:	.\SCOMEnhancedEmailNotification.ps1 -alertID xxxxx -mailTo @('John Doe;jdoe@mail.com','Richard Roe;rroe@provider.net') -pageTo @('Team Pager;teampage@page.provider.com')
#=====================================================================================================
#In OpsMgr 2012, the AlertID parameter passed in is '$Data/Context/DataItem/AlertId$' (single quote)
#Quotation marks are required otherwise the AlertID parameter will not be treated as a string.
param(
	[string]$alertID = $(throw 'A valid, quote-delimited, SCOM AlertID must be provided for -AlertID.'),
	[string[]]$mailto,
	[string[]]$pageto,
	[switch]$diag
)
Set-PSDebug -Strict

#### Setup Error Handling: ####
$error.clear()
#$erroractionpreference = "SilentlyContinue"
$erroractionpreference = "Inquire"

#### Setup local option variables: ####
## Logging: 
#Remove '$alertID' from the following two log file names to prevent the drive from filling up with diag logs:
$errorLogFile = 'C:\local\logs\SCOMNotifyErr-' + $alertID + '.log'
$diagLogFile = 'C:\local\logs\SCOMNotifyDiag-' + $alertID + '.log'
#$errorLogFile = 'C:\local\logs\SCOMNotifyErr.log'
#$diagLogFile = 'C:\local\logs\SCOMNotifyDiag.log'
## Mail: 
$SMTPHost = "smtp.uvm.edu"
$SMTPPort = 25
$Sender = New-Object System.Net.Mail.MailAddress("OpsMgr@lifeboat.campus.ad.uvm.edu", "Lifeboat OpsMgr Notification")
#If error occured while excuting the script, the recipient for error notification email.
$ErrRecipient = New-Object System.Net.Mail.MailAddress("saa-ad@uvm.edu", "SAA Windows Administration Team")
##Set Culture Info (for knowledgebase article language selection):
$cultureInfo = [System.Globalization.CultureInfo]'en-US'
##Get the FQDN of the local computer (where the script is run)...
$RMS = $env:computername

#### Initialize Global Variables and Objects: ####
## Mail Message Object:
[string] $threadID = ''
$SMTPClient = New-Object System.Net.Mail.smtpClient
$SMTPClient.host = $SMTPHost
$SMTPClient.port = $SMTPPort
##Load SCOM PS Module
if ((get-module | ? {$_.name -eq 'OperationsManager'}) -eq $null) {
	Import-Module OperationsManager -ErrorAction SilentlyContinue -ErrorVariable Err | Out-Null
}
## Management Group Object:
$mg = get-SCOMManagementGroup
##Get Web Console URL
$WebConsoleBaseURL = (get-scomwebaddresssetting | Select-Object -Property WebConsoleUrl).webconsoleurl
#### End Initialize ####


#### Begin Parse Input Parameters: ####
##Get recipients names and email addresses from "-to" array parameter: ##
if ((!$mailTo) -and (!$pageTo)) {
	write-host "An array of name/email address pairs must be provided in either the -mailTo or -pageTo parameter, in the format `@(`'me;my@mail.com`',`'you;you@mail.net`')"
	exit
}
$mailRecips = @()
Foreach ($item in $mailTo) {
	$to = New-Object psobject
	$name = ($item.split(";"))[0]
	$email = ($item.split(";"))[1]
	Add-Member -InputObject $to -MemberType NoteProperty -Name Name -Value $name
	Add-Member -InputObject $to -MemberType NoteProperty -Name Email -Value $email
	$mailRecips += $to
	Remove-Variable to
	Remove-Variable name
	Remove-Variable email
}
$pageRecips = @()
Foreach ($item in $pageTo) {
	$to = New-Object psobject
	$name = ($item.split(";"))[0]
	$email = ($item.split(";"))[1]
	Add-Member -InputObject $to -MemberType NoteProperty -Name Name -Value $name
	Add-Member -InputObject $to -MemberType NoteProperty -Name Email -Value $email
	$pageRecips += $to
	Remove-Variable to
	Remove-Variable name
	Remove-Variable email
}
if ($diag -eq $true) {
	[string] $("mailRecipients:") | Out-File $diagLogFile -Append 
	$mailRecips | Out-File $diagLogFile -Append
	[string] $("pageRecipients:") | Out-File $diagLogFile -Append 
	$pageRecips | Out-File $diagLogFile -Append
}
## Parse "-AlertID" input parameter: ##
$alertID = $alertID.toString()
#remove "{" and "}" around the $alertID if exist
if ($alertID.substring(0,1) -match "{") {
	$alertID = $alertID.substring(1, ( $alertID.length -1 ))
}
if ($alertID.substring(($alertID.length -1), 1) -match "}") {
	$alertID = $alertID.substring(0, ( $alertID.length -1 ))
}
#### End Parse input parameters ####


#### Function Library: ####
function getResStateName($resStateNumber){
	[string] $resStateName = $(get-ScomAlertResolutionState -resolutionStateCode $resStateNumber).name
	$resStateName
}
function setResStateColor($resStateNumber) {
	switch($resStateNumber){
		"0" { $sevColor = "FF0000" }	#Color is Red
		"1" { $sevColor = "FF0000" }	#Color is Red
		"255" { $sevColor = "3300CC" }	#Color is Blue
		default { $sevColor = "FFF00" }	#Color is Yellow
	}
	$sevColor
}
function stripCruft($cruft) {
	#Removes "cruft" data from messages. 
	#Intended to make subject lines and alphanumeric pages easier to read
	$cruft = $cruft.replace("®","")
	$cruft = $cruft.replace("(R)","")
	$cruft = $cruft.replace("Microsoftr ","")
	$cruft = $cruft.replace("Microsoft ","")
	$cruft = $cruft.replace("Microsoft.","")
	$cruft = $cruft.replace("Windows ","")
	$cruft = $cruft.replace(" without Hyper-V","")
	$cruft = $cruft.replace("Serverr","Server")
	$cruft = $cruft.replace(" Standard","")
	$cruft = $cruft.replace(" Enterprise","")
	$cruft = $cruft.replace(" Edition","")
	$cruft = $cruft.replace(".campus","")
	$cruft = $cruft.replace(".CAMPUS","")	
	$cruft = $cruft.replace(".ad.uvm.edu","")
	$cruft = $cruft.replace(".AD.UVM.EDU","")
	$cruft = $cruft.trim()
	return $cruft
}
function fnMamlToHTML($MAMLText){
	$HTMLText = "";
	$HTMLText = $MAMLText -replace ('xmlns:maml="http://schemas.microsoft.com/maml/2004/10"');
	$HTMLText = $HTMLText -replace ("maml:para", "p");
	$HTMLText = $HTMLText -replace ("maml:");
	$HTMLText = $HTMLText -replace (&quot;</section>&quot;);
	$HTMLText = $HTMLText -replace (&quot;<section>&quot;);
	$HTMLText = $HTMLText -replace (&quot;<section>&quot;);
	$HTMLText = $HTMLText -replace (&quot;<title>&quot;, &quot;<h3>&quot;);
	$HTMLText = $HTMLText -replace (&quot;</title>&quot;, &quot;</h3>&quot;);
	$HTMLText = $HTMLText -replace (&quot;&quot;, &quot;<li>&quot;);
	$HTMLText = $HTMLText -replace (&quot;&quot;, &quot;</li>&quot;);
	$HTMLText;
}
function fnTrimHTML($HTMLText){
	$TrimedText = &quot;&quot;;
	$TrimedText = $HTMLText -replace (&quot;&lt;&quot;, &quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;<h1>&quot;, &quot;<h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;</h1>&quot;, &quot;</h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;<h2>&quot;, &quot;<h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;</h2>&quot;, &quot;</h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;<H1>&quot;, &quot;<h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;</H1>&quot;, &quot;</h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;<H2>&quot;, &quot;<h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;</H2>&quot;, &quot;</h3>&quot;)
	$TrimedText;
}
function buildEmail {
	## Format the message for full-HTML email
	[string] $escTxt = &quot;&quot;
	if ($resState -eq '1') {$escTxt = '- Repeat Count ' + $escLev.ToString()}
	[string] $script:mailSubj = &quot;SCOM - $resStateName $escTxt - $alertSev | $moPath | $alertName&quot;
	$mailSubj = stripCruft($mailSubj)
	[string] $script:mailErrSubj = &quot;Error emailing SCOM Notification for Alert ID $alertID&quot;
	[string] $webConsoleURL = $WebConsoleBaseURL+&quot;?DisplayMode=Pivot&amp;AlertID=%7b$alertID%7d&quot;
	[string] $psCmd = &quot;Get-SCOMAlert -Id `&quot;$alertID`&quot; | format-list *&quot;
	# Format the Mail Message Body (do not indent this block!)
	$script:MailMessage.isBodyHtml = $true
	$script:mailBody = @&quot;



<p><b>Alert Resolution State:<Font color='$sevColor'> $resStateName </Font></b><br />
<b>Alert Severity:<Font color='$sevColor'> $alertSev</Font></b><br />
<b>Object Source (Display Name):</b> $moSource <br />
<b>Object Path:</b> $moPath <br />
</p>
<p>
<p><b>Alert Name:</b> $alertName <br />
<b>Alert Description:</b> <br />
$alertDesc <br>
&quot;@
	if (($resState -eq 0) -or ($resState -eq 1)) {
		if ($isMonitorAlert -eq $true) {
$script:mailBody = $mailBody + @&quot;
<b>Alert Monitor Name:</b> $MonitorName <br />
<b>Alert Monitor Description:</b> $MonitorDescription
</p>
&quot;@
		}elseif ($isMonitorAlert -eq $false) {
			$script:mailBody = $mailBody + @&quot;
<b>Alert Rule Name:</b> $RuleName <br />
<b>Alert Rule Description:</b> $RuleDescription <br />
&quot;@
		}
	}
$script:mailBody = $mailBody + @&quot;
<b>Alert Context Properties:</b><br /> 
$alertCX <br />
<b>Time Raised:</b> $timeRaised <br />
<b>Alert ID:</b> $alertID <br />
<b>Notification Status:</b> $($alert.CustomField1) </br>
<b>Notification Repeat Count:</b> $($escLev.ToString()) </p>
<p>
<b>PowerShell Alert Retrieval:</b> $psCmd <br />
<b>Web Console Link:</b> <a href="&quot;$webConsoleURL&quot;">$webConsoleURL</a> </p>
&quot;@
	if (($resState -eq 0) -or ($resState -eq 1)) {
		foreach ($article in $arrArticles) {
		$articleContent = $article.content
$script:mailBody = $mailBody + @&quot;
<p>
<b>Knowledge Article / Company Knowledge `-$($article.Language):</b>
<hr>
<p> $articleContent
<hr>
<p>

&quot;@
		}
	}
$script:mailErrBody = @&quot;

<p>Error occurred when excuting script located at $RMS for alert ID $alertID.
<p>
<p>Alert Resolution State: $resStateName
<p>
<p>$error
<p>
<p><b>**Use below command to view the full details of this alert in SCOM Powershell console:</b>
<p>$psCmd
<p>
<p> SCOM link:<a href="&quot;$webConsoleURL&quot;"> $webConsoleURL </a>
 

&quot;@ 
}
function buildPage {
	## Format the message for primitive alpha-numeric pager
	$script:moPath = stripCruft($moPath)
	[string] $escTxt = ''
	if ($resState -eq '1') {$escTxt = '- Rep Count ' +$escLev.ToString()}
	[string] $script:mailSubj = &quot;SCOM - $resStateName $escTxt | $moPath&quot;
	[string] $script:mailErrSubj = &quot;Error emailing SCOM Notification for Alert ID $alertID&quot;
	#UFT8 makes the message body look like trash.  Use ASCII (the default) instead.
	#$mailMessage.BodyEncoding =  [System.Text.Encoding]::UTF8 
	$script:MailMessage.isBodyHtml = $false
	$script:moSource = stripCruft($moSource)
	$script:alertName = stripCruft($alertName)
	$script:mailBody = &quot;| $moSource | $alertName | $timeRaised&quot; 
	$script:mailBody = stripCruft($mailBody)
}
function buildHeaders {
	param(
		[array]$recips
	)
	## Complete the MailMessage object:
	$script:MailMessage.Sender = $Sender
	$script:MailMessage.From = $Sender
	$script:MailMessage.Headers.Add('references',$threadID)
	# Regular (non-error) format
	if ($error.count -eq &quot;0&quot;) { 				
		$script:MailMessage.Subject = $mailSubj
		Foreach ($item in $recips) {
			$to = New-Object System.Net.Mail.MailAddress($item.email, $item.name)
			$script:MailMessage.To.add($to)
			Remove-Variable to
		}
		$script:MailMessage.Body = $mailBody
	} 
	# Error format:
	else {									
		$script:MailMessage.Subject = $mailErrSubj
		$script:MailMessage.To.add($ErrRecipient)
		$script:MailMessage.Body = $mailErrBody
	}
	## Log the message if in diag mode:
	if ($diag -eq $true) {
		[string] $('Mail Message Object Content:') | Out-File $diagLogFile -Append
		$mailMessage | fl * | Out-File $diagLogFile -Append
	}
}
#### End Function Library ####


#### Clean up existing logs: ####
if (Test-Path $errorLogFile) {Remove-Item $errorLogFile -Force}
if (Test-Path $diagLogFile) {Remove-Item $diagLogFile -Force}
if ($diag -eq $true) {
	[string] $(&quot;AlertID : `t&quot; + $alertID) | Out-File $diagLogFile -Append
	[string] $(&quot;MailTo      : `t&quot; + $mailto) | Out-File $diagLogFile -Append
	[string] $(&quot;PageTo      : `t&quot; + $pageto) | Out-File $diagLogFile -Append
	#[string] $(&quot;Format  : `t&quot; + $format) | Out-File $diagLogFile -Append
}



#### Begin Alert Handling: ####
## Locate the specific alert:
$alert = Get-SCOMAlert -Id $alertID
if ($diag -eq $true) {
	[string] $('SCOM Alert Object Content:') | Out-File $diagLogFile -Append
	$alert | fl | Out-File $diagLogFile -Append
}
## Read Alert Informaiton:
[string] $alertName = $alert.Name
[string] $alertDesc = $alert.Description
#[string] $alertPN = $alert.principalName
[string] $moSource = $alert.monitoringObjectDisplayName 	# Display name is &quot;Path&quot; in OpsMgr Console.
[string] $moId = $alert.monitoringObjectID.tostring()
#[string] $moName = $alert.MonitoringObjectName 			# Formerly &quot;strAgentName&quot;
[string] $moPath = $alert.MonitoringObjectPath 				# Formerly &quot;pathName
#[string] $moFullName = $alert.MonitoringObjectFullName 	# Formerly &quot;alertFullName&quot;
[string] $ruleID = $alert.MonitoringRuleId.Tostring()
[string] $resState = ($alert.resolutionstate).ToString()
[string] $resStateName = getResStateName $resState
[string] $alertSev = $alert.Severity.ToString() 			# Formerly &quot;severity&quot;
if ($alertSev.ToLower() -match &quot;error&quot;) {
	$alertSev = &quot;Critical&quot; 									# Rename Severity to &quot;Critical&quot;
}
[string] $sevColor = setResStateColor $resState				# Assign color to alert severity
#$problemID = $alert.ProblemId
$alertCx = $(1($alert.Context)).DataItem.Property `
	| Select-Object -Property Name,'#text' `
	| ConvertTo-Html -Fragment								# Alert Context property data, in HTML
$localTimeRaised = ($alert.timeraised).tolocaltime()
[string] $timeRaised = get-date $localTimeRaised -Format &quot;MMM d, h:mm tt&quot;
[bool] $isMonitorAlert = $alert.IsMonitorAlert
$escLev = 1
if ($alert.CustomField2) {
	[int] $escLev = $alert.CustomField2
}
## Lookup available Knowledge articles, if new alert:
if (($resState -eq 0) -or ($resState -eq 1)) {
	$articles = $mg.Knowledge.GetKnowledgeArticles($ruleId)
	
	if (!$error) {	#no point retrieving the monitoring rule when there's error processing the alert
		#if failed to get knowledge article, remove the error from $error because not every rule and monitor will have knowledge articles.
		if ($isMonitorAlert -eq $false) {
			$rule = Get-SCOMRule -Id $ruleID		
			$ruleName = $rule.DisplayName
			$ruleDescription = $rule.Description
			if ($RuleDescription.Length -lt 1) {$RuleDescription = &quot;None&quot;}
		} elseif ($isMonitorAlert) {
			$monitor = Get-SCOMMonitor -Id $ruleID
			$monitorName = $monitor.DisplayName
			$monitorDescription = $monitor.Description
			if ($monitorDescription.Length -lt 1) {$monitorDescription = &quot;None&quot;}
		}
		#Convert Knowledge articles
		$arrArticles = @()
		Foreach ($article in $articles) {
			If ($article.Visible) {
				$LanguageCode = $article.LanguageCode
				#Retrieve and format article content
				$MamlText = $null
				$HtmlText = $null
				if ($article.MamlContent -ne $null) {
					$MamlText = $article.MamlContent
					$articleContent = fnMamlToHtml($MamlText)
				}
					
				if ($article.HtmlContent -ne $null) {
					$HtmlText = $article.HtmlContent
					$articleContent = fnTrimHTML($HtmlText)
				}
				$objArticle = New-Object psobject
				Add-Member -InputObject $objArticle -MemberType NoteProperty -Name Content -Value $articleContent
				Add-Member -InputObject $objArticle -MemberType NoteProperty -Name Language -Value $LanguageCode
				$arrArticles += $objArticle
				Remove-Variable LanguageCode, articleContent
			}
		}	
	}
	if ($Articles -eq $null) {
		$articleContent = &quot;No resolutions were found for this alert.&quot;
	}
}
## End Knowledge Article Lookup
#### End Alert Handling ####



#### Begin Mail Processes:
if ($mailto) {
	# For all alerts, send full HTML email:
	$MailMessage = New-Object System.Net.Mail.MailMessage
	buildEmail
	buildHeaders -recips $mailRecips
	invoke-command -ScriptBlock {$SMTPClient.Send($MailMessage)} -errorVariable smtpRet
}
if ($pageTo) {
	# For page-worthy alerts, format short message and send:
	$MailMessage = New-Object System.Net.Mail.MailMessage
	buildPage
	buildHeaders -recips $pageRecips
	invoke-command -ScriptBlock {$SMTPClient.Send($MailMessage)} -errorVariable smtpRet
}
#### End Mail Message Formatting #### 


# Populate CustomField1 and 2 to indicate that a notification has been sent, with repeat count.
if (!$smtpRet) { 							# IF the message was sent (apparently)...
	[string] $updateReason = &quot;Updated by Email notification script.&quot;
	[string] $custVal1 = &quot;notified&quot;
	if ($resState -eq &quot;0&quot;) { 				# . AND IF this is a &quot;new&quot; alert...
		$alert.ResolutionState = 1			# ..Set the resolution state to &quot;Notified&quot;
		$alert.CustomField2 = $escLev		# ..Set CustomField2 to the current notification retry count (presumably 1)
		if (!$alert.CustomField1) {			# ..AND if CustomField1 is not already defined...
			$alert.CustomField1 = $custVal1	# ... Set CustomField1.
		}
		$alert.Update($updateReason)
	} 
	elseif ($resState -eq &quot;1&quot;) {		# .Or,If this is a &quot;notified&quot; alert
		if ($alert.CustomField2) {		# ..and the notification retry count exists..
			$escLev += 1				# ...Increment by one.
		}
		$alert.CustomField2 = $escLev
		$alert.Update($updateReason)
	}
}



Write-Host $error
##Make sure the script is closed
if ($error.count -ne &quot;0&quot;) {
	[string]$('AlertID string: ' + $alertID) | Out-File $errorLogFile
	[string]$('Alert Object Content: ') | Out-File $errorLogFile
	$alert | Format-List * | Out-File $errorLogFile
	[string]$('Error Object contents:') | Out-File $errorLogFile
	$Error | Out-File $errorLogFile
}
#Remove-Variable alert
#Remove-Module OperationsManager

Operations Manager – Discovered Entity Cleanup

We have been tracking a problem with some of our Operations Manager Server 2008 R2 agents. We have a pool of single CPU VMs that have been reporting “Operations Manager Agent CPU too high” alerts every ten hours or so (give or take a few hours). Unfortunately, I am not able to catch the agents while the CPU spike is taking place. Maybe I could set up a “Data Collector Set” to gather lots of process information when a CPU spike condition occurs, but I am feeling lazy and don’t want to do it.

So instead, I am taking a different approach… disabling non-essential discoveries to see if this lightens the load on the agents enough to stop the CPU spikes.  I thought I knew how to do this already, but my fist pass failed, and I had to learn something new (gasp!).  My thanks to Jonathan Almquist for his post on this subject:
http://blogs.technet.com/b/jonathanalmquist/archive/2008/09/14/remove-disabledmonitoringobject.aspx
Without that one, I would still be foundering.

I our case, I wanted to suppress discovery of System Center Configuration Manager 2007 Clients in the SCCM 2007 Management Pack.  To accomplish this, we need to identify the pertinent discovery rules, create a group that contains the agents that we want to exclude from discovery, then override the discovery for this new group.  We then can speed cleanup of the now obsolete discovered objects using the PowerShell “remove-disabledMonitoringObject” cmdlet.

  1. Go to the OpsMgr console, change to the the Authoring->Management Pack Objects->Object Discoveries view.  Use the “change scope” option to limit the displayed discovery rules to only those in the Configuration Manager management packs.  In this instance, we see there are rules for “Microsoft ConfigMgr 2007 Clients Discovery” and “Microsoft ConfigMgr 2007 Advanced Client Discovery”.  I will disable discovery for both of these.  Before moving on, take careful note of the “target” column.  In this case the target is “MOM 2005 Backward Compatibility Computer”, not “Windows Computer”, as you might expect.
  2. Change to the Authoring->Groups view.  Create a group that includes only objects of the type you identified in the first step.  I used dynamic inclusion rules to add all entities that do not match the naming convention of our Configuration Manager servers.
  3. Now go back to the Object Discoveries view, find the rules you want to override again, and add an override for objects in your new group.
  4. You could wait a few discovery cycles for the discovered entities to go away, or just pop into the OpsMgr PowerShell console, and run “remove-DisabledMonitoringObject”.  If you did your override rules properly, your undesirable objects should disappear right away.

I now have removed discovery and monitoring of the SCCM Client on all of the Windows Servers in my monitored environment.  We now shall see if this makes the OpsMgr Agent CPU utilization alerts go away.

Automatic Maintenance Mode after Windows Update

Since our original deployment of Microsoft Operations Manager 2005 (now System Center Operations Manager 2007 R2), we have struggled with handling of alerting during maintenance windows for our servers.  As many a SCOM admin can tell you, your management server can do a whole lot of squawking if you fail to set maintenance mode before a server reboot.

In the past, we configured notification blackout windows during scheduled Windows Update times.  This worked, sort of… While we did not get paged during system reboots, we none-the-less had a lot of alerts that needed to be closed out every time a system rebooted.  Why?  Because suppressing notification does not suppress alerting.  What we really needed to do was to put systems into maintenance mode before the Windows Update triggered reboot.  This always seemed to difficult to implement, so we never did it.  Scripting of Maintenance Mode requires that any account attempting to start MM have "admin" rights on the RMS.  We did not want to go there with distributed scripts, and so we were at an impasse.  The other big disadvantage to this approach was that it suppressed notifications for all systems in the infrastructure during scheduled maintenance windows, not just the ones that needed a reboot at that particular time.  If there were no Windows Update-induced reboots that evening, too bad.

But now, thanks to Group Policy Client-Side Extensions, PowerShell, various improvements in SCOM 2007 R2 Maintenance Mode, and the fine work of blogger and SCOM developer Derek Harkin, we have a workable solution to this problem.

Prerequisites:

  • GP Client-Side Extensions with item-level targeting (requires that CSE update be applied to any Server 2003 systems under management, and also use of a Vista-later system running a current Group Policy Management Console)
  • Derek Harkin’s Maintenance Mode Management Pack for SCOM 2007 R2:
    http://derekhar.blogspot.com/2009/11/new-agent-maintenance-mode.html
  • Our additional VBScript "WUTriggerMaintMode.vbs" to create events triggered by Windows Updates
  • "eventtriggers.exe" on Server 2003 (should be included with the OS)
  • "schtasks.exe" on Server 2008 (definately included with the OS)
  • Windows Scripting Host to run VBScripts on all managed targets (should be there by default…)
  • PowerShell on the RMS (required to install SCOM 2007 R2 anyway…) 

How it Works:

  • Group Policy Preferences run every day to refresh the scripts and Scheduled Tasks on all managed systems.  "MaintModeTrigger2008.xml", "WUTriggerMaintMode.vbs" and "MaintMode.vbs" are distributed to all systems, and tasks are created which perform the following task:
    Watch the "SYSTEM" event log for event ID 22 from source "WindowsUpdateClient".  When this event is seen, run the "MaintMode.vbs script with the arguments "ON 20M"
  • The MaintMode.vbs script will create a WSH entry in the Application event log which will state that the system needs to enter Maintenance Mode for 20 minutes.
  • A trigger set by the Maintenance Mode management pack will detect this event, and run a PowerShell script on the Root Management Server (RMS) that will put the desired system into Maintenance Mode.

Installation:

  1. Configure your Windows Update policy to delay at five minutes after update application before initiating system reboot.
  2. Install the Maintenance Mode management pack following the included directions.
  3. Create a Group Policy object which is linked to your managed servers for the purpose of configuring Windows Update-triggered Maintenance Mode settings.
  4. Copy the MaintMode.vbs script and, our "WUTriggerMaintMode.vbs", and custom XML file to all managed Windows servers using Group Policy Preferences.  I copied my scripts to “%SystemDrive%\local\scripts” on all managed system, but you could just plan to run the scripts off of a trusted, highly-available network share.
    NOTE: If you will be copying the files from a network share to a local directory using GP Preferences, then you must grant “Domain Computers” read/execute/traverse rights to the parent directory that contains your files.  If you will be executing the files directly from the file server, you need grant “Domain computers” rights to only the files themselves.
  5. Create a Scheduled Task preference targeted to Server 2003 and Server 2003 R2 systems.  This task will use cscript.exe to run "WUTriggerMaintMode.vbs".  I configured this task to run daily, but you could adjust the interval to suit your needs (weekly, monthly, on next reboot, run once, etc.)
  6. Create a Scheduled Task preference targeted to Server 2008 and Server 2008 R2 systems.  This taks will run schtasks.exe to create a scheduled task from the "MaintModeTrigger2008.xml" task specification file.  I used the command line:
    schtasks.exe /Create /RU SYSTEM /TN "WUTriggerMaintMode" /XML [pathToFile]\MaintModeTrigger2008.xml /F

    Note:  I was unable to use the "Vista and later" version of the Scheduled Task preference, which is exposed when running GPMC on a Windows 7 client.  I am unclear why this did not work, but just to be safe you probably will need to use the "legacy" Scheduled Task tool.
    Again, you will need to set a schedule for this task that suits your environment.

  7. Force a Group Policy update on one of each OS type that you are managing to ensure that the GP Preferences are being distributed.
  8. Run the Scheduled Tasks that are created by the the policy to make sure that they work (at least on Server 2003, you can create synthetic events in the SYSTEM event log using "EVENTCREATE.exe".  You can set of the event triggers using this tool.).
  9. Enjoy the silence during your next system maintenance.

Required Files:

MaintModeTrigger2008.xml –

the file which creates a Server 2008 scheduled task that will run the MaintMode.vbs script after Windows Update signals that the server will be rebooted (an Event ID 22).

Contents:



  
    2009-12-04T13:05:53.1637625
    CAMPUS\admin-jgm
  
  
    
      PT30M
      true
      <QueryList><Query Id="0" Path="System"><Select Path="System">*[System[Provider[@Name='Microsoft-Windows-WindowsUpdateClient'] and EventID=22]]</Select></Query></QueryList>
    
  
  
    
      HighestAvailable
      S-1-5-18
    
  
  
    
      true
      false
    
    IgnoreNew
    false
    true
    true
    false
    false
    true
    true
    false
    false
    false
    PT1H
    7
  
  
    
      cscript.exe
      %SystemDrive%\local\scripts\MaintMode.vbs ON 20M
      %systemroot%\system32
    
  

WUTriggerMaintMode.vbs –

Uses "eventtriggers.exe" on Server 2003 systems to create an event trigger (and corresponding Scheduled Task) which will watch the system event log for Windows Update-triggered reboots (event ID 22) and run MaintMode.vbs when this happens.  By the way, I know that this is crumby code… it could/shold be updated to provide better error handling and return codes.

Contents:

' WUTriggerMaintMode.vbs - Trigger Maintenance Mode based on pending Windows
' Update-triggered reboot, as seen in the System event log
Option Explicit

Dim bSetTrig
Dim fLog
Dim oShell, oExec, oExec2, oFile
Dim sLine, sSub, sSysDrive

Set oShell = WScript.CreateObject("WScript.Shell")
sSysDrive = oShell.ExpandEnvironmentStrings("%SystemDrive%")

' Create Log File
Set oFile = CreateObject("Scripting.FileSystemObject")
Set fLog = oFile.CreateTextFile(sSysDrive & "\local\scripts\WUTriggerMaintMo" _
	& "de.log", true)

' Query for existing triggers:
Set oExec = oShell.Exec("eventtriggers.exe /query")

' If Existing triggers previously created by this script are present,
' delete them:
Do While Not oExec.StdOut.AtEndOfStream
	sLine = oExec.StdOut.ReadLine
	fLog.WriteLine(sLine)
	if InStr(sLine,"Windows Update - Reboot") then
		fLog.WriteLine("Existing trigger detected")
		sSub = Left(LTrim(sLine), 1)
		fLog.WriteLine("Trigger ID is: " & sSub)
		set oExec2 = oShell.Exec("eventtriggers.exe /delete /tid " & sSub)
		Do while oExec2.Status = 0
			WScript.sleep 100
		Loop
		fLog.WriteLine("Deletion of Event trigger attempted.")
	end if
Loop

' Create latest event trigger:
set oExec2 = oShell.Exec("eventtriggers.exe /create /tr ""Windows Update - R" _
	& "eboot Impending"" /l SYSTEM /eid 22 /tk ""cscript.exe %SystemDrive%\l" _
	& "ocal\scripts\MaintMode.vbs ON 20M"" /ru ""System"" ")
Do while oExec2.Status = 0
	WScript.sleep 100
Loop
fLog.WriteLine("Creation of event trigger attempted.")

Getting cruft objects out of Operations Manager

Recently I had to downgrade a SQL Express instance from the 2008 version back to 2005.  The downgrade solved my DB performance problems, but created a monitoring problem.  Operations Manager continued to believe that this server was running SQL 2008!

So, how do you get rid of a monitored object that is part of a dynamically discovered group?  The answer lies (as with most OpsMgr problems) in overrides:

http://blogs.msdn.com/boris_yanushpolsky/archive/2007/11/20/opsmgr-sp1-removing-instances-for-which-discovery-is-disabled.aspx

Boris of OpsMgr++ fame tells us to use the “Authoring” view in the OpsMgr console to find the “Object Discovery” rule that found your SQL instance (probably “SQL 2008 DB Engine”).  You then generate on override which will disable discovery for your named computer.  Since SQL discovery runs fairly infrequently, you may also want to override the same rule for all computers, forcing discovery to a more frequent interval (say… 300-600 seconds).

After discovery completes, open the OpsMgr PowerShell console, and run the “Remove-DisabledMonitoringObject” cmdlet (with no arguments).  If you are exceptionally lucky, your undesired object will disappear from the OpsMgr Monitoring view in short order.

OpsMgr Severity/Priority Levels: "What 'IS' is."

When working with OpsMgr overrides, I am always forgetting the mappings between alert severities and their corresponding numeric values in the database.  It is important to keep this straight, because if you set your overrides incorrectly, you risk either suppressing all notification for an alert, or even worse… increasing the number of notifications that you receive!

Marius provide the following mapping info in his fine blog on MSDN:

Mapping:

Alert Severity – Its corresponding integer value

Critical – 2
Warning – 1
Information – 0

Alert Priority – Its corresponding integer value

High – 2
Medium – 1
Low – 0

Read more here:

http://blogs.msdn.com/mariussutara/archive/2007/12/17/alert-severity-and-priority-use-with-override.aspx

So remember, when downgrading an alert from "Critical" to "Warning", change in from "Severity 2" to "Severity 1".  "Severity 3" will just cause more paging… TWTTTTH!