App-V Server Configuration, Load Balanced Configuration

In my last post I discussed issues around Kerberos configuration for an App-V 5 server cluster in a load balanced configuration.  Today I will discuss subsequent configuration requirements for making App-V publishing function in a load-balanced environment.

After configuring standalone app-V servers with Management and Publishing Server roles, I had good success with adding packages to the environment and publishing them.  However, when switching to a load balanced configuration, I experienced a failure of the publishing server to pick up on changes in the management configuration.  Helpful resources and troubleshooting notes follow:

  • A TechNet social page that I referenced in my previous post makes reference to this same problem:
    But does not point me towards any solutions.  This seems like some sort of permissions problem, so I put Sysinternals Procmon on watching w3wp.exe for “Access Denied” events, but I get nothing.  However, I do see a fair amount of database traffic at IIS startup time.
  • The following TechNet blog provided a key tipoff in App-V server diagnostics:
      The trick was to select “Show Analytic and Debug Logs” under “Actions” in the event viewer.  With this option enabled, I now see App-V management and publishing debug logs instead of just the default App-V event logs.  The debug logs contain the real error.  We see that the SID recorded for the publishing server in the management server database does not match the SID of the account making the connection!  What we needed to do was delete the publishing server entries from the management configuration, and create one new “server” under the name of the publishing server service account, not the computer account.  I just updated the SQL database entry manually, but I likely could have just used the Silverlight UI instead.  This change cleared up the mismatched SID error, but now we get an “access denied” error to the publishing metadata directory.
  • The following blog gives an excellent technical overview of App-V server infrastructure and the general troubleshooting process for resolving configuration issues:

    • Here it is suggested that I look at HKLM/Software/Microsoft/AppV/Server to review the management and publishing server configurations.  Sure enough, one problem seen here is that the publishing server is configured to connect to the management server on an http:// address.  However, I updated the management servers to use https://.  I modified those registry values and restarted IIS.  Still no luck… 
    • This blog explains how published applications are read out of a metadata xml file that is exposed to the publishing server by the management server.  Both are stored in c:\programdata\microsoft\appv.  When running Procmon.exe against w3wp.exe we see “Access Denied” to these directories by our service account.  After adding “modify” rights for the service account to these directories, metadata updates again start to happen.

It is unsurprising that switching form the use of a local server account to an AD service account caused access problems for the App-V server.  The difficulty of discovering where account info and rights needed to be updated was a bit of a surprise.  But thanks to the blog-o-sphere and the mighty “procmon.exe”, we have our answers.

Now on to performance testing…

App-V 5 Server, F5 Load Balancers, and Kerberos

More fun today with Kerberos and load balancers.  Today’s challenge related to getting the Microsoft App-V publishing server to work with an F5 load balancer in a Layer 4/n-Path/DSR configuration.  Everything was working when I was accessing the individual server nodes, but when I switched to using the load balanced name and address, authentication started to fail.

After lots of log searching I eventually tried a wire trace, and found the following Kerberos error in the response from the App-V server to the App-V client:

Lots of different resources helped here:

  • This TechNet page explains various Kerberos errors and why they might occur:

    Of note is the scenario where the account handling the authentication request does not hold the SPN for which the request was made.  I set the SPN for my IIS application pool identity, but further analysis of the error packet shows that it was handled by my App-V server machine account, not the service account.  Augh!  Why?

  • This thread on TechNet Social was the biggest help:

    The user posted all of the steps they followed in configuring IIS and the service account SPN, including the tidbit:
    changed the authentication of the “Management Service” web site to useAppPoolCredentials=”true”
    I have never used this particular setting, so I dug into it…

  • The following MSDN article explains the IIS 7.0 feature of “kernel authentication”, how it affects the need for SPN entries, and its interplay with application pool identity accounts:

    Basically, with kernel-mode authentication, the SYSTEM account will handle all Kerberos authentication by default.  This explains why we were seeing Kerberos errors in the communications with the App-V client… the IIS pool identity account was not handling Kerberos delegation!

    Of special interest is this statement:
    Disable Kernel mode authentication and follow the general steps for Kerberos as in the previous IIS 6.0 version.
    [Recommended for Performance reasons]
    Let Kernel mode authentication be enabled and the Application pool’s identity be used for Kerberos ticket decryption. The only thing you need to do here is:
    1. Run the Application pool under a common custom domain account.
    2. Add this attribute “useAppPoolCredentials” in the ApplicationHost.config file.

  • This TechNet page documents how to configure Kerberos auth in IIS, and mentions the use of the IIS appcmd.exe to set the “useAppPoolCredentials” option:
    Included is the exact command line required to set the value to true:
     appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -useAppPoolCredentials:true
    (But the page does not really tell you what it is for, which is where the MSDN article comes in handy.)

So, Kerberos under IIS 7 and later has some nuances not present in IIS 6.  I wonder how I did not encounter this before?

Using SQL Logins with “AlwaysOn” Availability Groups

I had some fun today configuring a SQL 2012 “AlwaysOn” Availability Group working with a database that was configured to use SQL Logins.  I am working with the vendor to see if we can use Integrated Authentication instead, but in the meantime I managed to get failover functional.

The problem I was experiencing was that on failover, the database users defined in the database lost their mappings to the SQL Login on the server.  I had created SQL Logins on both nodes of the Availability Group with identical usernames and passwords, but the mapping still failed.  Turns out this was happening because SQL Logins (like AD and NT accounts) have SIDs, and these SIDs are used to map database users to logins.  To correct the problem, I needed to create SQL Logins with identical SIDs on the two servers.


  1. Create SQL Logins on one node of the Availability Group and perform mappings.
  2. Lookup the SIDs of the users using the following query:
    SELECT SUSER_SID (‘[SqlLogin]‘)
  3. Script out creation of the matching logins:
    USE [master]
  4. Failover the Availability Group and verify that user mappings are working as planned.  Verify that passwords are working, too.

Another useful tidbit here is a script to re-map SQL database users to local SQL logins:

USE myDatabase
sp_change_users_login @Action=’update_one’, @UserNamePattern=’databaseUserName’, @LoginName=’SqlLoginName’;

Really I am surprised that I did not run into this problem with our SQL 2008 mirrored databases.  The problem should have been present there as well.

The case of the undeletable directory

I had a “fun” time today with some directories that I could not delete.  These were mandatory roaming profiles that I previously had attempted to upload to a file server using “robocopy”.  Owing to switches that I used when performing the upload, directory junctions were treated as files, and robocopy got caught in a recursive loop because of a circular reference to “Application Data”.  By the time I caught the problem, I had created a set of nested “Application Data” directories that must have been over 400 characters long.

All of the tricks I had used in the past to delete “deep” directories were failing me.  “Dir /s /q [dirName]” failed with “Access Denied”, even though I was the directory owner.  Running the command as System encountered the same problem.  I mapped a drive to the directory, as deep down in the nested folders as I could get.  From there, I was able to “CD” to the deepest “Application Data” directory, but I still could not delete the directory.  (I got a “directory not empty” error.)

Eventually, Google unveiled a suggestion to use “robocopy” to mirror an empty directory to the problematic directory.  (Unfortunately, I have lost track of the link, so I cannot give credit where it is due.)  “Good idea,” I thought.  Why not use the utility that created the problem to solve the problem?  After all, if robocopy can create paths that are 400 characters long, perhaps it can delete them, too.

To test, I simply created an empty directory “E:\foo”, and ran the command:
robocopy.exe /mir /e E:\foo E:\mandatory\corruptProfile

Robocopy quickly chewed though the problematic profile, and a few seconds later I had an empty directory that I was able to delete.  Hurray!

LiteTouch failures on UEFI systems

Today I got a complaint that a user could not deploy Windows 8 on a UEFI-enabled computer (a Latitude 10 tablet, to be precise) using our MDT 2012 Update 1 deployment share (in Lite Touch mode, of course).  At the same time, I was experiencing problems deploying Windows 8 on a Dell XPS 14 with UEFI firmware enabled.  Interestingly, the deployment problem did not occur when running with Legacy BIOS enabled.  What gives?

The error reported by Lite Touch was: “MDT FAILURE (5616): Verify BCDbootex”.  Much digging though the logs revealed the command line that triggered the problem (It was spelled out in the “LTIApply.log”).  This command was something like:
\\server\deployShare\Tools\x64\bcdboot c:\windows /l en-us /s v: /f UEFI

If I try to run the command above manually, I just get back the bcdboot.exe help dialog.  Further investigation revealed that the version of bcdboot.exe on the server share is dated to 2009.  This is the Windows 7 RTM release version!

I created a new deployment share to see if the bad version of bcdboot.exe gets placed there again… it did not.  So, I removed bcdboot.exe from the deployment share (along with an old version of ImageX.exe and some other expired utilities).  Upon re-running LiteTouch, deployment succeeds.  Both Windows 7 and Windows 8 boot media contain bcdboot.exe in the search path… no server side copy is required.  I wonder how it got there in the first place?  Maybe we were trying to make it easier for people to capture system images using the LiteTouch boot media.  Another entry for the department of unintended consequences…

VDI Profile Loading Delays

We are noticing that it takes rather a long time for users to log in to our VDI environment (~2 minutes, in some circumstances).  I did some analysis of login times using Sysinternals Procmon.  (Enable boot logging, use the “view process tree” feature to look at process times at logon.  See for details).  What I found was that a child process of explorer.exe called “ie4uinit.exe” was running for most of this time.  This process appears to be part of Microsoft “Active Setup” (discussed in some detail here:

So what if we disable Active Setup?  Noise on the Internet suggests that this is possible , simply be deleting the key:
HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup
as suggested here:

However, there is some indication that this could have unintended consequences.  In my case, it immediately caused a logon script to fail to run.  Bummer!

What other solutions are possible?  Members of the Windows in Higher Education mailing list recently recommended using mandatory profiles.  There is a reasonably good rundown of the mandatory profile creation process here:
Missing details are:

  1. It is possible for the mandatory roaming profile to be stored locally (i.e. “C:\Users\VDI_Mandatory.V2″ to avoid over-the-network profile copy delays.  However, in our View environment, using a network location appears to be faster!
  2. The mandatory roaming profile can be specified using the Group Policy settings in Computer -> Policies -> Administrative Templates -> System -> User Profiles.  (See “Set roaming profile path for all users logging onto this computer” and “Delete cached copies of roaming profile”.)

In testing, I found initial logon times were reduced from two minutes to approximately 20 seconds.  Good!  (But still not great.)  Additional benefits are that it is no longer necessary to run the logon script that I developed to customize the Start Menu and Task Bar.  I also can remove the Group Policy preferences that clean up local profiles on the computer.

MBAM Configuration Nuances

This week we are continuing testing of the new Microsoft Bitlocker Administration and Management 2.0 tool (MBAM).

MBAM is not overly complicated, but it does have several service tiers and dependencies which make initial setup a bit irksome. After plowing though configuration of a SQL database, SQL Reporting Services, and IIS, we are still need to configured MBAM Group Policy settings, and then we needed to do a fir number of tweaks to make the service actually work. Here are the most significant deviations from the official documentation:

  1. The Group Policy templates for MBAM are not uploaded to the AD Policy Store during product installation, nor does the documentation recommend that you complete this step. However, if you want to be able to edit MBAM Policy from any workstation in the domain, you really do need to upload the ADMX templates. Making this happen is easy… just use the MBAM installer to install the MBAM policy templates locally, then open c:\windows\PolicyDefinitions, and copy BitLockerManagement.admx and BitLockerUserManagement.admx to \\[domain]\SYSVOL\[domain]\Policies\PolicyDefinitions (you will need domain admin rights to do this. Also copy the corresponding .adml files in the local language directory of your local PolicyDefinitions directory to the local language directory on the domain controller (in my case, these are in the “en-US” subdirectory).
  2. After installing the MBAM Client and policy settings, clients were failing to auto-initiate encryption, and were failing to report status to the management server.  The MBAM Admin Event Logs were showing the following error:
    Log Name: Microsoft-Windows-MBAM/Admin
    Source: Microsoft-Windows-MBAM
    Event ID: 4
    Task Category: None
    Level: Error
    User: SYSTEM
    Description: An error occurred while sending encryption status data.
    Error code: 0x803d0013

    This is occurring for a few reasons.  One, the MBAM server is not trusted for delegation, so it cannot perform Kerberos authentication in IIS.  Two, the public URL for MBAM services ( does not match the internal name of the server (BAM1).  To fix this, we needed perform a few additional configuration steps:

    1. Create the following key and value on the MBAM management server:
      DWORD(32-bit) - DisableMachineVerification
      Value = 1
    2. On the MBAM Administration Server AD object, enable the “Trust for delegation for any service (Kerberos Only) option”, under the Delegation tab.
    3. Use the “setspn” utility to add additional principal names for the public URL of the server to the AD server account:
      setspn -A HOST/ MYDOMAIN\MyServer$
      setspn -A HTTP/ MYDOMAIN\MyServer$
      setspn -A RestrictedKrbHost/ MYDOMAIN\MyServer$
      (Note that if using a service account to run the MBAM Administration Service, you should use “setspn” to set the HOST/HTTP names for the service account instead of the domain computer account).
    4. It appears that it may also be necessary to add the “BackConnectionHostNames” Reg_multi_Sz value to “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0″ to include any public dns names used by the MBAM Administration Server (this likely only is necessary in a load balanced configuration).

    We then needed to perform an IISRESET on the management server, and cycle the MBAM Clients.

  3. The MBAM Help Desk web application was failing to display reports.  This was happening because the installer grabbed the unencrypted reporting services URL from the reporting services instance.  I had to open:
    C:\inetpub\Microsoft BitLocker Management Solution\Help Desk Website\web.config
    Then locate the tag, and edit the URL value to the SSL version of the Reporting Services web site.
  4. The MBAM documentation claims that you will use MBAM policies in place of standard Windows BitLocker policies.  This is somewhat misleading… Many MBAM policy settings also will change the “classic” BitLocker policy settings, so it will appear that you have configured both classic and MBAM policies in the editor.  This would not really be a problem were it not for the fact that MBAM policies are not comprehensive.  You may need to return to the “classic” settings to configure appropriate behavior in your environment.  For example, we experienced difficulty in encrypting a Dell Latitude 10 tablet using MBAM.  On this machine, we saw the following error in the MBAM Admin Event Log:
    Event ID: 2
    An error occurred while applying MBAM policies.
    Volume ID:\\?\Volume{VolumeGUID}\
    Error Code: 0x803100B6
    Details:  No pre-boot keyboard or Windows recovery Environment detected.  The user may not be able to provide the required input to unlock the volume.

    This error is happening because our policy is set to “Allow PIN” (BitLocker PIN Authenticator is allowed, but not required).  Apparently, MBAM default-fails the attempted encryption, even though this is not a “fatal” error.  To allow encryption to continue, I needed to set the classic policy “Enable use of BitLocker authentication requiring preboot keyboard input on slates” as defined here:
    With this policy in place, encryption completes successfully on the tablet computer.

Other than these caveats, the tool does appear to be working.  Setting up our PGP Universal Server was easier, but suffering though the pain of ongoing PGP disk encryption support was agonizing.  Hopefully a little time spent on configuring a solid BitLocker support environment will bear lasting fruit for our constituents down the road.

Additional Resources:

Rick Delserone’s MBAM: Real World Information – A rundown on MBAM Certificate Configuration, Group Policy Templates, and undocumented registry settings:

SQL Reporting Services – Load Balancing Notes

As part of our evaluation of a SQL Consolidation Server, I am attempting a deployment of SQL Reporting Services in a load-balanced environment (using F5 load balancers in layer 4/fast npath config). I have followed my usual procedure for setting up a npath/direct server return config on Windows Server (with a couple of caveats… on Server 2012 the name of the MS Loopback Adapter has been changed to ‘Microsoft KM-TEST Loopback Adapter’). I then installed Reporting Services on both load balanced servers and connected them to the same back end database, and configured them to use the same SSL certificate.

Problems arise when attempting to connect to the load-balanced Reporting Services URL over SSL. Initial connection succeeds, but authentication fails. Log trawling reveals the following event:

Log Name:    System
Source:      LSA (LsaSrv)
Event ID:    6037
Level:       Warning

The program ReportingServicesService.exe, with the assigned process ID 13432, could not authenticate locally by using the target name HTTP/ The target name used is not valid. A target name should refer to one of the local computer names, for example, the DNS host name.

Try a different target name.

I found a similar error discussed at the following site:

The solution there worked for me, too:

  1. Go to REGEDIT and open the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
  2. Right click MSV1_0 –> New -> Multi-String Value
  3. Type BackConnectionHostNames and click Enter.
  4. Right click on newly created value and select Modify.
  5. Enter the hostname of the site: WEBSITENAME (and on a new line enter the FQDN,
  6. Restart IIS

I also had considered enabling Kerberos authentication for Reporting Services, but I have decided to forgo this step (since, I think, we will want users to be able to retrieve reports from non-domain-joined computers).  However, if we did want to enable Kerberos, three things need to happen:

  1. Use SETSPN to add the HTTP/[reportServerHostName] [domain]\[reportServerServiceAccount]
  2. Configure the Reporting Services Service Account to be trusted for Kerberos Delegation (using AD Users and Computers, under the “Delegation” tab)
  3. Modify the rsreportserver.config file (in %ProgramFiles%\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\ReportServer).  Find the “AuthenticationTypes” section, and specify <RSWindowsNegotiate/> as the first/only entry.
  4. Restart Reporting Services

SQL Server 2012, Transparent Data Encryption, and Availability Groups

We are looking into using Microsoft Bitlocker Administration and Monitoring (MBAM) 2.0 to manage BitLocker in our environment. One requirement for MBAM is a SQL Server database instance that supports Transparent Database Encryption (TDE).  (Update 2013-06-04:  Microsoft now claims that TDE is “optional” with MBAM 2.0, which is nice to know.  If only they had told me this before I went to the trouble of setting up SQL 2012 Enterprise just for this project!)  Currently we also are in the process of investigating the creation of a consolidation SQL 2012 Enterprise Edition “Always On” Availability Group. I wanted to see if I could create the MBAM Recovery Database in a SQL 2012 Availability Group. This proved slightly tricky… fortunately I was able to find a decent reference here:

The trick is, you need to create SQL Certificates that on each member server of the Availability Group that have the same name and are generated from the same private key. The procedure follows…

On the first server in the group, create a SQL Master Key and Certificate by running the following code. The script will create a backup file in your SQL Server data directory. Move this file to an archival location. If you lose the file and password, you will not be able to recover encrypted databases in a disaster event:


-- Create a Master Key

-- Backup the Master Key
   TO FILE = 'Server_MasterKey'
   ENCRYPTION BY Password = 'Password2';

-- Create Certificate Protected by Master Key
CREATE Certificate SQLCertTDEMaster
   WITH Subject = 'Certificate to protect TDE key';

-- Backup the Certificate
BACKUP Certificate SQLCertTDEMaster
   TO FILE = 'SQLCertTDEMaster_cer'
   WITH Private KEY (
       FILE = 'SQLCertTDEMaster_key',
       ENCRYPTION BY Password = 'Password3'

Now create a master key on any secondary servers in the availability group, and create the same cert by using the backup file from the first step, above. You will need to copy the certificate backup files to the local server data directory, or use a network share that is accessible to the account running the script:

-- Create a Master Key
-- Backup the Master Key
   TO FILE = 'Server_MasterKey'
   ENCRYPTION BY Password = 'Password2';

-- Create Certificate Protected by Master Key
CREATE Certificate SQLCertTDEMaster
   FROM FILE = 'SQLCertTDEMaster_cer'
   WITH Private KEY (
       FILE = 'SQLCertTDEMaster_key',
       Decryption BY Password = 'Password3'

To avoid needless trouble, create your new database and add it to your availability group before encrypting the database. Once the database is created, you can initiate encryption by opening SQl Management Studio, right-clicking your database, select tasks, then select “Manage Database Encryption”. Select the option to generate the database encryption key using a server certificate. Select the certificate created above, and select the option to “set database encryption on”.

Once the database is encrypted, be sure to test availability group failover to make sure the secondary servers are able to work with the encrypted database.

Improving Notifications in System Center Operations Manager 2012

Anyone who depends on System Center Operations Manager 2012 (or any earlier version of SCOM, back to MOM) likely has noticed that notifications are a bit of a weak spot in the product.

To address this, we have use the “command channel” to improve the quality of messages coming out of SCOM.  Building on the backs of giants, we implemented a script that takes an AlertID from SCOM, and generated nicely formatted email and alpha-numeric pager messages with relevant alert details.

More recently, we have identified the need to generate follow-up notifications when an initial alert does not get addressed.  I went back to our original script, and updated it to use a new, custom Alert ResolutionState (“Notified”), and I have added logic to update the Alert CustomField1 and CustomField2 with data that is useful in determining whether or not an alert should get a new notification, and how many times follow-up notifications have been sent.

Heart-felt appreciation goes out to Tao Yang for his awesome work on his “SCOMEnhancedEmailNotification.ps1″ script, which served as the core for my work here.

Here is my version… I don’t have a lot of time to explain it, but hopefully the comments give you enough to go on. Apologies for the rather bad munging of quotation marks… wordpress hates me this month. If you want to use this code, search for ampersand-quot-semicolon, replace with actual quotation marks.

# AUTHOR:	J. Greg Mackinnon, Adapted from 1.1 release by Tao Yang 
# DATE:		2013-05-21
# Name:		SCOMEnhancedEmailNotification.PS1
# Version:	3.0
# COMMENT:	SCOM Enhanced Email notification which includes detailed alert information
# Update:	2.0 - 2012-06-30	- Major revision for compatibility with SCOM 2012
#								- Cmdlets updated to use 2012 names
#								- &quot;Notified&quot; Resolution Status logic removed
#								- Snapin Loading and PSDrive Mappings removed (replaced with Module load)
#								- HTML Email reformatted for readability
#								- Added '-format' parameter to allow for alphanumeric pager support
#								- Added '-diag' boolean parameter to create options AlertID-based diagnostic logs
# Update:   2.2 - 2013-05-16    - Added logic to update &quot;CustomField1&quot; alert data to reflect that notification has been sent for new alerts.
#								- Added logic to update &quot;CustomField2&quot; alert data to reflect the repeat count for new alert notification sends.
#								- Added support for specifying alerts with resolution state &quot;acknowledged&quot;
#                               - Did some minor adjustments to improve execution time and reduce memory overhead.
# Update:	3.0 - 2013-05-20	- Updated to reduce volume of PowerShell instance spawned by SCOM.  Added &quot;mailTo&quot; and &quot;pageTo&quot; paramerters to allow sending of both short
#                                         and long messages from a single script instance.
#								- Converted portions of script to subroutine-like functions to allow repetition (buildHeaders, buildPage, buildMail)
#								- Restored &quot;Notified&quot; resolution state logic.
#								- Renamed several variables for my own sanity.
#								- Added article lookup updates from Tao Yang 2.0 script.
# Usage:	.\SCOMEnhancedEmailNotification.ps1 -alertID xxxxx -mailTo @('John Doe;','Richard Roe;') -pageTo @('Team Pager;')
#In OpsMgr 2012, the AlertID parameter passed in is '$Data/Context/DataItem/AlertId$' (single quote)
#Quotation marks are required otherwise the AlertID parameter will not be treated as a string.
	[string]$alertID = $(throw 'A valid, quote-delimited, SCOM AlertID must be provided for -AlertID.'),
Set-PSDebug -Strict

#### Setup Error Handling: ####
#$erroractionpreference = &quot;SilentlyContinue&quot;
$erroractionpreference = &quot;Inquire&quot;

#### Setup local option variables: ####
## Logging: 
#Remove '$alertID' from the following two log file names to prevent the drive from filling up with diag logs:
$errorLogFile = 'C:\local\logs\SCOMNotifyErr-' + $alertID + '.log'
$diagLogFile = 'C:\local\logs\SCOMNotifyDiag-' + $alertID + '.log'
#$errorLogFile = 'C:\local\logs\SCOMNotifyErr.log'
#$diagLogFile = 'C:\local\logs\SCOMNotifyDiag.log'
## Mail: 
$SMTPHost = &quot;;
$SMTPPort = 25
$Sender = New-Object System.Net.Mail.MailAddress(&quot;;, &quot;Lifeboat OpsMgr Notification&quot;)
#If error occured while excuting the script, the recipient for error notification email.
$ErrRecipient = New-Object System.Net.Mail.MailAddress(&quot;;, &quot;SAA Windows Administration Team&quot;)
##Set Culture Info (for knowledgebase article language selection):
$cultureInfo = [System.Globalization.CultureInfo]'en-US'
##Get the FQDN of the local computer (where the script is run)...
$RMS = $env:computername

#### Initialize Global Variables and Objects: ####
## Mail Message Object:
[string] $threadID = ''
$SMTPClient = New-Object System.Net.Mail.smtpClient
$ = $SMTPHost
$SMTPClient.port = $SMTPPort
##Load SCOM PS Module
if ((get-module | ? {$ -eq 'OperationsManager'}) -eq $null) {
	Import-Module OperationsManager -ErrorAction SilentlyContinue -ErrorVariable Err | Out-Null
## Management Group Object:
$mg = get-SCOMManagementGroup
##Get Web Console URL
$WebConsoleBaseURL = (get-scomwebaddresssetting | Select-Object -Property WebConsoleUrl).webconsoleurl
#### End Initialize ####

#### Begin Parse Input Parameters: ####
##Get recipients names and email addresses from &quot;-to&quot; array parameter: ##
if ((!$mailTo) -and (!$pageTo)) {
	write-host &quot;An array of name/email address pairs must be provided in either the -mailTo or -pageTo parameter, in the format `@(`'me;`',`'you;`')&quot;
$mailRecips = @()
Foreach ($item in $mailTo) {
	$to = New-Object psobject
	$name = ($item.split(&quot;;&quot;))[0]
	$email = ($item.split(&quot;;&quot;))[1]
	Add-Member -InputObject $to -MemberType NoteProperty -Name Name -Value $name
	Add-Member -InputObject $to -MemberType NoteProperty -Name Email -Value $email
	$mailRecips += $to
	Remove-Variable to
	Remove-Variable name
	Remove-Variable email
$pageRecips = @()
Foreach ($item in $pageTo) {
	$to = New-Object psobject
	$name = ($item.split(&quot;;&quot;))[0]
	$email = ($item.split(&quot;;&quot;))[1]
	Add-Member -InputObject $to -MemberType NoteProperty -Name Name -Value $name
	Add-Member -InputObject $to -MemberType NoteProperty -Name Email -Value $email
	$pageRecips += $to
	Remove-Variable to
	Remove-Variable name
	Remove-Variable email
if ($diag -eq $true) {
	[string] $(&quot;mailRecipients:&quot;) | Out-File $diagLogFile -Append 
	$mailRecips | Out-File $diagLogFile -Append
	[string] $(&quot;pageRecipients:&quot;) | Out-File $diagLogFile -Append 
	$pageRecips | Out-File $diagLogFile -Append
## Parse &quot;-AlertID&quot; input parameter: ##
$alertID = $alertID.toString()
#remove &quot;{&quot; and &quot;}&quot; around the $alertID if exist
if ($alertID.substring(0,1) -match &quot;{&quot;) {
	$alertID = $alertID.substring(1, ( $alertID.length -1 ))
if ($alertID.substring(($alertID.length -1), 1) -match &quot;}&quot;) {
	$alertID = $alertID.substring(0, ( $alertID.length -1 ))
#### End Parse input parameters ####

#### Function Library: ####
function getResStateName($resStateNumber){
	[string] $resStateName = $(get-ScomAlertResolutionState -resolutionStateCode $resStateNumber).name
function setResStateColor($resStateNumber) {
		&quot;0&quot; { $sevColor = &quot;FF0000&quot; }	#Color is Red
		&quot;1&quot; { $sevColor = &quot;FF0000&quot; }	#Color is Red
		&quot;255&quot; { $sevColor = &quot;3300CC&quot; }	#Color is Blue
		default { $sevColor = &quot;FFF00&quot; }	#Color is Yellow
function stripCruft($cruft) {
	#Removes &quot;cruft&quot; data from messages. 
	#Intended to make subject lines and alphanumeric pages easier to read
	$cruft = $cruft.replace(&quot;®&quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;(R)&quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;Microsoftr &quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;Microsoft &quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;Microsoft.&quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;Windows &quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot; without Hyper-V&quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;Serverr&quot;,&quot;Server&quot;)
	$cruft = $cruft.replace(&quot; Standard&quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot; Enterprise&quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot; Edition&quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;.campus&quot;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;.CAMPUS&quot;,&quot;&quot;)	
	$cruft = $cruft.replace(&quot;;,&quot;&quot;)
	$cruft = $cruft.replace(&quot;.AD.UVM.EDU&quot;,&quot;&quot;)
	$cruft = $cruft.trim()
	return $cruft
function fnMamlToHTML($MAMLText){
	$HTMLText = &quot;&quot;;
	$HTMLText = $MAMLText -replace ('xmlns:maml=&quot;;');
	$HTMLText = $HTMLText -replace (&quot;maml:para&quot;, &quot;p&quot;);
	$HTMLText = $HTMLText -replace (&quot;maml:&quot;);
	$HTMLText = $HTMLText -replace (&quot;</section>&quot;);
	$HTMLText = $HTMLText -replace (&quot;<section>&quot;);
	$HTMLText = $HTMLText -replace (&quot;<section>&quot;);
	$HTMLText = $HTMLText -replace (&quot;<title>&quot;, &quot;<h3>&quot;);
	$HTMLText = $HTMLText -replace (&quot;</title>&quot;, &quot;</h3>&quot;);
	$HTMLText = $HTMLText -replace (&quot;&quot;, &quot;<li>&quot;);
	$HTMLText = $HTMLText -replace (&quot;&quot;, &quot;</li>&quot;);
function fnTrimHTML($HTMLText){
	$TrimedText = &quot;&quot;;
	$TrimedText = $HTMLText -replace (&quot;&lt;&quot;, &quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;&quot;)
	$TrimedText = $TrimedText -replace (&quot;<h1>&quot;, &quot;<h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;</h1>&quot;, &quot;</h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;<h2>&quot;, &quot;<h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;</h2>&quot;, &quot;</h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;<H1>&quot;, &quot;<h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;</H1>&quot;, &quot;</h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;<H2>&quot;, &quot;<h3>&quot;)
	$TrimedText = $TrimedText -replace (&quot;</H2>&quot;, &quot;</h3>&quot;)
function buildEmail {
	## Format the message for full-HTML email
	[string] $escTxt = &quot;&quot;
	if ($resState -eq '1') {$escTxt = '- Repeat Count ' + $escLev.ToString()}
	[string] $script:mailSubj = &quot;SCOM - $resStateName $escTxt - $alertSev | $moPath | $alertName&quot;
	$mailSubj = stripCruft($mailSubj)
	[string] $script:mailErrSubj = &quot;Error emailing SCOM Notification for Alert ID $alertID&quot;
	[string] $webConsoleURL = $WebConsoleBaseURL+&quot;?DisplayMode=Pivot&amp;AlertID=%7b$alertID%7d&quot;
	[string] $psCmd = &quot;Get-SCOMAlert -Id `&quot;$alertID`&quot; | format-list *&quot;
	# Format the Mail Message Body (do not indent this block!)
	$script:MailMessage.isBodyHtml = $true
	$script:mailBody = @&quot;

<p><b>Alert Resolution State:<Font color='$sevColor'> $resStateName </Font></b><br />
<b>Alert Severity:<Font color='$sevColor'> $alertSev</Font></b><br />
<b>Object Source (Display Name):</b> $moSource <br />
<b>Object Path:</b> $moPath <br />
<p><b>Alert Name:</b> $alertName <br />
<b>Alert Description:</b> <br />
$alertDesc <br>
	if (($resState -eq 0) -or ($resState -eq 1)) {
		if ($isMonitorAlert -eq $true) {
$script:mailBody = $mailBody + @&quot;
<b>Alert Monitor Name:</b> $MonitorName <br />
<b>Alert Monitor Description:</b> $MonitorDescription
		}elseif ($isMonitorAlert -eq $false) {
			$script:mailBody = $mailBody + @&quot;
<b>Alert Rule Name:</b> $RuleName <br />
<b>Alert Rule Description:</b> $RuleDescription <br />
$script:mailBody = $mailBody + @&quot;
<b>Alert Context Properties:</b><br /> 
$alertCX <br />
<b>Time Raised:</b> $timeRaised <br />
<b>Alert ID:</b> $alertID <br />
<b>Notification Status:</b> $($alert.CustomField1) </br>
<b>Notification Repeat Count:</b> $($escLev.ToString()) </p>
<b>PowerShell Alert Retrieval:</b> $psCmd <br />
<b>Web Console Link:</b> <a href="&quot;$webConsoleURL&quot;">$webConsoleURL</a> </p>
	if (($resState -eq 0) -or ($resState -eq 1)) {
		foreach ($article in $arrArticles) {
		$articleContent = $article.content
$script:mailBody = $mailBody + @&quot;
<b>Knowledge Article / Company Knowledge `-$($article.Language):</b>
<p> $articleContent

$script:mailErrBody = @&quot;

<p>Error occurred when excuting script located at $RMS for alert ID $alertID.
<p>Alert Resolution State: $resStateName
<p><b>**Use below command to view the full details of this alert in SCOM Powershell console:</b>
<p> SCOM link:<a href="&quot;$webConsoleURL&quot;"> $webConsoleURL </a>

function buildPage {
	## Format the message for primitive alpha-numeric pager
	$script:moPath = stripCruft($moPath)
	[string] $escTxt = ''
	if ($resState -eq '1') {$escTxt = '- Rep Count ' +$escLev.ToString()}
	[string] $script:mailSubj = &quot;SCOM - $resStateName $escTxt | $moPath&quot;
	[string] $script:mailErrSubj = &quot;Error emailing SCOM Notification for Alert ID $alertID&quot;
	#UFT8 makes the message body look like trash.  Use ASCII (the default) instead.
	#$mailMessage.BodyEncoding =  [System.Text.Encoding]::UTF8 
	$script:MailMessage.isBodyHtml = $false
	$script:moSource = stripCruft($moSource)
	$script:alertName = stripCruft($alertName)
	$script:mailBody = &quot;| $moSource | $alertName | $timeRaised&quot; 
	$script:mailBody = stripCruft($mailBody)
function buildHeaders {
	## Complete the MailMessage object:
	$script:MailMessage.Sender = $Sender
	$script:MailMessage.From = $Sender
	# Regular (non-error) format
	if ($error.count -eq &quot;0&quot;) { 				
		$script:MailMessage.Subject = $mailSubj
		Foreach ($item in $recips) {
			$to = New-Object System.Net.Mail.MailAddress($, $
			Remove-Variable to
		$script:MailMessage.Body = $mailBody
	# Error format:
	else {									
		$script:MailMessage.Subject = $mailErrSubj
		$script:MailMessage.Body = $mailErrBody
	## Log the message if in diag mode:
	if ($diag -eq $true) {
		[string] $('Mail Message Object Content:') | Out-File $diagLogFile -Append
		$mailMessage | fl * | Out-File $diagLogFile -Append
#### End Function Library ####

#### Clean up existing logs: ####
if (Test-Path $errorLogFile) {Remove-Item $errorLogFile -Force}
if (Test-Path $diagLogFile) {Remove-Item $diagLogFile -Force}
if ($diag -eq $true) {
	[string] $(&quot;AlertID : `t&quot; + $alertID) | Out-File $diagLogFile -Append
	[string] $(&quot;MailTo      : `t&quot; + $mailto) | Out-File $diagLogFile -Append
	[string] $(&quot;PageTo      : `t&quot; + $pageto) | Out-File $diagLogFile -Append
	#[string] $(&quot;Format  : `t&quot; + $format) | Out-File $diagLogFile -Append

#### Begin Alert Handling: ####
## Locate the specific alert:
$alert = Get-SCOMAlert -Id $alertID
if ($diag -eq $true) {
	[string] $('SCOM Alert Object Content:') | Out-File $diagLogFile -Append
	$alert | fl | Out-File $diagLogFile -Append
## Read Alert Informaiton:
[string] $alertName = $alert.Name
[string] $alertDesc = $alert.Description
#[string] $alertPN = $alert.principalName
[string] $moSource = $alert.monitoringObjectDisplayName 	# Display name is &quot;Path&quot; in OpsMgr Console.
[string] $moId = $alert.monitoringObjectID.tostring()
#[string] $moName = $alert.MonitoringObjectName 			# Formerly &quot;strAgentName&quot;
[string] $moPath = $alert.MonitoringObjectPath 				# Formerly &quot;pathName
#[string] $moFullName = $alert.MonitoringObjectFullName 	# Formerly &quot;alertFullName&quot;
[string] $ruleID = $alert.MonitoringRuleId.Tostring()
[string] $resState = ($alert.resolutionstate).ToString()
[string] $resStateName = getResStateName $resState
[string] $alertSev = $alert.Severity.ToString() 			# Formerly &quot;severity&quot;
if ($alertSev.ToLower() -match &quot;error&quot;) {
	$alertSev = &quot;Critical&quot; 									# Rename Severity to &quot;Critical&quot;
[string] $sevColor = setResStateColor $resState				# Assign color to alert severity
#$problemID = $alert.ProblemId
$alertCx = $(1($alert.Context)).DataItem.Property `
	| Select-Object -Property Name,'#text' `
	| ConvertTo-Html -Fragment								# Alert Context property data, in HTML
$localTimeRaised = ($alert.timeraised).tolocaltime()
[string] $timeRaised = get-date $localTimeRaised -Format &quot;MMM d, h:mm tt&quot;
[bool] $isMonitorAlert = $alert.IsMonitorAlert
$escLev = 1
if ($alert.CustomField2) {
	[int] $escLev = $alert.CustomField2
## Lookup available Knowledge articles, if new alert:
if (($resState -eq 0) -or ($resState -eq 1)) {
	$articles = $mg.Knowledge.GetKnowledgeArticles($ruleId)
	if (!$error) {	#no point retrieving the monitoring rule when there's error processing the alert
		#if failed to get knowledge article, remove the error from $error because not every rule and monitor will have knowledge articles.
		if ($isMonitorAlert -eq $false) {
			$rule = Get-SCOMRule -Id $ruleID		
			$ruleName = $rule.DisplayName
			$ruleDescription = $rule.Description
			if ($RuleDescription.Length -lt 1) {$RuleDescription = &quot;None&quot;}
		} elseif ($isMonitorAlert) {
			$monitor = Get-SCOMMonitor -Id $ruleID
			$monitorName = $monitor.DisplayName
			$monitorDescription = $monitor.Description
			if ($monitorDescription.Length -lt 1) {$monitorDescription = &quot;None&quot;}
		#Convert Knowledge articles
		$arrArticles = @()
		Foreach ($article in $articles) {
			If ($article.Visible) {
				$LanguageCode = $article.LanguageCode
				#Retrieve and format article content
				$MamlText = $null
				$HtmlText = $null
				if ($article.MamlContent -ne $null) {
					$MamlText = $article.MamlContent
					$articleContent = fnMamlToHtml($MamlText)
				if ($article.HtmlContent -ne $null) {
					$HtmlText = $article.HtmlContent
					$articleContent = fnTrimHTML($HtmlText)
				$objArticle = New-Object psobject
				Add-Member -InputObject $objArticle -MemberType NoteProperty -Name Content -Value $articleContent
				Add-Member -InputObject $objArticle -MemberType NoteProperty -Name Language -Value $LanguageCode
				$arrArticles += $objArticle
				Remove-Variable LanguageCode, articleContent
	if ($Articles -eq $null) {
		$articleContent = &quot;No resolutions were found for this alert.&quot;
## End Knowledge Article Lookup
#### End Alert Handling ####

#### Begin Mail Processes:
if ($mailto) {
	# For all alerts, send full HTML email:
	$MailMessage = New-Object System.Net.Mail.MailMessage
	buildHeaders -recips $mailRecips
	invoke-command -ScriptBlock {$SMTPClient.Send($MailMessage)} -errorVariable smtpRet
if ($pageTo) {
	# For page-worthy alerts, format short message and send:
	$MailMessage = New-Object System.Net.Mail.MailMessage
	buildHeaders -recips $pageRecips
	invoke-command -ScriptBlock {$SMTPClient.Send($MailMessage)} -errorVariable smtpRet
#### End Mail Message Formatting #### 

# Populate CustomField1 and 2 to indicate that a notification has been sent, with repeat count.
if (!$smtpRet) { 							# IF the message was sent (apparently)...
	[string] $updateReason = &quot;Updated by Email notification script.&quot;
	[string] $custVal1 = &quot;notified&quot;
	if ($resState -eq &quot;0&quot;) { 				# . AND IF this is a &quot;new&quot; alert...
		$alert.ResolutionState = 1			# ..Set the resolution state to &quot;Notified&quot;
		$alert.CustomField2 = $escLev		# ..Set CustomField2 to the current notification retry count (presumably 1)
		if (!$alert.CustomField1) {			# ..AND if CustomField1 is not already defined...
			$alert.CustomField1 = $custVal1	# ... Set CustomField1.
	elseif ($resState -eq &quot;1&quot;) {		# .Or,If this is a &quot;notified&quot; alert
		if ($alert.CustomField2) {		# ..and the notification retry count exists..
			$escLev += 1				# ...Increment by one.
		$alert.CustomField2 = $escLev

Write-Host $error
##Make sure the script is closed
if ($error.count -ne &quot;0&quot;) {
	[string]$('AlertID string: ' + $alertID) | Out-File $errorLogFile
	[string]$('Alert Object Content: ') | Out-File $errorLogFile
	$alert | Format-List * | Out-File $errorLogFile
	[string]$('Error Object contents:') | Out-File $errorLogFile
	$Error | Out-File $errorLogFile
#Remove-Variable alert
#Remove-Module OperationsManager