Posts Tagged ‘F5’

Integrating SharePoint 2013 with ADFS and Shibboleth… The Motion Picture

Time again to attempt to implement that exciting technology, Federation Services (Web Single Sign On, SAML, WS-Federation, or whatever you want to call it) with SharePoint. Last time we tried this, SharePoint 2010 was a baby product, MS was just testing the waters with SAML 2.0 support in ADFS 2.0, and Shibboleth 2 was pretty new stuff here at UVM. The whole experience was unsatisfying. SharePoint STS configuration was full of arcane PowerShell commands, ADFS setup was complicated by poor farm setup documentation, and interop of Shibboleth 2 with ADFS 2 was not documented at all.  After wading though all of that mess, we ended up with user names being displayed as “i:05.t|adfsServiceName|userPrincipalName” (bleagh!), and with many applications that could not deal with Web SSO authenticators.  My general conclusion was that SharePoint really was not ready for federated login.

Four years later, things have changed a bit.  STS configuration still requires dense PowerShell commands, but at least it is better documented.  ADFS and Shibboleth interoperability also are excellently documented at this point.  Microsoft has most Office apps working with passive SAML authenticators, and has pledged to get the rest working this year.  While I would not judge the use of ADFS (with or without Shibboleth) to be the easy route to take to SharePoint 2013 deployment, it at least looks functional at this point.  So let’s kick the tires and see how it works…

Part 1: ADFS Setup

First, we need to setup ADFS.  We chose to deploy “ADFS 3″ using Windows Server 2012 R2 as the OS platform.  ADFS 3 is required to support the new “workplace join” feature of Server 2012 R2.  Since we want to test this, we would need ADFS 3 anyway.  Unfortunately, ADFS on Server 2012 R2 is pretty virgin territory, and does not have the same troubleshooting resources available for it as do earlier releases.

Most of the configuration steps followed the TechNet documentation without variation.  We did find that we needed to modify the ACL on the service account used to run ADFS… we added the “Authenticated Users” principal to the ACL, and assigned the “Read all properties” right.

For us, the most complicating factor in ADFS 3 deployment is the replacement of IIS with the Windows Kernel-mode HTTP server “http.sys”.  When we started experiencing connectivity problems with various clients to ADFS, we had no experience with HTTP.sys to assist in troubleshooting.  Most articles on HTTP.sys relate to remote desktop services, and with Server 2003.  Our problems with HTTP.sys were rooted in an undocumented requirement for clients to submit SNI (Service Name Indicator) information in their TLS “CLIENT HELLO” sequence.  I had to open a support case with Microsoft to resolve this problem, and only afterword was I able to find any Internet discussions that reflected MS advice:
http://social.msdn.microsoft.com/Forums/vstudio/en-US/d514b5a0-c01c-4ce4-b589-bca890e5921d/how-to-properly-setup-lb-probe-for-adfs-30-servers?forum=Geneva

It seems that if http.sys is bound using the hostname:port format, then TLS will require SNI.  If the binding is instead specified using ipAddr:port, SNI will not be required.  To fix our problem, we just needed to add a second HTTPS certificate binding using an IP address.  In this case, we just used “0.0.0.0:443″.  Here is the procedure:

  1. On each ADFS server and proxy, open an elevated command prompt
  2. run: netsh http show sslcert
  3. Record the certificate hash and application ID for the certificate used by ADFS
  4. run: netsh http add sslcert ipport=0.0.0.0:443 certhash= appid={}

Part 2: Configuring SharePoint to use ADFS

I started with Microsoft’s guide to configuring SAML authentication for SharePoint 2013 using ADFS:
http://technet.microsoft.com/en-us/library/hh305235(v=office.15).aspx
This is a well written guide, and fairly easy to follow.  The only issue I take with the article is the recommendation to use the “emailAddress” claim as the “identifier claim” in SharePoint.  In many federated login scenarios, a foreign Idp may not want to release the email address attribute.  However, some variation of UPN likely will be released.  In the case of the InCommon federation, the “ePPN” value (eduPersonPrincipalName) generally is available to federation partners.  For this reason I chose to implement “UPN” as the “identifier claim” in the last command of phase 3 of the document:

$ap = New-SPTrustedIdentityTokenIssuer -Name  -Description  -realm $realm -ImportTrustCertificate $cert -ClaimsMappings $emailClaimMap,$upnClaimMap,$roleClaimMap,$sidClaimMap -SignInUrl $signInURL -IdentifierClaim $roleClaimmap.InputClaimType

Part 3: Migrating SharePoint Users From Windows to Claims

Since we are planning an upgrade, not a new deployment, we need migrate existing Windows account references in SharePoint to federated account references. To make this happen, we need to establish the federated account identity format. I simply log in to an open-access SharePoint site as an ADFS user, and record the account information. In this case, out accounts look like this:

i:0e.t|adfs.uvm.edu|user@campus.ad.uvm.edu

and groups:

c:0-.t|adfs.uvm.edu|samAccountName

We then can use PowerShell to find all account entries in SharePoint, and use the “Move-SPUser” PowerShell cmdlet to convert them.  I am still working on a final migration script for the production cutover, and I will try to post it here when it is ready.

Of some concern is keeping AD group permissions functional.  It turns out that SharePoint will respect AD group permissions for ADFS principals, but there are a few requirements:

  1. The incoming login token needs to contain the claim type “http://schemas.microsoft.com/ws/2008/06/identity/claims/role“,
    and that this claim type needs to contain the SamAccountID (or CN) of the AD
    Group that was granted access to a site.  (In ADFS, this means that you need to release the AD LDAP Attribute “Token-Groups – Unqualified Names” as the outgoing claim type “Role”.
  2. When adding AD Groups as permissions in SharePoint, we need to use the “samAccountName” LDAP attribute as the identifier claim.  The LDAPCP (see Part 4) utility makes this easy as it will do this for us automatically when configured to search AD.

Requirement 1 could be a problem when using Shibboleth as the authentication provider.  Our Shibboleth deployment does not authenticate against AD, so a Shibboleth ticket will not contain AD LDAP “tokenGroup” data in the “role” claim.  I am working with the Shibboleth guys to see if there is any way to augment Shibboleth tokens with data pulled from AD.

Part 4:  The SharePoint PeoplePicker and ADFS

Experienced SharePoint users all know (and mostly love) the “people picker” that searches Active Directory to validate user and group names that are to be added to the access list for a SharePoint site.  One of the core problems with federation services is that they are authentication systems only.  ADFS and Shibboleth do not implement a directory service.  You cannot do a lookup on an ADFS principal that a user adds to a SharePoint site.  This is particularly irksome, since all of our ADFS users actually have matching accounts in Active Directory.

Fortunately, there is a solution… you can add a “Custom Claims Provider” into SharePoint which will augment incoming ADFS claims with matching user data pulled from Active Directory.  This provider also integrates with the PeoplePicker to allow querying of AD to validate Claims users that are being added to a SharePoint site.  A good write-up can be found here:
http://blogs.msdn.com/b/kaevans/archive/2013/05/26/fixing-people-picker-for-saml-claims-users-using-ldap.aspx

“But I don’t want to compile a SharePoint solution using Visual Studio,” I hear you (and me) whine.  No problem… there is a very good pre-build solution available from CodePlex:
http://ldapcp.codeplex.com/discussions/361947

Normally I do not like using third-part add-ons in SharePoint.  I will make an exception for LDAPCP because:

  1. It works.
  2. It saves me hours of Visual Studio work.
  3. It is a very popular project and thus likely to survive on CodePlex until it is no longer needed.
  4. If the project dies, we can implement our own Claims Provider using templates provided elsewhere with (hopefully) minimal fuss.

My only outstanding problem with LDAPCP is that it will not query principals in our Guest AD forest.  However, there are some suggestions from the developer along these lines:
http://ldapcp.codeplex.com/discussions/478092

To summarize, the developer recommends compiling our own version of LDAPCP from the provided source code.   We would use the method “SetLDAPConnections” found in the “LDAPCP_Custom.cs” source file to add an additional LDAP query source to the solution.  I will try this as time permits.

Part 5: Transforming Shibboleth tokens to ADFS

So far we have not strayed too far from well-trodden paths on the Internet.  Now we get to the fun part… configuring ADFS as a relying party to our Shibboleth Idp, then transforming the incoming Shibboleth SAML token into an ADFS token that can be consumed by SharePoint.

Microsoft published a rather useful guide on ADFS/Shibboleth/InCommon integration:
http://technet.microsoft.com/en-us/library/gg317734(WS.10).aspx
Using this guide we were able to set up ADFS as a relying party to our existing Shibboleth Idp with minimal fuss.  Since we already have an Idp, we skipped most of Step 1, and then jumped to Step 4 as we did not need to configure Shibboleth as an SP to ADFS.

I had the local “Shibboleth Guy” add our ADFS server to the relying parties configuration file on the Shib server, and release “uvm-common” attributes for this provider. This allows SharePoint/ADFS users to get their “eduPersonPrincipalName” (ePPN) released to ADFS/SharePoint from Shibboleth. However, SharePoint (and ADFS) do not natively understand this attribute, so we configure a “Claim Rule” on the “Claims Provider Trust” with Shibboleth.  The rule is an “Acceptance Transformation Rule” that we title “Transform ePPN to UPN”, and it has the following syntax:

c:[Type == "urn:oid:1.3.6.1.4.1.5923.1.1.1.6"]
=> issue(Type = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn",
Issuer = c.Issuer, OriginalIssuer = c.OriginalIssuer, Value = c.Value,
ValueType = c.ValueType);

The “urn:oid:1.3.6.1.4.1.5923.1.1.1.6″ bit is the SAML 2 identifier for the ePPN type.

After configuring a basic Relying Party trust with SharePoint 2013, I need to configure Claims Rules that will release Shibboleth User attributes/claims to SharePoint.  You could use a simple “passthough” rule for this.  However, I want incoming Shibboleth tokens that have a “@uvm.edu” UPN suffix to be treated as though they are Active Directory users.  To accomplish this, I need to do a claims transformation. In AD, the user UPN has the “@campus.ad.uvm.edu” suffix, so let’s transform the Shibboleth UPN using a Claim Rule on the SharePoint Relying Party Trust:

c:[Type == "urn:oid:1.3.6.1.4.1.5923.1.1.1.6", Value =~ "@uvm\.edu$"]
=> issue(Type = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn", Issuer = "AD AUTHORITY", OriginalIssuer = c.OriginalIssuer, Value = regexreplace(c.Value, "^(?[^@]+)@(.+)$", "${user}@campus.ad.uvm.edu"), ValueType = c.ValueType);

NOTE: Code munged by WordPress. Contact me if you need exact syntax!

This appears to work… the first RegEx “@uvm\.edu$” should match an incoming UPN that ends with “@uvm.edu”. In the second set of regExps, we create a capture group for the user portion of the UPN (which is everything from the start of the value up to (but not including) the “@” character), and place the captured data into a variable called “user”. We then replace everything trailing the user portion with “@campus.ad.uvm.edu”.

However, as noted above, the incoming Shibboleth SAML token does not contain AD group data in the “Role” attribute, so users authenticating from Shibboleth cannot get access to sites where they have been granted access using AD groups.  Not good! Fortunately, there is a solution.

An only-hinted at, and certainly not well documented capability of the Claim Rule Language is the ability to create an issuable token with claims originating from more than one identity store. In our case, we need to supplement the incoming Shibboleth SAML token with “roles” claims obtained from Active Directory. I do this during the release of the ADFS token to the SharePoint relying party, using the following rule:

c:[Type == "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn", Value =~ "^.+@campus.ad.uvm.edu$"]
=> issue(store = "Active Directory", types = ("http://schemas.microsoft.com/ws/2008/06/identity/claims/role"), query = ";tokenGroups;CAMPUS\{0}", param = regexreplace(c.Value, "^(?.+)@campus.ad.uvm.edu$", "${user}"));

NOTE: code munged by WordPress. Contact me if you need exact syntax!

The issuance part of this rule bears some discussion. We use the “query” string to collect tokenGroups (or recursive group memberships). The query is run against the Active Directory ADFS attribute store (hence store = “Active Directory”). The query must take the format:
QUERY = [QUERY_FILTER];[ATTRIBUTES];[DOMAIN_NAME]\[USERNAME]
In our case, there is no filter specified. According to TechNet, the default filter is “samAccountName={0}”, which really is what we want. We want to query a particular samAccountName for its tokenGroups. The DOMAIN\Username portion needs to be present, but the username portion is ignored. We could have used “jimmyJoeJimBob” instead of “{0}”.

The {0} represents the value collected in the “param” section of the issuance rule. I used a regular expression replacement rule to strip out the samAccountName from the UPN. Of course, I could simply have used the UPN in the query filter, which would have been cleaner. Maybe I will update this for a future project.

I had to solicit help from the Windows Higher Education list to figure this one out. There is limited documentation on the “query” portion of this rule available here. It is the only documentation I can find, and it does document the bizarre LDAP/AD query syntax discussed above:
http://technet.microsoft.com/en-us/library/adfs2-help-attribute-stores%28WS.10%29.aspx

…And some more general docs on the Claim Rule Language here:
http://social.technet.microsoft.com/wiki/contents/articles/4792.understanding-claim-rule-language-in-ad-fs-2-0-higher.aspx?CommentPosted=true#commentmessage

…And an interesting use-case breakdown of the Language here:
https://jorgequestforknowledge.wordpress.com/category/active-directory-federation-services-adfs/claims-rule-language/

…And a video from Microsoft’s “Identity Architects” here:
http://www.youtube.com/watch?v=G279c_5tHfs

Part 6:  Where are you from?  Notes on Home Realm Discovery

When an ADFS server has multiple “Claims Provider Trusts” defined, the ADFS login page automatically will create a “WAYF”, or “Where are you from?” page to allow the user to select from multiple authentication providers.  In our case, the user would see “Active Directory” and “UVM Shibboleth”.  Since I would not want to confuse people with unnecessary choices, we can disable the display of one of these choices using PowerShell:

Set-AdfsRelyingPartyTrust -TargetName "SharePoint 2013" -ClaimsProviderName "UVM Shibboleth"

In this sample, “SharePoint 2013″ is the name of the relying party defined in ADFS for which you want to set WAYF options.  “UVM Shibboleth” is the Claims Provider Trust that you want used.  This value can be provided as an array, but in this case we are going to provide only one value… the one authenticator that we want to use.  After configuring this change, ADFS logins coming from SharePoint get sent straight to Shibboleth for authentication.

Part 7: The Exciting Results

Only a sysadmin could call this exciting…

Given how heavily MS invested in implementing WS-Federation and WS-Trust into their products (MS Office support for federation services was, to the best of my knowledge, focused entirely on the WS-* protocols implemented in ADFS 1.0), I was not expecting any client beyond a web browser to work with Shibboleth.  Imagine my surprise…

Browsers:

IE 11 and Chrome both login using Shib with no problems.  Firefox works, but not without a glitch… upon being redirected back to SharePoint from our webauth page, we get a page full of un-interpreted html code.  Pressing “f5″ to refresh clears the problem.

Office 2013 Clients:

All core Office 2013 applications appear to support opening of SharePoint documents from links in the browser.  Interestingly, it appears that Office is able to share ADFS tokens obtained by Internet Explorer, and vice versa.  The ADFS token outlives the browser session, too, so you actually have to log off of ADFS prevent token re-use.  I tested the “Export to Excel” and “Add to Outlook” options in the SharePoint ribbon, and both worked without a fuss.

Getting Office apps to open content in SharePoint directly also works, although its a bit buggy.  Sometimes our webauth login dialog does not clear cleanly after authentication.

SkyDrive Pro (the desktop version included with Office 2013) (soon to be “OneDrive for Business) also works with Shibboleth login, amazingly.  The app-store version does not work with on-premises solutions at all, so I could not test it.

Mobile Clients:

I was able to access a OneNote Notebook that I stored in SharePoint using OneNote for Android.  However, it was not easy.  OneNote for Android does not have a dialog that allow for the adding of notebooks from arbitrary network locations.  I first had to add the notebook from a copy of OneNote 2013 for Windows that was linked to my Microsoft account.  The MS account then recorded the existence of the notebook.  When I logged in to OneNote on the Android, it picked up on the SharePoint-backed notebook and I was able to connect.

The OneNote “metro” app does not appear to have the same capability as the Android app.  I cannot get it to connect to anything other than Office 365 or CIFS-backed files.

I was unable to test Office for iOS or Android because I do not own a device on which those apps are supported.

I still need to look at the “Office Document Connection” that comes with Office for the Mac, and at WebDAV clients, and perhaps some other third-party SharePoint apps to see if they work.

App-V Server Configuration, Load Balanced Configuration

In my last post I discussed issues around Kerberos configuration for an App-V 5 server cluster in a load balanced configuration.  Today I will discuss subsequent configuration requirements for making App-V publishing function in a load-balanced environment.

After configuring standalone app-V servers with Management and Publishing Server roles, I had good success with adding packages to the environment and publishing them.  However, when switching to a load balanced configuration, I experienced a failure of the publishing server to pick up on changes in the management configuration.  Helpful resources and troubleshooting notes follow:

  • A TechNet social page that I referenced in my previous post makes reference to this same problem:
    http://social.technet.microsoft.com/Forums/en-US/2b39e2b8-aba1-4e96-b18f-c5bcb9f12687/load-balancing-two-appv-50-servers-the-publishing-service-is-not-able-to-contact-the-management
    But does not point me towards any solutions.  This seems like some sort of permissions problem, so I put Sysinternals Procmon on watching w3wp.exe for “Access Denied” events, but I get nothing.  However, I do see a fair amount of database traffic at IIS startup time.
  • The following TechNet blog provided a key tipoff in App-V server diagnostics:
    http://blogs.technet.com/b/appv/archive/2013/01/23/how-to-collect-app-v-5-0-debug-traces.aspx
      The trick was to select “Show Analytic and Debug Logs” under “Actions” in the event viewer.  With this option enabled, I now see App-V management and publishing debug logs instead of just the default App-V event logs.  The debug logs contain the real error.  We see that the SID recorded for the publishing server in the management server database does not match the SID of the account making the connection!  What we needed to do was delete the publishing server entries from the management configuration, and create one new “server” under the name of the publishing server service account, not the computer account.  I just updated the SQL database entry manually, but I likely could have just used the Silverlight UI instead.  This change cleared up the mismatched SID error, but now we get an “access denied” error to the publishing metadata directory.
  • The following blog gives an excellent technical overview of App-V server infrastructure and the general troubleshooting process for resolving configuration issues:
    http://kirxblog.wordpress.com/2012/05/11/app-v-5-beta-basic-troubleshooting/

    • Here it is suggested that I look at HKLM/Software/Microsoft/AppV/Server to review the management and publishing server configurations.  Sure enough, one problem seen here is that the publishing server is configured to connect to the management server on an http:// address.  However, I updated the management servers to use https://.  I modified those registry values and restarted IIS.  Still no luck… 
    • This blog explains how published applications are read out of a metadata xml file that is exposed to the publishing server by the management server.  Both are stored in c:\programdata\microsoft\appv.  When running Procmon.exe against w3wp.exe we see “Access Denied” to these directories by our service account.  After adding “modify” rights for the service account to these directories, metadata updates again start to happen.

It is unsurprising that switching form the use of a local server account to an AD service account caused access problems for the App-V server.  The difficulty of discovering where account info and rights needed to be updated was a bit of a surprise.  But thanks to the blog-o-sphere and the mighty “procmon.exe”, we have our answers.

Now on to performance testing…

App-V 5 Server, F5 Load Balancers, and Kerberos

More fun today with Kerberos and load balancers.  Today’s challenge related to getting the Microsoft App-V publishing server to work with an F5 load balancer in a Layer 4/n-Path/DSR configuration.  Everything was working when I was accessing the individual server nodes, but when I switched to using the load balanced name and address, authentication started to fail.

After lots of log searching I eventually tried a wire trace, and found the following Kerberos error in the response from the App-V server to the App-V client:
KRB5KRB_AP_ERR_MODIFIED

Lots of different resources helped here:

  • This TechNet page explains various Kerberos errors and why they might occur:
    http://blogs.technet.com/b/askds/archive/2008/06/11/kerberos-authentication-problems-service-principal-name-spn-issues-part-3.aspx

    Of note is the scenario where the account handling the authentication request does not hold the SPN for which the request was made.  I set the SPN for my IIS application pool identity, but further analysis of the error packet shows that it was handled by my App-V server machine account, not the service account.  Augh!  Why?

  • This thread on TechNet Social was the biggest help:

    http://social.technet.microsoft.com/Forums/en-US/2b39e2b8-aba1-4e96-b18f-c5bcb9f12687/load-balancing-two-appv-50-servers-the-publishing-service-is-not-able-to-contact-the-management

    The user posted all of the steps they followed in configuring IIS and the service account SPN, including the tidbit:
    changed the authentication of the “Management Service” web site to useAppPoolCredentials=”true”
    I have never used this particular setting, so I dug into it…

  • The following MSDN article explains the IIS 7.0 feature of “kernel authentication”, how it affects the need for SPN entries, and its interplay with application pool identity accounts:
    http://blogs.msdn.com/b/webtopics/archive/2009/01/19/service-principal-name-spn-checklist-for-kerberos-authentication-with-iis-7-0.aspx

    Basically, with kernel-mode authentication, the SYSTEM account will handle all Kerberos authentication by default.  This explains why we were seeing Kerberos errors in the communications with the App-V client… the IIS pool identity account was not handling Kerberos delegation!

    Of special interest is this statement:
    Either:
    Disable Kernel mode authentication and follow the general steps for Kerberos as in the previous IIS 6.0 version.
    Or,
    [Recommended for Performance reasons]
    Let Kernel mode authentication be enabled and the Application pool’s identity be used for Kerberos ticket decryption. The only thing you need to do here is:
    1. Run the Application pool under a common custom domain account.
    2. Add this attribute “useAppPoolCredentials” in the ApplicationHost.config file.

  • This TechNet page documents how to configure Kerberos auth in IIS, and mentions the use of the IIS appcmd.exe to set the “useAppPoolCredentials” option:
    http://technet.microsoft.com/en-us/library/dd759186.aspx
    Included is the exact command line required to set the value to true:
     appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -useAppPoolCredentials:true
    (But the page does not really tell you what it is for, which is where the MSDN article comes in handy.)

So, Kerberos under IIS 7 and later has some nuances not present in IIS 6.  I wonder how I did not encounter this before?

SQL Reporting Services – Load Balancing Notes

As part of our evaluation of a SQL Consolidation Server, I am attempting a deployment of SQL Reporting Services in a load-balanced environment (using F5 load balancers in layer 4/fast npath config). I have followed my usual procedure for setting up a npath/direct server return config on Windows Server (with a couple of caveats… on Server 2012 the name of the MS Loopback Adapter has been changed to ‘Microsoft KM-TEST Loopback Adapter’). I then installed Reporting Services on both load balanced servers and connected them to the same back end database, and configured them to use the same SSL certificate.

Problems arise when attempting to connect to the load-balanced Reporting Services URL over SSL. Initial connection succeeds, but authentication fails. Log trawling reveals the following event:

Log Name:    System
Source:      LSA (LsaSrv)
Event ID:    6037
Level:       Warning

Detail:
The program ReportingServicesService.exe, with the assigned process ID 13432, could not authenticate locally by using the target name HTTP/myserver.mydomain.edu. The target name used is not valid. A target name should refer to one of the local computer names, for example, the DNS host name.

Try a different target name.

I found a similar error discussed at the following site:

http://sharepoint400.blogspot.com/2011/02/program-w3wpexe-with-assigned-process.html

The solution there worked for me, too:

  1. Go to REGEDIT and open the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
  2. Right click MSV1_0 –> New -> Multi-String Value
  3. Type BackConnectionHostNames and click Enter.
  4. Right click on newly created value and select Modify.
  5. Enter the hostname of the site: WEBSITENAME (and on a new line enter the FQDN, WEBSITENAME.domain.com)
  6. Restart IIS

I also had considered enabling Kerberos authentication for Reporting Services, but I have decided to forgo this step (since, I think, we will want users to be able to retrieve reports from non-domain-joined computers).  However, if we did want to enable Kerberos, three things need to happen:

  1. Use SETSPN to add the HTTP/[reportServerHostName] [domain]\[reportServerServiceAccount]
  2. Configure the Reporting Services Service Account to be trusted for Kerberos Delegation (using AD Users and Computers, under the “Delegation” tab)
  3. Modify the rsreportserver.config file (in %ProgramFiles%\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\ReportServer).  Find the “AuthenticationTypes” section, and specify <RSWindowsNegotiate/> as the first/only entry.
  4. Restart Reporting Services

F5 Load Balancing – Performance (Layer 4) mode and Windows Server 2008

NOTE: This article has been updated with a correction to the NetSH commands. Previously I documented the “forwarding” should be enabled on the interfaces, but “weak host receive” and “weak host send” is more accurate, as documented here:

http://devcentral.f5.com/weblogs/rkorock/Default.aspx

Recently we had a problem with a web applicaiton configured for SSL-offload on our Load Balancers.  Our F5 Guru (Ben Coddington) recommended that we swich to a “Layer 4 forwarding” configuration.  In this mode, the F5 will forward TCP packets from the client directly to the web server without altering packet content, which is just what we needed.

Making this work on Server 2008 took a bit of extra leg work, though.  Here are the bones of it:

  • On the F5, create a new Virtual Server using the Type category “Performance (Layer 4)”.  Make sure that address translation and port translation are disabled.
  • Create a new F5 Pool that uses a simple port 443/ssl health monitor.  You could use any of a number of load balancing methods, but I cose “Round Robin” because it is in keeping with the “simpler is better” school of thought.
  • On the Server 2008 system, add a “loopback adapter” in the Device Manager.  (At the root of the MMC console, right-click the computer and select “Add legacy device”.  It will be of type “network adapter”, from manfacturer “Microsoft”, and have a name containing “loopback adapter”).
  • Assign the load balanced IP to the loopback adpater with netmask “255.255.255.255″.
  • Here is the trick… you must now allow “weak host receive” on all network interfaces involved with load balancing on the Server 2008 system, and “weak host send” on the loopback interface. If this step is skipped, the Windows server will drop all packets destined for the load balancer address:
    • netsh
      interface ipv4
      set interface "Loopback Connection" weakhostreceive=enable
      set interface "Public Network" weakhostreceive=enabled
      set interface "Loopback Connection" weakhostsend=enabled
      exit
      
  • Make sure you have a vaild SSL certificate configured on all RDGateway systems in your farm.

That’s about it… The F5 will forward all packets sent to the load balanced IP to the next pool member in the rotation (barring persistence).  The Server 2008 host will receive the packet, and forward it to the loopback adapter (following TCP/IP routing logic).  The Server 2008 host will reply directly to the client.  Amazingly, it all seems to work.

Improving the Stability of ECTS

ECTS, the external collaboration toolkit for sharepoint, has been a bit of  a thorn in my side.  While fulfilling a vital need at our institution, it is difficult to support, and contains many bugs that apparently are not going to get fixed.

However, I recently I managed to pull off a few minor tweaks for ECTS that should improve stability in our environment.

Fix 1:  ECTSEx

First off, I installed the EXTS Extensions available in Codeplex:

http://ectsex.codeplex.com/

My thanks to David McWee for providing these accelerators to the ECTS management interfaces.  We are now able to pull up info on ECTS users considerably more easily.  These extensions also allow password reset information to go directly to the user for whom they are intended.  Good stuff.  Unfortunately, the interface is not behaving on all nodes of our SharePoint web farm, which brings us to our next fix…

Fix 2:  Load Balancer reconfiguration

We have been experiencing intermittent problems which suggests duplication of actions by the ECTS processes.  For example, a user may receive two account activation notices, each with different passwords.  A user may change a password, only to have the new password expired immediately, followed by prompting that your password has been reset by an administrator.

My assumption is that somehow requests to either the web front end servers of the backing AD LDS instance are being replicated, resulting in replicated account management operations.  To combat this, I have reconfigured our F5 load balancers as follows:

  1. For the Big-IP Virtual Server which handles traffic to “PartnerPoint.uvm.edu” (our ECTS front-end), I set the Persistence Profile to “none”.  This prevent users from continuing to use a LB node if a more preferred node is online.
  2. For the Big-IP pool connected to PartnerPoint, we activate “Priority Group Activation” with the “Less than 1″ attribute.  This will cause clients to be routed to a different Pool member priority group if there are less than one servers remaining in the highest priority group.
  3. We then set the “sharepoint3″ node to be in priority group 2, and “sharepoint2″ to priority group 1.  Now when users connect to the virtual server, they are routed to priority group 2 (sharepoint3), and only go to sharepoint2 if sharepoint3 is down.
  4. Repeat these three steps for the AD LDS virtual server.  This limits LDAP traffic to sharepoint3, unless sharepoint 3 is down.
  5. Now we need to limit traffic to the ECTS admin interfaces to connect preferentially against sharepoint3 as well.  We do this with iRules.  Create a new iRule which will intercept traffic to the ECTS management pages, and route it to a different pool:
    1
    2
    3
    4
    5
    
    when HTTP_REQUEST {
     if {[HTTP::uri] starts_with "/ECTS/"} {
     pool WSS_partner_admin
     }
    }
  6. We then create a Pool with priority group activation, as outlined in steps 1-3, above, and assign this iRule to the main SharePoint virtual server.

I am able to verify though packet captures that all requests to the ECTS admin pages, and all traffic to the ECTS front ends is being routed preferentially to “sharepoint3″.  So far, this seems to have helped.  If nothing else, the admin pages are now loading consistently, which was not true for us in the past.

Sharepoint – farm build procedure

After a semi-disaster with SharePoint earlier this week, I have been forced into the view that I really should have our SharePoint infrastructure hosted on more than one web server.  To that end, I am planning the deployment of a new, 2+ node Windows SharePoint Services farm.

Initial architecture will be something like this:

  • Host: SharePoint2
    • Roles:  Web front end, Search Server query and crawl, ECTS ADAM Instance
  • Host: SharePoint3
    • Roles:  Web front end, Search Server query and index, ECTS ADAM Instance
  • Hosts: WinDB1 and WinDB2
    • Roles:  Back-end SQL Database failover cluster
  • F5 Big-IP Local Traffic Manager (hardware load balancer)

Once initial rollout is complete, we likely will want to add:

  • Hosts: WinDB1 and WinDB2
    • Reconfigured in a SQL mirrored configuration

Here is an outline of the SharePoint2/3 build procedure:

  1. Install Server 2008 x64 Standard OS
    1. Activate Roles:  IIS (with ASP.NET support), AD Lightweight Directory Services (AKA AD LDS, AKA ADAM).
    2. Activate Features:  .Net Framework 3.0, PowerShell, SMTP Server
    3. Activate Feature “Desktop Experience” if you want access to “cleanmgr.exe” (Disk Cleanup wizard) and other desktop niceties.
    4. To save disk space, you might want to delete the hiberfile.sys (Hibernation file), by running “powercfg.exe /hibernate off”.
      1. Relocate the page file?
      2. Do something about WinSXS directory bloat (likely impossible without a service pack)
      3. Get IIS log files under control!
  2. Configure SMTP Server using IIS 6.0 Manager.  (FTP and SMTP Services cannot be managed with the IIS 7 Manager!)
    1. Configure BadMail and Drop Box directories so that they are not located on the System volume
    2. Configure to listen only on primary server IP
    3. Configure firewall to accept SMTP conections only from our In-Mail Gateways (using external firewall in our case, but this could be done with SMTP service settings and/or the Server 2008 firewall as well). <-DON’T FORGET – SMTP is allowed to the hosts, but need to go over the config with our DNS/mail admins!
  3. Install Search Server Express x64 bits:
    1. Perform “complete” install (Search Server will not install a SQL 2005 instance, as is the case with WSS installer).  Under “file location”, specify “E:\Office\12.0\Data” as the index storage location.
    2. Skip running of the Configuration Wizard after install.
  4. Install SharePoint Administration Kit v2.0
    1. Exclude Profile replicator component as it will not work on WSS
  5. Clone the server as many times as deemed necessary. (At present, make one clone!).  Any cloned systems must be sysprep-ed before joining the domain.  Once preped, join the computers, configure networking.
  6. If planning to add this server to a load balanced cluster, install NLB feature:
    NOTE:  We will not be doing this as we plan to use F5 hardware LBs instead.

    • from “administrator” cmd shell, run “ocsetup NetworkLoadBalancingFullServer”
    • Don’t join to a production NLB cluster until SharePoint configuration is complete!
  7. Replicate AD LDS (ADAM) instance to new machine, if required.
    (We need this as we are extending an existing ADAM instance to the new farm)

    1. In Server Manager, Click on “AD Lightweight Directory Services” Role,
    2. Click “AD LDS Setup Wizard”
      1. Select “A replica of an existing instance”
      2. Name the instance “ECTSInstance”
      3. Accept standard LDAP ports
      4. specify a partnerpoint server to replicate from, use standard LDAP ports.
      5. Select the “OU=ects,…” partition set for replication (this should be the only partition!)
      6. Select secondary (non-system) volume as target for AD LDS data… generally this will be “E:\Microsoft ADAM\ECTSInstance\data”
      7. Specify domain service account to run the AD LDS instance.
      8. Add “domain admins” to the AD LDS Administrators list.  Finish the wizard.
    3. Run the campus…bat file located in e:\Microsoft ADAM\ECTSInstance\data\.  This will register the Kerberos Service Principal Names required for LDP replication mutual authentication.
    4. Open the “Local Security Policy” Admin tool.  Add the domain service account to the “generate security audits” User Rights Assignment branch.
    5. Open the AD Users and Computers tool, locate the computer object on which you installed the Instance.  Give the LDS service account “create all child objects” to the computer object.
    6. Add the cluster load balanced SSL cert into the Personal certificate store of the ECTSInstance service account.
      1. Request wildcard certificate using the procedure outlined here:
        http://erlend.oftedal.no/blog/?blogid=7
        (We use the web interface for requesting a certificate, make user we use the RSA SChannel crypto provider to generate the request, use the “SHA-1” hash, use PKCS10 format, and use the “UVM – Web Server” request template.  For load-balanced LDAP servers, we must request a wildcard certificate (*.uvm.edu)
        NOTE: This step will not have to be repeated again until the current cert expires.  To add another AD LDS server, export the cert from a current server, import into the new server
      2. Export the request cert to file selecting “export all extended attributes” and “export private key” options.
      3. Import the cert into the “Personal” branch of the service account’s certificate store on the target server.  Make sure that you import “all extended attributes”, and the private key.  Do not select the use of advanced encryption password.
      4. Restart AD LDS and test SSL connections.
      5. If all is not working (as is the case with one of my two servers), here is where we get into undocumented territory.  Here are some helpful resources for debugging:
        1. I set SChannel diag logging to verbose :
          • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel
            REG_DWORD EventLogging, value 0×7
          • Restart ECTSInstance, look for “SChannel” entries in the server “application” even logs.  These logs will tell you which certificate the system attempted to use, and why access failed.
        2. You may need to add the wildcard cert to the Local Computer Certificate Store as well… run MMC, add the “Certificates” snap-in for “Service Account”, using the “ECTS Instance” service.  Navigate to the “Personal” branch, run an import action, import the wildcard with all extended attributes and the private key.
        3. Now locate the physical copy of this cert in c:\programdata\microsoft\crypto\RSA\MachineKeys (it will be the file with the most recently modified time stamp).  Add “read/execute” permissions to this file for the AD LDS service account, then restart the LDS instance.
    7. Force mutual authentication for replication traffic:
      1. Run ADSI Edit
      2. “Connect to”, enter the AD LDS server name in the Computer field, select the “Configration” well-known naming context.  As documented in http://technet.microsoft.com/en-us/library/cc794841.aspx, get “properties” on the “CN=Configuration…” partition, and change the value of “msDSReplAuthenticationMode” to “2”.
    8. Set local password policy – this controls password policy of AD LDS accounts:
      1. Add the Sharepoint server computer account to the “ETS – SharePoint Password Policy” GP Object.  After running “gpupdate /target:computer /force”, verify the settings by doing the following:
        1. Open Local Security Policy control panel
        2. Expand “Account Policies”->”Password Policy”
        3. Settings applied should follow the 1/365/0/8/Disabled/Disabled format.  (we may want to revisit this policy later).
  8. Run the SharePoint Products and Technologies Configuration Wizard:
    1. Connect to an existing Farm
    2. Enter “WINBD” as the database server.  The wizard will correctly select “SharePoint_FarmConfig” as the configuration database.  The correct service account username will be provided… you need to enter the password.
    3. Click “Advanced Settings”, specify that you which the server to host the Central Admnistration site.
      1. If setup fails with the error:
        “SharePoint Configuration Wizard failed with an exception “Error during encryption or decryption. System error code 997″
        A solution can be found here:
        http://blogs.msdn.com/priyo/archive/2007/08/11/add-new-sharepoint-server-to-existing-server-farm-an-unhandled-exception-occurred-in-the-user-interface-exception-information-unable-to-connect-to-the-remote-server.aspx
        Essentially we just run “stsadm –o updatefarmcredentials –userlogin “domain\service_acount” –password <thePassword>” on the first SharePoint server, then re-run the wizard.
    4. Update the “Central Admin” shortcut to point to the local Central Admin site by doing the following registry hack:
      http://blogs.technet.com/wbaer/archive/2007/08/30/sharepoint-3-0-central-administration-url-on-multiple-web-front-end-servers.aspx
      Essentially, edit the key:
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\WSS
      Then locate CentralAdministrationURL and change it to point to the local server.
  9. Configure Search Service:
    1. When Search is run in an environment where SharePoint services are accessed from a FQDN which is different from the physical host name (i.e. our environment, or any other environment with load balancers), you will need to work around the “loopback security check” feature of Windows.  Failing to do so will result in “access denied” errors in the crawl logs.  My thanks to Shawn Feldman for discovering this:
      http://blogs.msdn.com/fledman/archive/2008/09/18/access-denied-with-windows-server-2008-and-moss-when-crawling.aspx
      The relevant work-around is documented here (see “Method 2”):
      http://support.microsoft.com/kb/896861
      We simply need to add the public FQDN of our SharePoint server to:
      Key: HKLM\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
      Value: REG_MULTI_SZ, sharepoint.uvm.edu
      And then restart the IISAdmin service.
    2. Open the search admin page from SharePoint Central Administration:
      1. Access Crawling –> Content Sources
        1. Click the “Local Office SharePoint Server sites” default source.
        2. Define a crawling schedule for the SharePoint application
        3. Click “new content source” to add any additional content sources that are desired (i.e. our production file servers).
        4. Define additional crawl schedules for these new content sources.
        5. Set up “Crawl Rules” to exclude any directories from the content sources that you do not wish to have indexed.  In our environment, it was essential to exclude the “~snapshot” directories from the root of our NetApp file shares.
    3. Add Search Center to our SharePoint landing page:
      1. Create a new sub-site of type “Enterprise Search”
      2. Tune federated results by adding/removing web parts to the results page
    4. Add Federated Locations for additional search results:
      Ideally we would like to add results from out “GoogleWeb.uvm.edu” search appliance, and perhaps a Google “site” search for other close University partners

      1. For GSA (Google Search Appliance) Federation, it appears we will need to setup an RSS Transform:
        1. http://enterprise-code-samples.googlecode.com/svn/trunk/rss-stylesheet/Readme.htm
          http://enterprise-code-samples.googlecode.com/svn/trunk/rss-stylesheet/
        2. As the readme above recommends, we set up a new “front end” on the GSA for RSS results.  We called this Front End “RSS”.  We edit the “base_url” value to point to http://googleweb.uvm.edu/Search?. We then visit this URL and perform a query.  We take the resulting query results URL:
          http://googleweb.uvm.edu/search?q={searchTerms}&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&lr=&client=uvm2008&ud=1&oe=UTF-8&ie=UTF-8&proxystylesheet=uvm2008&site=default_collection
          and we substitute the name of our Front End in the place of the “proxystylesheet” and “client” values.  I also simplify the query just a bit as follows:
          http://googleweb.uvm.edu/search?q={searchTerms}&output=xml_no_dtd&lr=&client=rss&ud=1&oe=UTF-8&ie=UTF-8&proxystylesheet=rss&site=default_collection
        3. We then use the RSS query URL above as the template URL for a federated search query in the SharePoint Search Center.
        4. We edit the search results page to include federated results, and specify the “UVM GoogleWeb Search” federated location in the “Location” of the Federated results web part (using the “Modify Shared Web Part” command).
    5. Integrate Windows Search desktop tool with Search Server:<- Not yet done, but this can wait until after the production cut-over.

      1. Approve Windows Desktop Search 3.0->Windows Search 4.0 update on WSUS server
      2. Deploy Group Policy templates for Windows Search 4
      3. Configure:
        http://technet.microsoft.com/en-us/library/cc732491.aspx#BKMK_Addprimaryintranetsearch
  10. Install Infrastructure Update for WSS3 x64:
    1. Initiate the update on the first node in the cluster.
    2. When prompted, start the install on the second cluster node.
    3. When the configuration wizard completes on the second node, go back to the first and allow configuration to complete.
  11. Install Infrastructure Update for Search Server x64:
    1. Initiate the update on the first node in the cluster.
    2. When prompted, start the install on the second cluster node.
    3. When the configuration wizard completes on the second node, go back to the first and allow configuration to complete.
  12. Clean up IIS settings for the newly created Web Sites – configure binding, authentication and SSL  (Note that these procedures are only accurate when using Windows-native load balancers… when we transition to f5 load balancing, it will not be necessary to return custom errors from IIS as the f5 will handle HTTP-to-HTTPS redirections.) :
    1. SSL Cert Installation:
      Install SSL certs into “Personal” Store of the Computer account using the “Certificates” MMC snapin.
    2. Binding:
      Open the IIS Manager MMC snapin.  On each site, right-click and select “edit bindings”:

      1. For site “SharePoint – 443” (which represent the traditional “sharepoint.uvm.edu” URL), bind https and http protocols to port 80 and port 443, using the IP address for “sharepoint.uvm.edu” (132.198.102.12).  When binding SSL, select the appropriate cert from the “SSL Certificate” drop down menu.
      2. For “SharePoint – Internet” (which represents SharePointLite), bind https and http, ports 443 and 80, to “sharepointlite.uvm.edu”, IP 132.198.102.36.  Again, select the correct SSL cert for this site.
      3. For “SharePoint – Extranet” (which represents PartnerPoint), bind https and http, ports 443 and 80, to “partnerpoint.uvm.edu”, IP 132.198.102.49, selecting the matching SSL cert once again.
    3. SSL Configuration:
      1. In IIS Manager, open the “features view” for each site.
      2. Double-click “SSL Settings”
      3. Check “Require SSL”, leaving the default “ignore Client certificates” setting.
      4. Now double-click the “Error Pages” item for the server root.  Add a custom error for 403.4 (SSL required), pointing to our custom “redirect.html” javascript file.  We will need to have copied this file into “c:\inetpub\custerr\en-US\” before completing this step
      5. Now find the applicationHost.config file for the IIS server.  This should be located in “C:\Windows\system32\inetsrv\config”.  Locate the section for each site that serves SharePoint content (i.e. <location path=”SharePoint – 443”>), then locate the <httpErrors> tag under <system.webServer>.  In the httpErrors tag, change the value for “existingResponse” from “PassThrough” to “Replace” (response “Auto” also seems to work, but may produce inconsistent results).  This will prevent ASP.NET from replacing the 403.4 error response from the IIS server.  I am much indebted to this forum thread for this breakthrough:
        http://forums.iis.net/t/1113734.aspx
        Also helpful was the new “failed request tracing” module in IIS7:
        http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis7/
        More information on the meaning of the various existingResponse values can be found here:
        http://blogs.iis.net/ksingla/archive/2008/02/18/what-to-expect-from-iis7-custom-error-module.aspx
  13. Install the MS FilterPack 1.0 (Search Server can already index most Office 2007 documents, but this adds ability to index inside of One Note files and ZIP archives):
    1. Follow instructions at:
      http://support.microsoft.com/?id=946336
  14. Install Adobe iFilter, with 64-bit “thunking” DCOM service:
  15. Install MindManager extensions.
    1. DEPRECATED – We will discontinue this extension with the new upgrade as it does not work with MM v7 or v8
  16. Install ECTS components on each web front end server.
    • Having problems with installation script… what if we try the ECTS update available though CodePlex???
      1. If using updated ECTS files, it will be necessary to update the PartnerAdmin and PartnerConfig pages, as the self-service Site Collection Manager.  The existing pages will not work because the GUIDs on the Web Parts have changed.
      2. The ects_setup_sharepoint.vbs script still fails using the updated code… Since the codeplex team has not documented their changes, I think we will skip this option.
    • Troubleshooting issues:
      1. The ects_setup_sharepoint.vbs script succeeds in installing the ECTS solution, but fails when activating site features.  I suspect that “cscript” on Server 2008 is not processing return codes from stsadm.exe correctly, and this is reporting failure to install features (I am not positive about the reason for the script failure, although it certainly is not a result of stsadm.exe being broken.
        I was able to work around this problem by opening the ects_setup_sharepoint.vbs file in a text editor, searching for the error string that was sent to the console when the script failed, then running all of the operations in the script manually from that point forward.  Fortunately, all of the stsadm commands in the script are successful when run from the command line.
      2. ECTS is not compatible with MS Load Balancing out of the box.  I switched to a F5 load balancer before working through the problem.  It is possible that the problem I was having could have been fixed with the same “loopback security check” that caused problems during our F5 configuration
        http://support.microsoft.com/kb/896861
        In fact, we may have had the problem even with the f5 in place, but I would not know because I applied the loopback fix before implementing the F5.
        The error codes suggest that a login failure is occurring between the IIS application and the AD LDS LDAP instance.  When I try to connect to the load-balanced LDAP DNS name using the “LDP.exe” LDAP client, I also get an authentication error.  However, when I connect to the local server address, authentication works.
      3. As was the case when I first installed ECTS, the web.config files required a bit of hand-tuning to get services working correctly:
        http://www.uvm.edu/~jgm/wordpress/?p=112
        Once again, I had to modify the “ADAMConnectionString” in the web.config of each IIS site to reflect the actual DNS name of the load-balanced AD LDS servers.  I had installed ECTS using a different name initially, and the ECTS un-installation script did not clear out these values.
      4. I did find it necessary to deactivate all ECTS site collection features, re-activate them, then perform an IIS reset before my existing ECTS management pages would work again.  This seems pretty par for the course when removing and re-installing SharePoint solutions.
      5. After connecting the production content database and changing the URL for the server, I also needed to edit “LDAPHost” value to reflect the production name of the server in the “TEMPLATE\FEATURES\ECTSBase\Feature.xml” file (located in the “12″ hive).  This value was set at the time that ECTS was deployed to the server.  I expect that the same (wrong) LDAPHost value would get replicated to additional ECTS servers if we were to add new ones, because this setting is contained in the RESX package that gets built by the ECTS installer, and cached in the database for future deployment.  I expect I would need to do a full retract/remove/redeploy to fix the problem permanently.
  17. Install Globally-deployable solutions from the “fab 40” application template.  If you deploy a web front end into an existing farm, the files required by these features will get transferred automatically.  However, when building a new farm, we need to install them manually.  Currently required “server admin” templates are:
    • ApplicationTemplateCore
    • ChangeRequest
    • ContactsManagement
    • DocumentLibraryReview
    • EventPlanning
    • HelpDesk
    • InventoryTracking
    • ITTeamWorkspace
    • Knowledgebase
    • LendingLibrary
    • PhysicalAssetTracking
    • ProjectTrackingWorkspace
    • RoomEquipmentReservations
    • Procedure:
      • stsadm -o addsolution -filename <file_path>\<template_name>.wsp
      • stsadm -o deploysolution -name <template_name>.wsp –allowgacdeployment
  18. Install radEditor:
    1. Install ASP.NET Ajax for .NET 2.0, version 1.0
    2. Follow the Ajax configuration for SharePoint configuration guide found here:
      http://sharepoint.microsoft.com/blogs/mike/Lists/Posts/Post.aspx?ID=3
    3. Install radEditor using the included instructions.
    4. Copy radEditor configuration files from an existing production server to the new server:
      1. In the directory:
        ”C:\Program Files\Common Files\Microsoft Shared\web server extensions\wpresources\RadEditorSharePoint\[versionString]\RadControls\Editor”
        Backup the existing ListConfigFile.xml, ConfigFile.xml, ListToolsFile.xml, and ToolsFile.xml files.  Replace with versions customized for UVM.  Note that the MOSS LinkManager tool does not work in WSS.  Also note that when editing list content that does not support “Enhanced Content”, the first toolbar in the ListToolsFile.xml will be removed… in past versions, the toolbar named “enhancedTools” was removed.
      2. Copy the files ListConfigFile.xml, ConfigFile.xml, ListToolsFile.xml, and ToolsFile.xml to all other nodes in the cluster.
      3. perform an IISRESET.
      4. Update ONET.xml files in the “12” hive to activate the radEditor feature by default in all new sites (see ONET.xml template files on the prod web front end for examples).
        NOTE that the “RadEditor for non-IE browsers” and “RadEditor for IE” features have been collapsed into one unified feature.  Update the ONET.XML files accordingly!  (note that the feature ID for the main RadEditor List editor has not changed… only it’s name is different.  We did not have to insert a new default feature ID, but we did need to remove the “RadEditor for IE” feature because it is no longer present in RadEditor MOSS.)
      5. Run:
        stsadm –o uninstallfeature –name RadEditorFeatureRichHtml.
        This “Web Content Management” feature is not supported in WSS, so we may as well remove it to avoid confusion.
      6. Deactivate and then re-activate the radEditor features on at least one existing site, and test functionality.
  19. Install “Smiling Goat” Feed Reader (RSS/ATOM subscriber web part)
    1. This will require Feed Reader users to update their web parts!
    2. Current release here:
      http://www.codeplex.com/FeedReader/Release/ProjectReleases.aspx?ReleaseId=19830
      (version 3.1.0.1 at the time of this writing)
  20. Install SharePoint Training Kit:
  21. Tune web application settings to match production server:
    1. In “Operations”:
      1. Configure blocked file types – in our case we have been requested to all “MAT” files.  MAT is one of may obscure Office file extensions, but also is used by MatLab, a common mathematical/statistical application used on campus.
      2. Enable Usage Analysis Processing – relocate log files off of the system volume.
        1. You MUST grant read/write access to the local WSS_WPG to the folder that will hold the usage analysis logs.  If you fail to do so, you will see lots of errors in the Diagnostic Logs allong the lines of “Cannot create folder “<big ugly hexidecimal number>””.
      3. Configure incoming and outgoing email settings.
        1. Incoming mail requires SMTP to be installed on the server – managed with the old-school IIS 6.0 Manager MMC.
        2. We needed to use the “Advanced Settings” to specify the SMTP drop folder in the Incoming Mail settings page of Central Administration.  SharePoint diagnostics logs indicated that the timer service could not determine the drop folder location automatically.
        3. After configuring the mail drop folder, SharePoint still was unable to process messages.  “Access denied” events started queuing up in the event viewer.  A quick analysis using SysInternals ProcMon shows that the WSS Access account (being used by the OWSTIMER.EXE process) has no ability to access the drop folder defined above.  I just added “modify” access for that account to the drop folder, and incoming mail started processing immediately.
    2. Under “Application Management”:
      1. Tune the “Web application general settings”:
        1. Set upload limits for files
        2. Set time zone
        3. Set quota templates for new sites
      2. Config “site use confirmation and deletion”. <- Be sure to do this after PROD cutover!!!  Can’t do it ahead of time as users will start to get email from the pre-prod server!
      3. Enable self-service site creation. <- Also needs to be done after prod cut-over
      4. Configure “policy for web application” to allow selected groups of SharePoint administrators rights to all sites in the farm.
    3. Edit the footer of the “welcome” email message starting at line 5219 of “core.en-US.resx” in the 12-hive “resources” folder.  Replicate on all web front ends in the farm.
  22. Configure f5 load balancers:
    1. Lots of IP addresses required:
      1. “floating ip” for network on which F5s will communicate with the SharePoint web servers.
      2. 2x Self-IPs for the physical interfaces on the F5s which will handle the floating IP
      3. 3x IP addresses for the public SharePoint URLs (standard, lite, partner)
      4. 6x IP addresses for the private interfaces on each web server node (for standard, lite, and partner sites, one each on two web servers)
    2. Need to generate SSL Certificate/Key pair files for the load balancer “SSL Profiles” (the F5 uses separate files for certificates and keys, as opposed to IIS which uses a single CER file – we need to split the IIS files and upload to the F5.
      1. Using the certificate manager MMC snapin, export the production certs to PFX (PKCS #12) files (make sure to specify that you wish to export the private key, otherwise PKCS #12 is not an option in the wizard).
      2. Convert the pfx file into a PEM using the openssl executables (available for Windows here: http://www.slproweb.com/products/Win32OpenSSL.html) using the folowing command:
        openssl.exe pkcs12 -in [infile].pfx -out [outfile].pem -nodes
        (You will be prompted to enter the password for the PFX, if it has one.  The “-nodes” flag indicates that the output file will not be encrypted with a password)
      3. Open the PEM file created above with a text editor.  Remove the  private-key portion of the file (all content from “Bag Attributes” to “—–END RSA PRIVATE KEY—–”) and paste this into a separate “KEY” file (UNIX-style line endings, ANSI-encoded).
      4. Upload the new PEM and KEY files into your F5 load balancer to create a new SSL client profile.
    3. Set up two virtual servers for each SharePoint URL – one to redirect HTTP to HTTPS, one to handle regular traffic.
      1. Add the following iRule to perform the SSL redirect, and attach it to the Virtual Server listening on port 80:
        (This rule will redirect your browser to https://sharepoint.uvm.edu from either http://sharepoint or http://sharepoint.uvm.edu… well, actually it will redirect any non-HTTPS connection to a HTTPS version of itself, appending our domain “.uvm.edu” if it is missing.)

           1: when HTTP_REQUEST {
           2:   if { [HTTP::host] == "*.uvm.edu" } {
           3:     HTTP::redirect https://[HTTP::host][HTTP::uri]
           4:   } else {
           5:     HTTP::redirect https://[HTTP::host].uvm.edu[HTTP::uri]
           6:   }
           7: }
  23. TEST TEST TEST:
    1. Test each feature on both web front ends by alternately disabling the nodes in the load balancer configuration.
    2. Test again with both nodes enabled… watch for authentication and session persistence issues.
    3. Test all features in each access mapping – SP, SPLite, and Partner… web.config file variations could cause problems!
  24. Consider Deployment of “Group Board 2007” and “Sample Master Pages”:

And to complete the cutover:

  1. Redirect SharePoint traffic to a “maintenance underway” page, or just put the service in read-only mode.
  2. Backup the current production content database
  3. Detach the content database from the production server.
  4. Attach the content database to the new servers
  5. Change the SSL profiles used on the F5 load balancers to use the production certs.
  6. Take down the original SharePoint IIS server and disable.
  7. Change the listening port on the F5 load balancers to match the production server IPs.
  8. Configure Zone security and Alternative Access Mappings for the Farm to reflect the new production names:
    1. Make sure that https://sharepoint.uvm.edu, https://sharepointlite.uvm.edu, and https://partnerpoint.uvm.edu are all listed as “Public URLs” for the farm.
    2. Make sure that the HTTP equivalents of these URLS are listed as INTERNAL URLS in the AAM table if you are performing SSL termination on the load balancers.
  9. Contact SAA Mail admins to update the mail server LDAP records for SharePoint mail routing
  10. Dredge though this post for things that will need to be configured after the migration, and test mercilessly.