Posts Tagged ‘IIS’

App-V 5 Server, F5 Load Balancers, and Kerberos

More fun today with Kerberos and load balancers.  Today’s challenge related to getting the Microsoft App-V publishing server to work with an F5 load balancer in a Layer 4/n-Path/DSR configuration.  Everything was working when I was accessing the individual server nodes, but when I switched to using the load balanced name and address, authentication started to fail.

After lots of log searching I eventually tried a wire trace, and found the following Kerberos error in the response from the App-V server to the App-V client:
KRB5KRB_AP_ERR_MODIFIED

Lots of different resources helped here:

  • This TechNet page explains various Kerberos errors and why they might occur:
    http://blogs.technet.com/b/askds/archive/2008/06/11/kerberos-authentication-problems-service-principal-name-spn-issues-part-3.aspx

    Of note is the scenario where the account handling the authentication request does not hold the SPN for which the request was made.  I set the SPN for my IIS application pool identity, but further analysis of the error packet shows that it was handled by my App-V server machine account, not the service account.  Augh!  Why?

  • This thread on TechNet Social was the biggest help:

    http://social.technet.microsoft.com/Forums/en-US/2b39e2b8-aba1-4e96-b18f-c5bcb9f12687/load-balancing-two-appv-50-servers-the-publishing-service-is-not-able-to-contact-the-management

    The user posted all of the steps they followed in configuring IIS and the service account SPN, including the tidbit:
    changed the authentication of the “Management Service” web site to useAppPoolCredentials=”true”
    I have never used this particular setting, so I dug into it…

  • The following MSDN article explains the IIS 7.0 feature of “kernel authentication”, how it affects the need for SPN entries, and its interplay with application pool identity accounts:
    http://blogs.msdn.com/b/webtopics/archive/2009/01/19/service-principal-name-spn-checklist-for-kerberos-authentication-with-iis-7-0.aspx

    Basically, with kernel-mode authentication, the SYSTEM account will handle all Kerberos authentication by default.  This explains why we were seeing Kerberos errors in the communications with the App-V client… the IIS pool identity account was not handling Kerberos delegation!

    Of special interest is this statement:
    Either:
    Disable Kernel mode authentication and follow the general steps for Kerberos as in the previous IIS 6.0 version.
    Or,
    [Recommended for Performance reasons]
    Let Kernel mode authentication be enabled and the Application pool’s identity be used for Kerberos ticket decryption. The only thing you need to do here is:
    1. Run the Application pool under a common custom domain account.
    2. Add this attribute “useAppPoolCredentials” in the ApplicationHost.config file.

  • This TechNet page documents how to configure Kerberos auth in IIS, and mentions the use of the IIS appcmd.exe to set the “useAppPoolCredentials” option:
    http://technet.microsoft.com/en-us/library/dd759186.aspx
    Included is the exact command line required to set the value to true:
     appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -useAppPoolCredentials:true
    (But the page does not really tell you what it is for, which is where the MSDN article comes in handy.)

So, Kerberos under IIS 7 and later has some nuances not present in IIS 6.  I wonder how I did not encounter this before?

ApplicationXtender – 5.30 to 5.40SP1 upgrade planning

Previously I documented a rough outline of the AX 5.30 Infrastructure installation process:

http://www.uvm.edu/~jgm/wordpress/?p=71

With support for 5.30 expiring today, I think it high time we got our infrastructure up to date up to the most current version that is supported for use with SunGard Banner.

  1. Uninstall all previously existing AX components.  Purge any residual files from the IIS  publishing directories, “Program Files”, “Application Data”, and the registry.
  2. Set security for the global impersonation account according to the table on page 210 of the “concepts and planning guide”.
    1. Note that the account does not have to be a local administrator!
    2. However, the security accounts will have to have privileges to the resources accessed by the services (i.e. NTFS filesystems rights, shared folder access).
    3. Rendering Service -
      1. When granting rights to the DX data store, plan ahead.  Permissions could take a long time to apply.
      2. Requires Local Security Policy “Replace a Process Level Token” and “Adjust memory quotas for a process” rights.  Also, the “Allow service to interact with the desktop” box must be deselected in the “Log On” tab of the Rendering service properties.
    4. WebAccess.NET Services -
      1. Global Account needs only “Log on as a service” Local Security Policy assignment.  You can clear out all “legacy” security permissions as they are not needed for WebAccess!
  3. Install AX Desktop, installing all administration tools:
    1. msiexec /i “ApplicationXtender Desktop.msi” /qb ADDLOCAL=DocumentManager,AppGen,ConfigurationTools,ManagementTools
  4. Install the new License Server and install license file:
    1. Install the “ApplicationXtender License Server.msi” (FlexNet License Manager)
      1. Drop the .LIC license file into C:\Program Files\XtenderSolutions\Content Management\License Server
      2. Configure the Login identity of the “ApplicationXtender License Client Components” COM+ application to use the global impersonation account.  This component must be shut down to be reconfigured.  Details in EMC PowerLink solution esg92864.
      3. Restart the “ApplicationXtender License Service” Service.
    2. Install the “EMC License Server” (Proprietary License Server, to support DiskXtender)
      1. Install all current patches to the service
      2. Run the “License Server Administrator” GUI.
      3. Go to “Tools”, then “New License Wizard” to install the DiskXtender License.
  5. Install DiskXtender
    1. Install DiskXtender patches, in sequence
      1. When prompted for the DX service account, you must provide an account that has local “administrator” rights, and the ability to “log on as a service”.
    2. Verify and/or re-establish RPC partition maps – See the “Core Components” guide for instructions.
    3. Consider switching to DCOM security model, which will require modifying the “AE_PATHS” table in each data source db.   See page 160 of the “Desktop Install Guide” for details.
      1. This is not actually practical to do since it will break AX Desktop on any system that is not joined to the CAMPUS domain (and why would they not be joined, I wonder?)
  6. Launch AX Admin
    1. See “ApplicationXtender Desktop Installation Guide” for details
    2. Log in as SYSOP and perform the database upgrade, if prompted.
    3. Verify global settings:
      1. Add license server configuration: see “Core Components” guide for details.
      2. Web Access .NET must use Global credentials since we are using and Oracle database with Oracle security.
    4. Save the configuration and exit
  7. Launch AppGen, and verify functionality.
    1. Connect to each defined data source, one at a time.
    2. Perform database upgrades if prompted (this should be safe, but can take several minutes to complete).
  8. Set IIS web site root to use ASP.NET 2.0.
  9. Install AX Web Services, making sure to install the required “Utility Services” component.  “AX Web Services” and “Workflow” components are optional.
    1. See “AppXtender Core Components Admin Guide” for installation and config details.
    2. Choose IIS installation option, and install into “Default Web Site” (which should be the only site present)
    3. Ensure that “Default.aspx” is listed as an accepted default page for the “AppXtender” IIS web application.
  10. Install AX Web Access .NET
  11. Install AX Rending Server
  12. Run the Component Setup Wizard for all installed components
  13. Outside of my control:
    1. BannerXtender updates need to be applied to production Banner systems.
    2. DocSend and ECopy stations need to be upgraded to 5.40 AX Desktop releases
    3. DocAccel server needs 5.40 AX Desktop upgrade
    4. All AX desktop clients need updates, too.
    5. Anyone using WX WebAccess.NET ActiveX controls will need to upgrade these components.
  14. Test Test Test!


HTTP to HTTPS redirect using Iconic URL Rewriter

So, you have a site that needs to to run over SSL-only (shouldn’t they all?)? You don’t trust your clients to type that ever-important “s” after “http” (and why would they?)? You think they will get scared off by those “Secure connection required” error pages (they will!)? You are not running IIS7 (who is?)? Not using ASP.NET?

In the past we accomplished this using a client-side redirect, by creating a custom 404.3 error page with a Javascript redirect. This worked well, but what if you client systems won’t support javascript (i.e. it is a webdav connection)?

Codplex to the rescue! The venerable “Ionic URL Rewrite” ISAPI filter has been updated, and published on Codeplex:

http://www.codeplex.com/IIRF

Thanks, Cheeso!

IIRF now supports the ability to return URL redirects, in addition to simple rewrites.  To use IIRF to redirect a non-SSL URL to a secure version, follow the installation instructions included with IIRF.  Then:

  • Stop your production IIS site from listening on port 80 and enforce SSL usage.
  • Make sure that the production site is not using host headers that would override your port settings.
  • Set up a secondary IIS site which listens on port 80 only.  Add the IIRF ISAPI filter to this site.

Here is some sample entires you could use in the IsapiRewrite4.ini configration file to accomplish the redirect.  Note that [R] instead of [R=301] also works, but this performs a 302 “Temporary” redirect.  Conceptually I prefer a 301 (not that it matters because search crawlers are not hitting our Intranet sites):

Read More

Sharepoint – farm build procedure

After a semi-disaster with SharePoint earlier this week, I have been forced into the view that I really should have our SharePoint infrastructure hosted on more than one web server.  To that end, I am planning the deployment of a new, 2+ node Windows SharePoint Services farm.

Initial architecture will be something like this:

  • Host: SharePoint2
    • Roles:  Web front end, Search Server query and crawl, ECTS ADAM Instance
  • Host: SharePoint3
    • Roles:  Web front end, Search Server query and index, ECTS ADAM Instance
  • Hosts: WinDB1 and WinDB2
    • Roles:  Back-end SQL Database failover cluster
  • F5 Big-IP Local Traffic Manager (hardware load balancer)

Once initial rollout is complete, we likely will want to add:

  • Hosts: WinDB1 and WinDB2
    • Reconfigured in a SQL mirrored configuration

Here is an outline of the SharePoint2/3 build procedure:

  1. Install Server 2008 x64 Standard OS
    1. Activate Roles:  IIS (with ASP.NET support), AD Lightweight Directory Services (AKA AD LDS, AKA ADAM).
    2. Activate Features:  .Net Framework 3.0, PowerShell, SMTP Server
    3. Activate Feature “Desktop Experience” if you want access to “cleanmgr.exe” (Disk Cleanup wizard) and other desktop niceties.
    4. To save disk space, you might want to delete the hiberfile.sys (Hibernation file), by running “powercfg.exe /hibernate off”.
      1. Relocate the page file?
      2. Do something about WinSXS directory bloat (likely impossible without a service pack)
      3. Get IIS log files under control!
  2. Configure SMTP Server using IIS 6.0 Manager.  (FTP and SMTP Services cannot be managed with the IIS 7 Manager!)
    1. Configure BadMail and Drop Box directories so that they are not located on the System volume
    2. Configure to listen only on primary server IP
    3. Configure firewall to accept SMTP conections only from our In-Mail Gateways (using external firewall in our case, but this could be done with SMTP service settings and/or the Server 2008 firewall as well). <-DON’T FORGET – SMTP is allowed to the hosts, but need to go over the config with our DNS/mail admins!
  3. Install Search Server Express x64 bits:
    1. Perform “complete” install (Search Server will not install a SQL 2005 instance, as is the case with WSS installer).  Under “file location”, specify “E:\Office\12.0\Data” as the index storage location.
    2. Skip running of the Configuration Wizard after install.
  4. Install SharePoint Administration Kit v2.0
    1. Exclude Profile replicator component as it will not work on WSS
  5. Clone the server as many times as deemed necessary. (At present, make one clone!).  Any cloned systems must be sysprep-ed before joining the domain.  Once preped, join the computers, configure networking.
  6. If planning to add this server to a load balanced cluster, install NLB feature:
    NOTE:  We will not be doing this as we plan to use F5 hardware LBs instead.

    • from “administrator” cmd shell, run “ocsetup NetworkLoadBalancingFullServer”
    • Don’t join to a production NLB cluster until SharePoint configuration is complete!
  7. Replicate AD LDS (ADAM) instance to new machine, if required.
    (We need this as we are extending an existing ADAM instance to the new farm)

    1. In Server Manager, Click on “AD Lightweight Directory Services” Role,
    2. Click “AD LDS Setup Wizard”
      1. Select “A replica of an existing instance”
      2. Name the instance “ECTSInstance”
      3. Accept standard LDAP ports
      4. specify a partnerpoint server to replicate from, use standard LDAP ports.
      5. Select the “OU=ects,…” partition set for replication (this should be the only partition!)
      6. Select secondary (non-system) volume as target for AD LDS data… generally this will be “E:\Microsoft ADAM\ECTSInstance\data”
      7. Specify domain service account to run the AD LDS instance.
      8. Add “domain admins” to the AD LDS Administrators list.  Finish the wizard.
    3. Run the campus…bat file located in e:\Microsoft ADAM\ECTSInstance\data\.  This will register the Kerberos Service Principal Names required for LDP replication mutual authentication.
    4. Open the “Local Security Policy” Admin tool.  Add the domain service account to the “generate security audits” User Rights Assignment branch.
    5. Open the AD Users and Computers tool, locate the computer object on which you installed the Instance.  Give the LDS service account “create all child objects” to the computer object.
    6. Add the cluster load balanced SSL cert into the Personal certificate store of the ECTSInstance service account.
      1. Request wildcard certificate using the procedure outlined here:
        http://erlend.oftedal.no/blog/?blogid=7
        (We use the web interface for requesting a certificate, make user we use the RSA SChannel crypto provider to generate the request, use the “SHA-1” hash, use PKCS10 format, and use the “UVM – Web Server” request template.  For load-balanced LDAP servers, we must request a wildcard certificate (*.uvm.edu)
        NOTE: This step will not have to be repeated again until the current cert expires.  To add another AD LDS server, export the cert from a current server, import into the new server
      2. Export the request cert to file selecting “export all extended attributes” and “export private key” options.
      3. Import the cert into the “Personal” branch of the service account’s certificate store on the target server.  Make sure that you import “all extended attributes”, and the private key.  Do not select the use of advanced encryption password.
      4. Restart AD LDS and test SSL connections.
      5. If all is not working (as is the case with one of my two servers), here is where we get into undocumented territory.  Here are some helpful resources for debugging:
        1. I set SChannel diag logging to verbose :
          • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel
            REG_DWORD EventLogging, value 0×7
          • Restart ECTSInstance, look for “SChannel” entries in the server “application” even logs.  These logs will tell you which certificate the system attempted to use, and why access failed.
        2. You may need to add the wildcard cert to the Local Computer Certificate Store as well… run MMC, add the “Certificates” snap-in for “Service Account”, using the “ECTS Instance” service.  Navigate to the “Personal” branch, run an import action, import the wildcard with all extended attributes and the private key.
        3. Now locate the physical copy of this cert in c:\programdata\microsoft\crypto\RSA\MachineKeys (it will be the file with the most recently modified time stamp).  Add “read/execute” permissions to this file for the AD LDS service account, then restart the LDS instance.
    7. Force mutual authentication for replication traffic:
      1. Run ADSI Edit
      2. “Connect to”, enter the AD LDS server name in the Computer field, select the “Configration” well-known naming context.  As documented in http://technet.microsoft.com/en-us/library/cc794841.aspx, get “properties” on the “CN=Configuration…” partition, and change the value of “msDSReplAuthenticationMode” to “2”.
    8. Set local password policy – this controls password policy of AD LDS accounts:
      1. Add the Sharepoint server computer account to the “ETS – SharePoint Password Policy” GP Object.  After running “gpupdate /target:computer /force”, verify the settings by doing the following:
        1. Open Local Security Policy control panel
        2. Expand “Account Policies”->”Password Policy”
        3. Settings applied should follow the 1/365/0/8/Disabled/Disabled format.  (we may want to revisit this policy later).
  8. Run the SharePoint Products and Technologies Configuration Wizard:
    1. Connect to an existing Farm
    2. Enter “WINBD” as the database server.  The wizard will correctly select “SharePoint_FarmConfig” as the configuration database.  The correct service account username will be provided… you need to enter the password.
    3. Click “Advanced Settings”, specify that you which the server to host the Central Admnistration site.
      1. If setup fails with the error:
        “SharePoint Configuration Wizard failed with an exception “Error during encryption or decryption. System error code 997″
        A solution can be found here:
        http://blogs.msdn.com/priyo/archive/2007/08/11/add-new-sharepoint-server-to-existing-server-farm-an-unhandled-exception-occurred-in-the-user-interface-exception-information-unable-to-connect-to-the-remote-server.aspx
        Essentially we just run “stsadm –o updatefarmcredentials –userlogin “domain\service_acount” –password <thePassword>” on the first SharePoint server, then re-run the wizard.
    4. Update the “Central Admin” shortcut to point to the local Central Admin site by doing the following registry hack:
      http://blogs.technet.com/wbaer/archive/2007/08/30/sharepoint-3-0-central-administration-url-on-multiple-web-front-end-servers.aspx
      Essentially, edit the key:
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\WSS
      Then locate CentralAdministrationURL and change it to point to the local server.
  9. Configure Search Service:
    1. When Search is run in an environment where SharePoint services are accessed from a FQDN which is different from the physical host name (i.e. our environment, or any other environment with load balancers), you will need to work around the “loopback security check” feature of Windows.  Failing to do so will result in “access denied” errors in the crawl logs.  My thanks to Shawn Feldman for discovering this:
      http://blogs.msdn.com/fledman/archive/2008/09/18/access-denied-with-windows-server-2008-and-moss-when-crawling.aspx
      The relevant work-around is documented here (see “Method 2”):
      http://support.microsoft.com/kb/896861
      We simply need to add the public FQDN of our SharePoint server to:
      Key: HKLM\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
      Value: REG_MULTI_SZ, sharepoint.uvm.edu
      And then restart the IISAdmin service.
    2. Open the search admin page from SharePoint Central Administration:
      1. Access Crawling –> Content Sources
        1. Click the “Local Office SharePoint Server sites” default source.
        2. Define a crawling schedule for the SharePoint application
        3. Click “new content source” to add any additional content sources that are desired (i.e. our production file servers).
        4. Define additional crawl schedules for these new content sources.
        5. Set up “Crawl Rules” to exclude any directories from the content sources that you do not wish to have indexed.  In our environment, it was essential to exclude the “~snapshot” directories from the root of our NetApp file shares.
    3. Add Search Center to our SharePoint landing page:
      1. Create a new sub-site of type “Enterprise Search”
      2. Tune federated results by adding/removing web parts to the results page
    4. Add Federated Locations for additional search results:
      Ideally we would like to add results from out “GoogleWeb.uvm.edu” search appliance, and perhaps a Google “site” search for other close University partners

      1. For GSA (Google Search Appliance) Federation, it appears we will need to setup an RSS Transform:
        1. http://enterprise-code-samples.googlecode.com/svn/trunk/rss-stylesheet/Readme.htm
          http://enterprise-code-samples.googlecode.com/svn/trunk/rss-stylesheet/
        2. As the readme above recommends, we set up a new “front end” on the GSA for RSS results.  We called this Front End “RSS”.  We edit the “base_url” value to point to http://googleweb.uvm.edu/Search?. We then visit this URL and perform a query.  We take the resulting query results URL:
          http://googleweb.uvm.edu/search?q={searchTerms}&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&lr=&client=uvm2008&ud=1&oe=UTF-8&ie=UTF-8&proxystylesheet=uvm2008&site=default_collection
          and we substitute the name of our Front End in the place of the “proxystylesheet” and “client” values.  I also simplify the query just a bit as follows:
          http://googleweb.uvm.edu/search?q={searchTerms}&output=xml_no_dtd&lr=&client=rss&ud=1&oe=UTF-8&ie=UTF-8&proxystylesheet=rss&site=default_collection
        3. We then use the RSS query URL above as the template URL for a federated search query in the SharePoint Search Center.
        4. We edit the search results page to include federated results, and specify the “UVM GoogleWeb Search” federated location in the “Location” of the Federated results web part (using the “Modify Shared Web Part” command).
    5. Integrate Windows Search desktop tool with Search Server:<- Not yet done, but this can wait until after the production cut-over.

      1. Approve Windows Desktop Search 3.0->Windows Search 4.0 update on WSUS server
      2. Deploy Group Policy templates for Windows Search 4
      3. Configure:
        http://technet.microsoft.com/en-us/library/cc732491.aspx#BKMK_Addprimaryintranetsearch
  10. Install Infrastructure Update for WSS3 x64:
    1. Initiate the update on the first node in the cluster.
    2. When prompted, start the install on the second cluster node.
    3. When the configuration wizard completes on the second node, go back to the first and allow configuration to complete.
  11. Install Infrastructure Update for Search Server x64:
    1. Initiate the update on the first node in the cluster.
    2. When prompted, start the install on the second cluster node.
    3. When the configuration wizard completes on the second node, go back to the first and allow configuration to complete.
  12. Clean up IIS settings for the newly created Web Sites – configure binding, authentication and SSL  (Note that these procedures are only accurate when using Windows-native load balancers… when we transition to f5 load balancing, it will not be necessary to return custom errors from IIS as the f5 will handle HTTP-to-HTTPS redirections.) :
    1. SSL Cert Installation:
      Install SSL certs into “Personal” Store of the Computer account using the “Certificates” MMC snapin.
    2. Binding:
      Open the IIS Manager MMC snapin.  On each site, right-click and select “edit bindings”:

      1. For site “SharePoint – 443” (which represent the traditional “sharepoint.uvm.edu” URL), bind https and http protocols to port 80 and port 443, using the IP address for “sharepoint.uvm.edu” (132.198.102.12).  When binding SSL, select the appropriate cert from the “SSL Certificate” drop down menu.
      2. For “SharePoint – Internet” (which represents SharePointLite), bind https and http, ports 443 and 80, to “sharepointlite.uvm.edu”, IP 132.198.102.36.  Again, select the correct SSL cert for this site.
      3. For “SharePoint – Extranet” (which represents PartnerPoint), bind https and http, ports 443 and 80, to “partnerpoint.uvm.edu”, IP 132.198.102.49, selecting the matching SSL cert once again.
    3. SSL Configuration:
      1. In IIS Manager, open the “features view” for each site.
      2. Double-click “SSL Settings”
      3. Check “Require SSL”, leaving the default “ignore Client certificates” setting.
      4. Now double-click the “Error Pages” item for the server root.  Add a custom error for 403.4 (SSL required), pointing to our custom “redirect.html” javascript file.  We will need to have copied this file into “c:\inetpub\custerr\en-US\” before completing this step
      5. Now find the applicationHost.config file for the IIS server.  This should be located in “C:\Windows\system32\inetsrv\config”.  Locate the section for each site that serves SharePoint content (i.e. <location path=”SharePoint – 443”>), then locate the <httpErrors> tag under <system.webServer>.  In the httpErrors tag, change the value for “existingResponse” from “PassThrough” to “Replace” (response “Auto” also seems to work, but may produce inconsistent results).  This will prevent ASP.NET from replacing the 403.4 error response from the IIS server.  I am much indebted to this forum thread for this breakthrough:
        http://forums.iis.net/t/1113734.aspx
        Also helpful was the new “failed request tracing” module in IIS7:
        http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis7/
        More information on the meaning of the various existingResponse values can be found here:
        http://blogs.iis.net/ksingla/archive/2008/02/18/what-to-expect-from-iis7-custom-error-module.aspx
  13. Install the MS FilterPack 1.0 (Search Server can already index most Office 2007 documents, but this adds ability to index inside of One Note files and ZIP archives):
    1. Follow instructions at:
      http://support.microsoft.com/?id=946336
  14. Install Adobe iFilter, with 64-bit “thunking” DCOM service:
  15. Install MindManager extensions.
    1. DEPRECATED – We will discontinue this extension with the new upgrade as it does not work with MM v7 or v8
  16. Install ECTS components on each web front end server.
    • Having problems with installation script… what if we try the ECTS update available though CodePlex???
      1. If using updated ECTS files, it will be necessary to update the PartnerAdmin and PartnerConfig pages, as the self-service Site Collection Manager.  The existing pages will not work because the GUIDs on the Web Parts have changed.
      2. The ects_setup_sharepoint.vbs script still fails using the updated code… Since the codeplex team has not documented their changes, I think we will skip this option.
    • Troubleshooting issues:
      1. The ects_setup_sharepoint.vbs script succeeds in installing the ECTS solution, but fails when activating site features.  I suspect that “cscript” on Server 2008 is not processing return codes from stsadm.exe correctly, and this is reporting failure to install features (I am not positive about the reason for the script failure, although it certainly is not a result of stsadm.exe being broken.
        I was able to work around this problem by opening the ects_setup_sharepoint.vbs file in a text editor, searching for the error string that was sent to the console when the script failed, then running all of the operations in the script manually from that point forward.  Fortunately, all of the stsadm commands in the script are successful when run from the command line.
      2. ECTS is not compatible with MS Load Balancing out of the box.  I switched to a F5 load balancer before working through the problem.  It is possible that the problem I was having could have been fixed with the same “loopback security check” that caused problems during our F5 configuration
        http://support.microsoft.com/kb/896861
        In fact, we may have had the problem even with the f5 in place, but I would not know because I applied the loopback fix before implementing the F5.
        The error codes suggest that a login failure is occurring between the IIS application and the AD LDS LDAP instance.  When I try to connect to the load-balanced LDAP DNS name using the “LDP.exe” LDAP client, I also get an authentication error.  However, when I connect to the local server address, authentication works.
      3. As was the case when I first installed ECTS, the web.config files required a bit of hand-tuning to get services working correctly:
        http://www.uvm.edu/~jgm/wordpress/?p=112
        Once again, I had to modify the “ADAMConnectionString” in the web.config of each IIS site to reflect the actual DNS name of the load-balanced AD LDS servers.  I had installed ECTS using a different name initially, and the ECTS un-installation script did not clear out these values.
      4. I did find it necessary to deactivate all ECTS site collection features, re-activate them, then perform an IIS reset before my existing ECTS management pages would work again.  This seems pretty par for the course when removing and re-installing SharePoint solutions.
      5. After connecting the production content database and changing the URL for the server, I also needed to edit “LDAPHost” value to reflect the production name of the server in the “TEMPLATE\FEATURES\ECTSBase\Feature.xml” file (located in the “12″ hive).  This value was set at the time that ECTS was deployed to the server.  I expect that the same (wrong) LDAPHost value would get replicated to additional ECTS servers if we were to add new ones, because this setting is contained in the RESX package that gets built by the ECTS installer, and cached in the database for future deployment.  I expect I would need to do a full retract/remove/redeploy to fix the problem permanently.
  17. Install Globally-deployable solutions from the “fab 40” application template.  If you deploy a web front end into an existing farm, the files required by these features will get transferred automatically.  However, when building a new farm, we need to install them manually.  Currently required “server admin” templates are:
    • ApplicationTemplateCore
    • ChangeRequest
    • ContactsManagement
    • DocumentLibraryReview
    • EventPlanning
    • HelpDesk
    • InventoryTracking
    • ITTeamWorkspace
    • Knowledgebase
    • LendingLibrary
    • PhysicalAssetTracking
    • ProjectTrackingWorkspace
    • RoomEquipmentReservations
    • Procedure:
      • stsadm -o addsolution -filename <file_path>\<template_name>.wsp
      • stsadm -o deploysolution -name <template_name>.wsp –allowgacdeployment
  18. Install radEditor:
    1. Install ASP.NET Ajax for .NET 2.0, version 1.0
    2. Follow the Ajax configuration for SharePoint configuration guide found here:
      http://sharepoint.microsoft.com/blogs/mike/Lists/Posts/Post.aspx?ID=3
    3. Install radEditor using the included instructions.
    4. Copy radEditor configuration files from an existing production server to the new server:
      1. In the directory:
        ”C:\Program Files\Common Files\Microsoft Shared\web server extensions\wpresources\RadEditorSharePoint\[versionString]\RadControls\Editor”
        Backup the existing ListConfigFile.xml, ConfigFile.xml, ListToolsFile.xml, and ToolsFile.xml files.  Replace with versions customized for UVM.  Note that the MOSS LinkManager tool does not work in WSS.  Also note that when editing list content that does not support “Enhanced Content”, the first toolbar in the ListToolsFile.xml will be removed… in past versions, the toolbar named “enhancedTools” was removed.
      2. Copy the files ListConfigFile.xml, ConfigFile.xml, ListToolsFile.xml, and ToolsFile.xml to all other nodes in the cluster.
      3. perform an IISRESET.
      4. Update ONET.xml files in the “12” hive to activate the radEditor feature by default in all new sites (see ONET.xml template files on the prod web front end for examples).
        NOTE that the “RadEditor for non-IE browsers” and “RadEditor for IE” features have been collapsed into one unified feature.  Update the ONET.XML files accordingly!  (note that the feature ID for the main RadEditor List editor has not changed… only it’s name is different.  We did not have to insert a new default feature ID, but we did need to remove the “RadEditor for IE” feature because it is no longer present in RadEditor MOSS.)
      5. Run:
        stsadm –o uninstallfeature –name RadEditorFeatureRichHtml.
        This “Web Content Management” feature is not supported in WSS, so we may as well remove it to avoid confusion.
      6. Deactivate and then re-activate the radEditor features on at least one existing site, and test functionality.
  19. Install “Smiling Goat” Feed Reader (RSS/ATOM subscriber web part)
    1. This will require Feed Reader users to update their web parts!
    2. Current release here:
      http://www.codeplex.com/FeedReader/Release/ProjectReleases.aspx?ReleaseId=19830
      (version 3.1.0.1 at the time of this writing)
  20. Install SharePoint Training Kit:
  21. Tune web application settings to match production server:
    1. In “Operations”:
      1. Configure blocked file types – in our case we have been requested to all “MAT” files.  MAT is one of may obscure Office file extensions, but also is used by MatLab, a common mathematical/statistical application used on campus.
      2. Enable Usage Analysis Processing – relocate log files off of the system volume.
        1. You MUST grant read/write access to the local WSS_WPG to the folder that will hold the usage analysis logs.  If you fail to do so, you will see lots of errors in the Diagnostic Logs allong the lines of “Cannot create folder “<big ugly hexidecimal number>””.
      3. Configure incoming and outgoing email settings.
        1. Incoming mail requires SMTP to be installed on the server – managed with the old-school IIS 6.0 Manager MMC.
        2. We needed to use the “Advanced Settings” to specify the SMTP drop folder in the Incoming Mail settings page of Central Administration.  SharePoint diagnostics logs indicated that the timer service could not determine the drop folder location automatically.
        3. After configuring the mail drop folder, SharePoint still was unable to process messages.  “Access denied” events started queuing up in the event viewer.  A quick analysis using SysInternals ProcMon shows that the WSS Access account (being used by the OWSTIMER.EXE process) has no ability to access the drop folder defined above.  I just added “modify” access for that account to the drop folder, and incoming mail started processing immediately.
    2. Under “Application Management”:
      1. Tune the “Web application general settings”:
        1. Set upload limits for files
        2. Set time zone
        3. Set quota templates for new sites
      2. Config “site use confirmation and deletion”. <- Be sure to do this after PROD cutover!!!  Can’t do it ahead of time as users will start to get email from the pre-prod server!
      3. Enable self-service site creation. <- Also needs to be done after prod cut-over
      4. Configure “policy for web application” to allow selected groups of SharePoint administrators rights to all sites in the farm.
    3. Edit the footer of the “welcome” email message starting at line 5219 of “core.en-US.resx” in the 12-hive “resources” folder.  Replicate on all web front ends in the farm.
  22. Configure f5 load balancers:
    1. Lots of IP addresses required:
      1. “floating ip” for network on which F5s will communicate with the SharePoint web servers.
      2. 2x Self-IPs for the physical interfaces on the F5s which will handle the floating IP
      3. 3x IP addresses for the public SharePoint URLs (standard, lite, partner)
      4. 6x IP addresses for the private interfaces on each web server node (for standard, lite, and partner sites, one each on two web servers)
    2. Need to generate SSL Certificate/Key pair files for the load balancer “SSL Profiles” (the F5 uses separate files for certificates and keys, as opposed to IIS which uses a single CER file – we need to split the IIS files and upload to the F5.
      1. Using the certificate manager MMC snapin, export the production certs to PFX (PKCS #12) files (make sure to specify that you wish to export the private key, otherwise PKCS #12 is not an option in the wizard).
      2. Convert the pfx file into a PEM using the openssl executables (available for Windows here: http://www.slproweb.com/products/Win32OpenSSL.html) using the folowing command:
        openssl.exe pkcs12 -in [infile].pfx -out [outfile].pem -nodes
        (You will be prompted to enter the password for the PFX, if it has one.  The “-nodes” flag indicates that the output file will not be encrypted with a password)
      3. Open the PEM file created above with a text editor.  Remove the  private-key portion of the file (all content from “Bag Attributes” to “—–END RSA PRIVATE KEY—–”) and paste this into a separate “KEY” file (UNIX-style line endings, ANSI-encoded).
      4. Upload the new PEM and KEY files into your F5 load balancer to create a new SSL client profile.
    3. Set up two virtual servers for each SharePoint URL – one to redirect HTTP to HTTPS, one to handle regular traffic.
      1. Add the following iRule to perform the SSL redirect, and attach it to the Virtual Server listening on port 80:
        (This rule will redirect your browser to https://sharepoint.uvm.edu from either http://sharepoint or http://sharepoint.uvm.edu… well, actually it will redirect any non-HTTPS connection to a HTTPS version of itself, appending our domain “.uvm.edu” if it is missing.)

           1: when HTTP_REQUEST {
           2:   if { [HTTP::host] == "*.uvm.edu" } {
           3:     HTTP::redirect https://[HTTP::host][HTTP::uri]
           4:   } else {
           5:     HTTP::redirect https://[HTTP::host].uvm.edu[HTTP::uri]
           6:   }
           7: }
  23. TEST TEST TEST:
    1. Test each feature on both web front ends by alternately disabling the nodes in the load balancer configuration.
    2. Test again with both nodes enabled… watch for authentication and session persistence issues.
    3. Test all features in each access mapping – SP, SPLite, and Partner… web.config file variations could cause problems!
  24. Consider Deployment of “Group Board 2007” and “Sample Master Pages”:

And to complete the cutover:

  1. Redirect SharePoint traffic to a “maintenance underway” page, or just put the service in read-only mode.
  2. Backup the current production content database
  3. Detach the content database from the production server.
  4. Attach the content database to the new servers
  5. Change the SSL profiles used on the F5 load balancers to use the production certs.
  6. Take down the original SharePoint IIS server and disable.
  7. Change the listening port on the F5 load balancers to match the production server IPs.
  8. Configure Zone security and Alternative Access Mappings for the Farm to reflect the new production names:
    1. Make sure that https://sharepoint.uvm.edu, https://sharepointlite.uvm.edu, and https://partnerpoint.uvm.edu are all listed as “Public URLs” for the farm.
    2. Make sure that the HTTP equivalents of these URLS are listed as INTERNAL URLS in the AAM table if you are performing SSL termination on the load balancers.
  9. Contact SAA Mail admins to update the mail server LDAP records for SharePoint mail routing
  10. Dredge though this post for things that will need to be configured after the migration, and test mercilessly.