Posts Tagged ‘ECTS’

Improving the Stability of ECTS

ECTS, the external collaboration toolkit for sharepoint, has been a bit of  a thorn in my side.  While fulfilling a vital need at our institution, it is difficult to support, and contains many bugs that apparently are not going to get fixed.

However, I recently I managed to pull off a few minor tweaks for ECTS that should improve stability in our environment.

Fix 1:  ECTSEx

First off, I installed the EXTS Extensions available in Codeplex:

http://ectsex.codeplex.com/

My thanks to David McWee for providing these accelerators to the ECTS management interfaces.  We are now able to pull up info on ECTS users considerably more easily.  These extensions also allow password reset information to go directly to the user for whom they are intended.  Good stuff.  Unfortunately, the interface is not behaving on all nodes of our SharePoint web farm, which brings us to our next fix…

Fix 2:  Load Balancer reconfiguration

We have been experiencing intermittent problems which suggests duplication of actions by the ECTS processes.  For example, a user may receive two account activation notices, each with different passwords.  A user may change a password, only to have the new password expired immediately, followed by prompting that your password has been reset by an administrator.

My assumption is that somehow requests to either the web front end servers of the backing AD LDS instance are being replicated, resulting in replicated account management operations.  To combat this, I have reconfigured our F5 load balancers as follows:

  1. For the Big-IP Virtual Server which handles traffic to “PartnerPoint.uvm.edu” (our ECTS front-end), I set the Persistence Profile to “none”.  This prevent users from continuing to use a LB node if a more preferred node is online.
  2. For the Big-IP pool connected to PartnerPoint, we activate “Priority Group Activation” with the “Less than 1″ attribute.  This will cause clients to be routed to a different Pool member priority group if there are less than one servers remaining in the highest priority group.
  3. We then set the “sharepoint3″ node to be in priority group 2, and “sharepoint2″ to priority group 1.  Now when users connect to the virtual server, they are routed to priority group 2 (sharepoint3), and only go to sharepoint2 if sharepoint3 is down.
  4. Repeat these three steps for the AD LDS virtual server.  This limits LDAP traffic to sharepoint3, unless sharepoint 3 is down.
  5. Now we need to limit traffic to the ECTS admin interfaces to connect preferentially against sharepoint3 as well.  We do this with iRules.  Create a new iRule which will intercept traffic to the ECTS management pages, and route it to a different pool:
    when HTTP_REQUEST {
     if {[HTTP::uri] starts_with "/ECTS/"} {
     pool WSS_partner_admin
     }
    }
  6. We then create a Pool with priority group activation, as outlined in steps 1-3, above, and assign this iRule to the main SharePoint virtual server.

I am able to verify though packet captures that all requests to the ECTS admin pages, and all traffic to the ECTS front ends is being routed preferentially to “sharepoint3″.  So far, this seems to have helped.  If nothing else, the admin pages are now loading consistently, which was not true for us in the past.

External Collaboration Toolkit for SharePoint – Run Screaming

After two days of troubleshooting some vexing problems with ECTS, I have arrived at a new recommendation concerning this product:

Run Screaming

Okay, that may be a bit damning… here is a qualifier.  If you have no SLA with your customers, don’t mind lots of downtime, love C# programming, and otherwise find SharePoint troubleshooting highly amusing, then ECTS is the solution for you.  Otherwise, try something less painful.

The main discovery that pushes me to make this recommendation is that the ECTS solution is not supported by Microsoft.  The head of the MS Solutions Accelerator group once told me that all Solutinos Accelerators for currently supported MS products are supported by Microsoft.  This is not actually the case.  Always read the fine print for your Solutions Accelerators.  In the case of ECTS, we find the following in the ECTS FAQ, available in the “Informational Materials” download.  From http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=d9af2c25-989c-45c4-8008-1f15722190ed:

Answer: Although every effort was made to ensure that the External Collaboration Toolkit for SharePoint provides trouble-free installation and reliable operation, it is not officially supported by Microsoft. We will provide best effort support on the SharePoint – Collaboration forum, but cannot guarantee timely resolution of any issues.

Other pertinent bits of info from the Information Materials:

  • There is a suggestion that the product has been tested and is “supported” (whatever that means) only with Server 2003 (not Server 2008).
  • There is a reference to the commercial product called the “External Collaboration Manager“, which I expect was developed by the same consultants that created ECTS in the first place (although I have no proof of that).

I am going to pursue aggressively a Forms-based authentication solution using ADFS as a replacement for this service.

FWIW, here is a quick breakdown on the current set of problems, and what I did to fix them:

  1. When attempting to load the \sites\[sitename]\_layouts\ExternalCollaboration\aeu.aspx page (add new external user form), one user reported “Access Denied” error messages, even though he was in the site administrators list.
    • This happens because the aeu.aspx page checks to see is the currently logged in user is in the “Site Owners” group.  Of course, being a site administrator should be sufficient, but the page does not check for effective rights, but instead performs a simple ACL check.
  2. Users would intermittently experience page load errors when attempting to load the same “aeu.aspx” page.  Experientation with our load balancer indicated that the problem occured on  only one of the two web front ends in our farm.  Attempts to troubleshoot by using verbose Diagnostic logging, SysInternals ProcMon, and various browser debuggers (ieHTTPHeaders, Fiddler) turned up nothing.
    • Finally, I discovered that there was a single “ExternalCollaboration.RESX” file missing in the inetpub\wwwroot\wss\VirtualServers\[IIS-site]\App_GlobalResources\ directory.  After copying the RESX file to all three production IIS sites on both web front ends, and performing an IIS Reset, the page load error went away.  I don’t believe that these files ever got replicated to the second web front end after installation, so add this step to your farm build procedure.
  3. Our Account Services team reported that the “PartnerAdmin” page (or ECTS Administration web part) was reporting a “service unavailable” error.
    • This problem happened because all of the web.config files on our server reverted their “ADAMConnectionString” values to point to the pre-production host name for our ECTS LDAP service.  The event that triggered this service reversion is still unknown.  After I updated the string to point to “sharepoint.uvm.edu”, the problem was resolved.  Once again, I really need to re-install the whole solution using the correct LDAP connection string… the incorrect value is still cached somewhere in the SharePoint infrastructure that I cannot locate (most likely in configuration DB), waiting to bite me again.

Sharepoint – farm build procedure

After a semi-disaster with SharePoint earlier this week, I have been forced into the view that I really should have our SharePoint infrastructure hosted on more than one web server.  To that end, I am planning the deployment of a new, 2+ node Windows SharePoint Services farm.

Initial architecture will be something like this:

  • Host: SharePoint2
    • Roles:  Web front end, Search Server query and crawl, ECTS ADAM Instance
  • Host: SharePoint3
    • Roles:  Web front end, Search Server query and index, ECTS ADAM Instance
  • Hosts: WinDB1 and WinDB2
    • Roles:  Back-end SQL Database failover cluster
  • F5 Big-IP Local Traffic Manager (hardware load balancer)

Once initial rollout is complete, we likely will want to add:

  • Hosts: WinDB1 and WinDB2
    • Reconfigured in a SQL mirrored configuration

Here is an outline of the SharePoint2/3 build procedure:

  1. Install Server 2008 x64 Standard OS
    1. Activate Roles:  IIS (with ASP.NET support), AD Lightweight Directory Services (AKA AD LDS, AKA ADAM).
    2. Activate Features:  .Net Framework 3.0, PowerShell, SMTP Server
    3. Activate Feature “Desktop Experience” if you want access to “cleanmgr.exe” (Disk Cleanup wizard) and other desktop niceties.
    4. To save disk space, you might want to delete the hiberfile.sys (Hibernation file), by running “powercfg.exe /hibernate off”.
      1. Relocate the page file?
      2. Do something about WinSXS directory bloat (likely impossible without a service pack)
      3. Get IIS log files under control!
  2. Configure SMTP Server using IIS 6.0 Manager.  (FTP and SMTP Services cannot be managed with the IIS 7 Manager!)
    1. Configure BadMail and Drop Box directories so that they are not located on the System volume
    2. Configure to listen only on primary server IP
    3. Configure firewall to accept SMTP conections only from our In-Mail Gateways (using external firewall in our case, but this could be done with SMTP service settings and/or the Server 2008 firewall as well). <-DON’T FORGET – SMTP is allowed to the hosts, but need to go over the config with our DNS/mail admins!
  3. Install Search Server Express x64 bits:
    1. Perform “complete” install (Search Server will not install a SQL 2005 instance, as is the case with WSS installer).  Under “file location”, specify “E:\Office\12.0\Data” as the index storage location.
    2. Skip running of the Configuration Wizard after install.
  4. Install SharePoint Administration Kit v2.0
    1. Exclude Profile replicator component as it will not work on WSS
  5. Clone the server as many times as deemed necessary. (At present, make one clone!).  Any cloned systems must be sysprep-ed before joining the domain.  Once preped, join the computers, configure networking.
  6. If planning to add this server to a load balanced cluster, install NLB feature:
    NOTE:  We will not be doing this as we plan to use F5 hardware LBs instead.

    • from “administrator” cmd shell, run “ocsetup NetworkLoadBalancingFullServer”
    • Don’t join to a production NLB cluster until SharePoint configuration is complete!
  7. Replicate AD LDS (ADAM) instance to new machine, if required.
    (We need this as we are extending an existing ADAM instance to the new farm)

    1. In Server Manager, Click on “AD Lightweight Directory Services” Role,
    2. Click “AD LDS Setup Wizard”
      1. Select “A replica of an existing instance”
      2. Name the instance “ECTSInstance”
      3. Accept standard LDAP ports
      4. specify a partnerpoint server to replicate from, use standard LDAP ports.
      5. Select the “OU=ects,…” partition set for replication (this should be the only partition!)
      6. Select secondary (non-system) volume as target for AD LDS data… generally this will be “E:\Microsoft ADAM\ECTSInstance\data”
      7. Specify domain service account to run the AD LDS instance.
      8. Add “domain admins” to the AD LDS Administrators list.  Finish the wizard.
    3. Run the campus…bat file located in e:\Microsoft ADAM\ECTSInstance\data\.  This will register the Kerberos Service Principal Names required for LDP replication mutual authentication.
    4. Open the “Local Security Policy” Admin tool.  Add the domain service account to the “generate security audits” User Rights Assignment branch.
    5. Open the AD Users and Computers tool, locate the computer object on which you installed the Instance.  Give the LDS service account “create all child objects” to the computer object.
    6. Add the cluster load balanced SSL cert into the Personal certificate store of the ECTSInstance service account.
      1. Request wildcard certificate using the procedure outlined here:
        http://erlend.oftedal.no/blog/?blogid=7
        (We use the web interface for requesting a certificate, make user we use the RSA SChannel crypto provider to generate the request, use the “SHA-1” hash, use PKCS10 format, and use the “UVM – Web Server” request template.  For load-balanced LDAP servers, we must request a wildcard certificate (*.uvm.edu)
        NOTE: This step will not have to be repeated again until the current cert expires.  To add another AD LDS server, export the cert from a current server, import into the new server
      2. Export the request cert to file selecting “export all extended attributes” and “export private key” options.
      3. Import the cert into the “Personal” branch of the service account’s certificate store on the target server.  Make sure that you import “all extended attributes”, and the private key.  Do not select the use of advanced encryption password.
      4. Restart AD LDS and test SSL connections.
      5. If all is not working (as is the case with one of my two servers), here is where we get into undocumented territory.  Here are some helpful resources for debugging:
        1. I set SChannel diag logging to verbose :
          • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel
            REG_DWORD EventLogging, value 0×7
          • Restart ECTSInstance, look for “SChannel” entries in the server “application” even logs.  These logs will tell you which certificate the system attempted to use, and why access failed.
        2. You may need to add the wildcard cert to the Local Computer Certificate Store as well… run MMC, add the “Certificates” snap-in for “Service Account”, using the “ECTS Instance” service.  Navigate to the “Personal” branch, run an import action, import the wildcard with all extended attributes and the private key.
        3. Now locate the physical copy of this cert in c:\programdata\microsoft\crypto\RSA\MachineKeys (it will be the file with the most recently modified time stamp).  Add “read/execute” permissions to this file for the AD LDS service account, then restart the LDS instance.
    7. Force mutual authentication for replication traffic:
      1. Run ADSI Edit
      2. “Connect to”, enter the AD LDS server name in the Computer field, select the “Configration” well-known naming context.  As documented in http://technet.microsoft.com/en-us/library/cc794841.aspx, get “properties” on the “CN=Configuration…” partition, and change the value of “msDSReplAuthenticationMode” to “2”.
    8. Set local password policy – this controls password policy of AD LDS accounts:
      1. Add the Sharepoint server computer account to the “ETS – SharePoint Password Policy” GP Object.  After running “gpupdate /target:computer /force”, verify the settings by doing the following:
        1. Open Local Security Policy control panel
        2. Expand “Account Policies”->”Password Policy”
        3. Settings applied should follow the 1/365/0/8/Disabled/Disabled format.  (we may want to revisit this policy later).
  8. Run the SharePoint Products and Technologies Configuration Wizard:
    1. Connect to an existing Farm
    2. Enter “WINBD” as the database server.  The wizard will correctly select “SharePoint_FarmConfig” as the configuration database.  The correct service account username will be provided… you need to enter the password.
    3. Click “Advanced Settings”, specify that you which the server to host the Central Admnistration site.
      1. If setup fails with the error:
        “SharePoint Configuration Wizard failed with an exception “Error during encryption or decryption. System error code 997″
        A solution can be found here:
        http://blogs.msdn.com/priyo/archive/2007/08/11/add-new-sharepoint-server-to-existing-server-farm-an-unhandled-exception-occurred-in-the-user-interface-exception-information-unable-to-connect-to-the-remote-server.aspx
        Essentially we just run “stsadm –o updatefarmcredentials –userlogin “domain\service_acount” –password <thePassword>” on the first SharePoint server, then re-run the wizard.
    4. Update the “Central Admin” shortcut to point to the local Central Admin site by doing the following registry hack:
      http://blogs.technet.com/wbaer/archive/2007/08/30/sharepoint-3-0-central-administration-url-on-multiple-web-front-end-servers.aspx
      Essentially, edit the key:
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\WSS
      Then locate CentralAdministrationURL and change it to point to the local server.
  9. Configure Search Service:
    1. When Search is run in an environment where SharePoint services are accessed from a FQDN which is different from the physical host name (i.e. our environment, or any other environment with load balancers), you will need to work around the “loopback security check” feature of Windows.  Failing to do so will result in “access denied” errors in the crawl logs.  My thanks to Shawn Feldman for discovering this:
      http://blogs.msdn.com/fledman/archive/2008/09/18/access-denied-with-windows-server-2008-and-moss-when-crawling.aspx
      The relevant work-around is documented here (see “Method 2”):
      http://support.microsoft.com/kb/896861
      We simply need to add the public FQDN of our SharePoint server to:
      Key: HKLM\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
      Value: REG_MULTI_SZ, sharepoint.uvm.edu
      And then restart the IISAdmin service.
    2. Open the search admin page from SharePoint Central Administration:
      1. Access Crawling –> Content Sources
        1. Click the “Local Office SharePoint Server sites” default source.
        2. Define a crawling schedule for the SharePoint application
        3. Click “new content source” to add any additional content sources that are desired (i.e. our production file servers).
        4. Define additional crawl schedules for these new content sources.
        5. Set up “Crawl Rules” to exclude any directories from the content sources that you do not wish to have indexed.  In our environment, it was essential to exclude the “~snapshot” directories from the root of our NetApp file shares.
    3. Add Search Center to our SharePoint landing page:
      1. Create a new sub-site of type “Enterprise Search”
      2. Tune federated results by adding/removing web parts to the results page
    4. Add Federated Locations for additional search results:
      Ideally we would like to add results from out “GoogleWeb.uvm.edu” search appliance, and perhaps a Google “site” search for other close University partners

      1. For GSA (Google Search Appliance) Federation, it appears we will need to setup an RSS Transform:
        1. http://enterprise-code-samples.googlecode.com/svn/trunk/rss-stylesheet/Readme.htm
          http://enterprise-code-samples.googlecode.com/svn/trunk/rss-stylesheet/
        2. As the readme above recommends, we set up a new “front end” on the GSA for RSS results.  We called this Front End “RSS”.  We edit the “base_url” value to point to http://googleweb.uvm.edu/Search?. We then visit this URL and perform a query.  We take the resulting query results URL:
          http://googleweb.uvm.edu/search?q={searchTerms}&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&lr=&client=uvm2008&ud=1&oe=UTF-8&ie=UTF-8&proxystylesheet=uvm2008&site=default_collection
          and we substitute the name of our Front End in the place of the “proxystylesheet” and “client” values.  I also simplify the query just a bit as follows:
          http://googleweb.uvm.edu/search?q={searchTerms}&output=xml_no_dtd&lr=&client=rss&ud=1&oe=UTF-8&ie=UTF-8&proxystylesheet=rss&site=default_collection
        3. We then use the RSS query URL above as the template URL for a federated search query in the SharePoint Search Center.
        4. We edit the search results page to include federated results, and specify the “UVM GoogleWeb Search” federated location in the “Location” of the Federated results web part (using the “Modify Shared Web Part” command).
    5. Integrate Windows Search desktop tool with Search Server:<- Not yet done, but this can wait until after the production cut-over.

      1. Approve Windows Desktop Search 3.0->Windows Search 4.0 update on WSUS server
      2. Deploy Group Policy templates for Windows Search 4
      3. Configure:
        http://technet.microsoft.com/en-us/library/cc732491.aspx#BKMK_Addprimaryintranetsearch
  10. Install Infrastructure Update for WSS3 x64:
    1. Initiate the update on the first node in the cluster.
    2. When prompted, start the install on the second cluster node.
    3. When the configuration wizard completes on the second node, go back to the first and allow configuration to complete.
  11. Install Infrastructure Update for Search Server x64:
    1. Initiate the update on the first node in the cluster.
    2. When prompted, start the install on the second cluster node.
    3. When the configuration wizard completes on the second node, go back to the first and allow configuration to complete.
  12. Clean up IIS settings for the newly created Web Sites – configure binding, authentication and SSL  (Note that these procedures are only accurate when using Windows-native load balancers… when we transition to f5 load balancing, it will not be necessary to return custom errors from IIS as the f5 will handle HTTP-to-HTTPS redirections.) :
    1. SSL Cert Installation:
      Install SSL certs into “Personal” Store of the Computer account using the “Certificates” MMC snapin.
    2. Binding:
      Open the IIS Manager MMC snapin.  On each site, right-click and select “edit bindings”:

      1. For site “SharePoint – 443” (which represent the traditional “sharepoint.uvm.edu” URL), bind https and http protocols to port 80 and port 443, using the IP address for “sharepoint.uvm.edu” (132.198.102.12).  When binding SSL, select the appropriate cert from the “SSL Certificate” drop down menu.
      2. For “SharePoint – Internet” (which represents SharePointLite), bind https and http, ports 443 and 80, to “sharepointlite.uvm.edu”, IP 132.198.102.36.  Again, select the correct SSL cert for this site.
      3. For “SharePoint – Extranet” (which represents PartnerPoint), bind https and http, ports 443 and 80, to “partnerpoint.uvm.edu”, IP 132.198.102.49, selecting the matching SSL cert once again.
    3. SSL Configuration:
      1. In IIS Manager, open the “features view” for each site.
      2. Double-click “SSL Settings”
      3. Check “Require SSL”, leaving the default “ignore Client certificates” setting.
      4. Now double-click the “Error Pages” item for the server root.  Add a custom error for 403.4 (SSL required), pointing to our custom “redirect.html” javascript file.  We will need to have copied this file into “c:\inetpub\custerr\en-US\” before completing this step
      5. Now find the applicationHost.config file for the IIS server.  This should be located in “C:\Windows\system32\inetsrv\config”.  Locate the section for each site that serves SharePoint content (i.e. <location path=”SharePoint – 443”>), then locate the <httpErrors> tag under <system.webServer>.  In the httpErrors tag, change the value for “existingResponse” from “PassThrough” to “Replace” (response “Auto” also seems to work, but may produce inconsistent results).  This will prevent ASP.NET from replacing the 403.4 error response from the IIS server.  I am much indebted to this forum thread for this breakthrough:
        http://forums.iis.net/t/1113734.aspx
        Also helpful was the new “failed request tracing” module in IIS7:
        http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis7/
        More information on the meaning of the various existingResponse values can be found here:
        http://blogs.iis.net/ksingla/archive/2008/02/18/what-to-expect-from-iis7-custom-error-module.aspx
  13. Install the MS FilterPack 1.0 (Search Server can already index most Office 2007 documents, but this adds ability to index inside of One Note files and ZIP archives):
    1. Follow instructions at:
      http://support.microsoft.com/?id=946336
  14. Install Adobe iFilter, with 64-bit “thunking” DCOM service:
  15. Install MindManager extensions.
    1. DEPRECATED – We will discontinue this extension with the new upgrade as it does not work with MM v7 or v8
  16. Install ECTS components on each web front end server.
    • Having problems with installation script… what if we try the ECTS update available though CodePlex???
      1. If using updated ECTS files, it will be necessary to update the PartnerAdmin and PartnerConfig pages, as the self-service Site Collection Manager.  The existing pages will not work because the GUIDs on the Web Parts have changed.
      2. The ects_setup_sharepoint.vbs script still fails using the updated code… Since the codeplex team has not documented their changes, I think we will skip this option.
    • Troubleshooting issues:
      1. The ects_setup_sharepoint.vbs script succeeds in installing the ECTS solution, but fails when activating site features.  I suspect that “cscript” on Server 2008 is not processing return codes from stsadm.exe correctly, and this is reporting failure to install features (I am not positive about the reason for the script failure, although it certainly is not a result of stsadm.exe being broken.
        I was able to work around this problem by opening the ects_setup_sharepoint.vbs file in a text editor, searching for the error string that was sent to the console when the script failed, then running all of the operations in the script manually from that point forward.  Fortunately, all of the stsadm commands in the script are successful when run from the command line.
      2. ECTS is not compatible with MS Load Balancing out of the box.  I switched to a F5 load balancer before working through the problem.  It is possible that the problem I was having could have been fixed with the same “loopback security check” that caused problems during our F5 configuration
        http://support.microsoft.com/kb/896861
        In fact, we may have had the problem even with the f5 in place, but I would not know because I applied the loopback fix before implementing the F5.
        The error codes suggest that a login failure is occurring between the IIS application and the AD LDS LDAP instance.  When I try to connect to the load-balanced LDAP DNS name using the “LDP.exe” LDAP client, I also get an authentication error.  However, when I connect to the local server address, authentication works.
      3. As was the case when I first installed ECTS, the web.config files required a bit of hand-tuning to get services working correctly:
        http://www.uvm.edu/~jgm/wordpress/?p=112
        Once again, I had to modify the “ADAMConnectionString” in the web.config of each IIS site to reflect the actual DNS name of the load-balanced AD LDS servers.  I had installed ECTS using a different name initially, and the ECTS un-installation script did not clear out these values.
      4. I did find it necessary to deactivate all ECTS site collection features, re-activate them, then perform an IIS reset before my existing ECTS management pages would work again.  This seems pretty par for the course when removing and re-installing SharePoint solutions.
      5. After connecting the production content database and changing the URL for the server, I also needed to edit “LDAPHost” value to reflect the production name of the server in the “TEMPLATE\FEATURES\ECTSBase\Feature.xml” file (located in the “12″ hive).  This value was set at the time that ECTS was deployed to the server.  I expect that the same (wrong) LDAPHost value would get replicated to additional ECTS servers if we were to add new ones, because this setting is contained in the RESX package that gets built by the ECTS installer, and cached in the database for future deployment.  I expect I would need to do a full retract/remove/redeploy to fix the problem permanently.
  17. Install Globally-deployable solutions from the “fab 40” application template.  If you deploy a web front end into an existing farm, the files required by these features will get transferred automatically.  However, when building a new farm, we need to install them manually.  Currently required “server admin” templates are:
    • ApplicationTemplateCore
    • ChangeRequest
    • ContactsManagement
    • DocumentLibraryReview
    • EventPlanning
    • HelpDesk
    • InventoryTracking
    • ITTeamWorkspace
    • Knowledgebase
    • LendingLibrary
    • PhysicalAssetTracking
    • ProjectTrackingWorkspace
    • RoomEquipmentReservations
    • Procedure:
      • stsadm -o addsolution -filename <file_path>\<template_name>.wsp
      • stsadm -o deploysolution -name <template_name>.wsp –allowgacdeployment
  18. Install radEditor:
    1. Install ASP.NET Ajax for .NET 2.0, version 1.0
    2. Follow the Ajax configuration for SharePoint configuration guide found here:
      http://sharepoint.microsoft.com/blogs/mike/Lists/Posts/Post.aspx?ID=3
    3. Install radEditor using the included instructions.
    4. Copy radEditor configuration files from an existing production server to the new server:
      1. In the directory:
        ”C:\Program Files\Common Files\Microsoft Shared\web server extensions\wpresources\RadEditorSharePoint\[versionString]\RadControls\Editor”
        Backup the existing ListConfigFile.xml, ConfigFile.xml, ListToolsFile.xml, and ToolsFile.xml files.  Replace with versions customized for UVM.  Note that the MOSS LinkManager tool does not work in WSS.  Also note that when editing list content that does not support “Enhanced Content”, the first toolbar in the ListToolsFile.xml will be removed… in past versions, the toolbar named “enhancedTools” was removed.
      2. Copy the files ListConfigFile.xml, ConfigFile.xml, ListToolsFile.xml, and ToolsFile.xml to all other nodes in the cluster.
      3. perform an IISRESET.
      4. Update ONET.xml files in the “12” hive to activate the radEditor feature by default in all new sites (see ONET.xml template files on the prod web front end for examples).
        NOTE that the “RadEditor for non-IE browsers” and “RadEditor for IE” features have been collapsed into one unified feature.  Update the ONET.XML files accordingly!  (note that the feature ID for the main RadEditor List editor has not changed… only it’s name is different.  We did not have to insert a new default feature ID, but we did need to remove the “RadEditor for IE” feature because it is no longer present in RadEditor MOSS.)
      5. Run:
        stsadm –o uninstallfeature –name RadEditorFeatureRichHtml.
        This “Web Content Management” feature is not supported in WSS, so we may as well remove it to avoid confusion.
      6. Deactivate and then re-activate the radEditor features on at least one existing site, and test functionality.
  19. Install “Smiling Goat” Feed Reader (RSS/ATOM subscriber web part)
    1. This will require Feed Reader users to update their web parts!
    2. Current release here:
      http://www.codeplex.com/FeedReader/Release/ProjectReleases.aspx?ReleaseId=19830
      (version 3.1.0.1 at the time of this writing)
  20. Install SharePoint Training Kit:
  21. Tune web application settings to match production server:
    1. In “Operations”:
      1. Configure blocked file types – in our case we have been requested to all “MAT” files.  MAT is one of may obscure Office file extensions, but also is used by MatLab, a common mathematical/statistical application used on campus.
      2. Enable Usage Analysis Processing – relocate log files off of the system volume.
        1. You MUST grant read/write access to the local WSS_WPG to the folder that will hold the usage analysis logs.  If you fail to do so, you will see lots of errors in the Diagnostic Logs allong the lines of “Cannot create folder “<big ugly hexidecimal number>””.
      3. Configure incoming and outgoing email settings.
        1. Incoming mail requires SMTP to be installed on the server – managed with the old-school IIS 6.0 Manager MMC.
        2. We needed to use the “Advanced Settings” to specify the SMTP drop folder in the Incoming Mail settings page of Central Administration.  SharePoint diagnostics logs indicated that the timer service could not determine the drop folder location automatically.
        3. After configuring the mail drop folder, SharePoint still was unable to process messages.  “Access denied” events started queuing up in the event viewer.  A quick analysis using SysInternals ProcMon shows that the WSS Access account (being used by the OWSTIMER.EXE process) has no ability to access the drop folder defined above.  I just added “modify” access for that account to the drop folder, and incoming mail started processing immediately.
    2. Under “Application Management”:
      1. Tune the “Web application general settings”:
        1. Set upload limits for files
        2. Set time zone
        3. Set quota templates for new sites
      2. Config “site use confirmation and deletion”. <- Be sure to do this after PROD cutover!!!  Can’t do it ahead of time as users will start to get email from the pre-prod server!
      3. Enable self-service site creation. <- Also needs to be done after prod cut-over
      4. Configure “policy for web application” to allow selected groups of SharePoint administrators rights to all sites in the farm.
    3. Edit the footer of the “welcome” email message starting at line 5219 of “core.en-US.resx” in the 12-hive “resources” folder.  Replicate on all web front ends in the farm.
  22. Configure f5 load balancers:
    1. Lots of IP addresses required:
      1. “floating ip” for network on which F5s will communicate with the SharePoint web servers.
      2. 2x Self-IPs for the physical interfaces on the F5s which will handle the floating IP
      3. 3x IP addresses for the public SharePoint URLs (standard, lite, partner)
      4. 6x IP addresses for the private interfaces on each web server node (for standard, lite, and partner sites, one each on two web servers)
    2. Need to generate SSL Certificate/Key pair files for the load balancer “SSL Profiles” (the F5 uses separate files for certificates and keys, as opposed to IIS which uses a single CER file – we need to split the IIS files and upload to the F5.
      1. Using the certificate manager MMC snapin, export the production certs to PFX (PKCS #12) files (make sure to specify that you wish to export the private key, otherwise PKCS #12 is not an option in the wizard).
      2. Convert the pfx file into a PEM using the openssl executables (available for Windows here: http://www.slproweb.com/products/Win32OpenSSL.html) using the folowing command:
        openssl.exe pkcs12 -in [infile].pfx -out [outfile].pem -nodes
        (You will be prompted to enter the password for the PFX, if it has one.  The “-nodes” flag indicates that the output file will not be encrypted with a password)
      3. Open the PEM file created above with a text editor.  Remove the  private-key portion of the file (all content from “Bag Attributes” to “—–END RSA PRIVATE KEY—–”) and paste this into a separate “KEY” file (UNIX-style line endings, ANSI-encoded).
      4. Upload the new PEM and KEY files into your F5 load balancer to create a new SSL client profile.
    3. Set up two virtual servers for each SharePoint URL – one to redirect HTTP to HTTPS, one to handle regular traffic.
      1. Add the following iRule to perform the SSL redirect, and attach it to the Virtual Server listening on port 80:
        (This rule will redirect your browser to https://sharepoint.uvm.edu from either http://sharepoint or http://sharepoint.uvm.edu… well, actually it will redirect any non-HTTPS connection to a HTTPS version of itself, appending our domain “.uvm.edu” if it is missing.)

           1: when HTTP_REQUEST {
           2:   if { [HTTP::host] == "*.uvm.edu" } {
           3:     HTTP::redirect https://[HTTP::host][HTTP::uri]
           4:   } else {
           5:     HTTP::redirect https://[HTTP::host].uvm.edu[HTTP::uri]
           6:   }
           7: }
  23. TEST TEST TEST:
    1. Test each feature on both web front ends by alternately disabling the nodes in the load balancer configuration.
    2. Test again with both nodes enabled… watch for authentication and session persistence issues.
    3. Test all features in each access mapping – SP, SPLite, and Partner… web.config file variations could cause problems!
  24. Consider Deployment of “Group Board 2007” and “Sample Master Pages”:

And to complete the cutover:

  1. Redirect SharePoint traffic to a “maintenance underway” page, or just put the service in read-only mode.
  2. Backup the current production content database
  3. Detach the content database from the production server.
  4. Attach the content database to the new servers
  5. Change the SSL profiles used on the F5 load balancers to use the production certs.
  6. Take down the original SharePoint IIS server and disable.
  7. Change the listening port on the F5 load balancers to match the production server IPs.
  8. Configure Zone security and Alternative Access Mappings for the Farm to reflect the new production names:
    1. Make sure that https://sharepoint.uvm.edu, https://sharepointlite.uvm.edu, and https://partnerpoint.uvm.edu are all listed as “Public URLs” for the farm.
    2. Make sure that the HTTP equivalents of these URLS are listed as INTERNAL URLS in the AAM table if you are performing SSL termination on the load balancers.
  9. Contact SAA Mail admins to update the mail server LDAP records for SharePoint mail routing
  10. Dredge though this post for things that will need to be configured after the migration, and test mercilessly.

ECTS Login Errors – Troubleshooting

Users of our ECTS implementation “PartnerPoint” are not an overly happy set.  Most of the problems that we have experienced are centered around login errors.  This application is particularly prone to login errors for the following reasons:

  • Randomly generated initial password is too complex – data entry errors cause login denial
  • Password expiration errors are not transparent – We need to capture the error that is seen in a password expiration instance.
  • Passwords generated by ECTS (either during account creation or a ECTS admin reset) are temporary and must be changed on next login.  The account attribute “eatmuPwdGenerated” holds this information.
  • Password strength requirements are not displayed in the forms, and password strength errors are not detailed or helpful (i.e. they do not tell you why your password is unacceptable).
  • Login errors are not detailed or helpful – they do not tell you if the account is locked or disabled, it the password is expired, or if you simply entered an invalid username/password combination.
  • Even when login is successful, users often will get “access denied” messages because of permissions problems:
    • The ECTS “Add External User” dialog generally refuses to add permissions to the ACL for a site… it only works consistently when you add the user to an existing site group
    • Sign-in for ECTS users requires at least “Read” permissions to the to-level site in a site collection.  You cannot grant external users rights to a child site with no permissions in the parent.

Clearing account lockouts – accomplished by setting the “lockoutTime” attribute of the AD LDS account to “0”.  This causes the “badPwdCount” attribute to be reset to zero.  Note that you cannot set “badPwdCount”, nor “badPasswordTime” as these attributes are owned by “SYSTEM”, and thus cannot be edited manually.  Solution located in the “EggHead Cafe”:
http://www.eggheadcafe.com/forumarchives/windowsserveractive_directory/nov2005/post24794404.asp

Other AD LDS account attributes to watch – see MSDN Active Directory Schema documentation for details:
http://msdn.microsoft.com/en-us/library/ms675090(VS.85).aspx

  • eatmuPwdGenerated – attribute added by ECTS installer.  Indicated whether current password was generated by ECTS, or by the user.  Reset when the user successfully logs in and sets his own password.
  • msDS-UserPasswordExpired – populated, but apparently not accurate or used.
  • msDS-User-Account-Control-Computed – Most commonly used for reporting on account state.  This value is computed from other fields, and should not be modified directly.
  • pwdLastSet – in active use for ECTS accounts, uses “Large Integer” value, formatted as “NT Time”, or number of 100 nanosecond intervals from 1 Jan, 1601.  This can be converted to a readable date using “w32tm.exe /ntte [string]”.  This value cannot be reset to a specific value, although you can use the special values “0” (meaning “must change at next login”) or “-1” (meaning “now”); however, ADSI Edit does not allow you to enter the value of “-1”, so we would have to use a different tool to set a “-1” value.  Setting the value to “0” will break login as there is no LDAP method for requesting a password change.

Setting up Inverness – the External Collaboration Toolkit for SharePoint

Decision Points:

Need to decide…

  • Where to host ADAM – We want it replicated for additional fault tolerance:
  • The best option may be to run it on each SharePoint server in your farm… ADAM supports clustering via network load balancing.  If you have setup up NLB for SharePoint, it then should not be too difficult to set it up for ADAM as well.  This also eases configuration on Internet-facing servers… you don’t need to allow additional firewall rules from the SharePoint web server to an ADAM server in your trusted network.
  • Where to host the ECTS configuration database:
    • Place the DB external to the SharePoint server for scale out … you don’t want to install an essential component on a web farm server that you want to be able to take down to apply maintenance.
  • External URL for SharePoint:
    • I should be something short, and fairly memorable.  It also should sounds distinct from the Internal URL… I am trying “PartnerPoint.uvm.edu” to start out with .

    Gotchas:

    • ADAM Setup:
    • ECTS requires SSL for ADAM… check the Event Viewer “ADAM” events for SSL errors.  Documentation suggests giving the ADAM service account rights to the required certificate in the “All Users” profile… this worked for me in TEST once my certificate has a Fully Qualified Domain Name in its subject line.  My auto-enrollment generated certs only have the FQDN in the SAN (subject alternative name) field, thus I needed to generate a new cert for SSL to work.  Once doing the PROD deployment, this fix did not work, and I had to copy the PROD SSL cert from the “Computer Account” Personal certificate store to the “Service Account” Personal certificate store (the store labeled “ECTS Instance”)
    • In our production environment I had trouble running the setup script… this was because I was trying to use ports that ADAM considered “invalid”.  Keep an eye on %windir%debugadamsetup.log if scripts fail… as indicated here:
      http://technet2.microsoft.com/windowsserver/en/library/2080b841-2211-400f-b393-04675a1653651033.mspx?mfr=true
  • SharePoint app configuration:
    • It seems that host headers may be required by the setup scripts… if initial config fails, try adding host headers to internal and external web sites, then running setup again.
    • Connection Strings are for ADAM and the ECTS database are stored in the web.config files of each SP web site instance… you can inspect these to validate your config.  Also, you can change the ECTS database using these connect strings.  By doing to I was able to rename the database to “ECTSTest” so that I will be able to install the PROD database on the same server.
    • In more complex environments where you are adding ECTS to an existing SharePoint server, the ects_setup_sharepoint.vbs script may not update all of the web.config files on your server.  This was not a problem on my test server, but it was a real pain in production.  To fix the issue, I copied the following sections from my Extranet site web.config to by internal (“Default” zone) web site web.config:
      <connectionStrings>
      <add name=”ADAMConnectionString” connectionString=LDAP://myserver:636/CN=Users,OU=ECTS,dc=mycontext />
      <add name=”DBConnectionString” connectionString=”Data Source=MYDB; Database=ECTS; Integrated Security=SSPI” />
      </connectionStrings>
      <location path=”_layouts/ExternalCollaboration/PasswordReset.aspx”>
      <system.web>
      <authorization>
      <allow users=”*” />
      </authorization>
      </system.web>
      </location>
      <system.net>
      <mailSettings>
      <smtp from=PartnerAdmin@myserver>
      <network host=”smtp.myzone.net” port=”25″ defaultCredentials=”true” />
      </smtp>
      </mailSettings>
      </system.net>
  • ECTS “Configuration Utility” web part:  This is tricky, as the documentation is a bit vague on this.  You must add the account that you are going to use to run the utility to the “datareaders” and “datawriters” roles of the ECTS database IN ADDITION TO the WSS service account.  If the correct permissions are not added, configuration will fail… somewhat silently.
  • Web Part Security:  Access to the ECTS web parts is dictated though SharePoint site groups.  The users that you want to access the site must be added explicitly, not though group objects.  I.E. Lets say we have a user named “Jimmy Joe Jim Bob” (with NetID “jjjb”).  He is a member of the Active Directory group “SharePoint Gods” (DOMAINSPGods).  We want Jimmy Joe to be a “Site Collection Approver”, I must add the NetID “jjjb” to the “External Site Approvers” SharePoint group.  I cannot add the group “DOMAINSPGods”… it just does not work that way.
  • Help Resources:

    “SharePoint – Collaboration” forum on MSDN:
    http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=2012&SiteID=1

    ECTS online documentation on TechNet:
    http://technet.microsoft.com/en-us/library/cc268155.aspx

    Here is one that really helped me:
    (actually, the same information is available on many web sites, but this is really concise.  Getting “real” debug output helps to identify the source of config problem.  I found these stack traces a lot more useful than looking at SharePoint diagnostic logs)

    http://www.sharepointblogs.com/michael/archive/2007/06/28/sharepoint-under-the-hood-see-real-error-description-and-callstack-stack-trace.aspx