FAST Search and the ticking time bomb

Arpit Patel

December 14, 2011

All Post

Tick…tick..tick…when you install FAST Search for SharePoint (FS4S), you probably have a time bomb set to go off exactly one year later to the minute. Unless you configured trust certificates between servers using a Certificate Authority, FAST/SharePoint uses a self-signed certificate with a one year expiration.

Worse, the ULS logs only point out that SharePoint can’t connect to the Content Distributor (green text is environment specific):

Failed to connect to srv-fast01.ReplaceWithYourDomain.com:13391 sp. Error=Failed to initialize session with document engine: Unable to resolve Contentdistributor [documentsubmitterworkerthread.cpp:132] d:officesourcesearchnativegatherpluginscontentpidocumentsubmitterworkerthread.cpp

I also saw the following error which is either misleading or unrelated:

At memory capacity. Load is 80%, configured to block at 80%. have been waiting 00:57 to queue this document [documentmanager.cpp:969]

To make matters worse, without connectivity to FAST, crawls hang and get stuck saying “Stopping”, clearing the FAST Index hangs in Central Admin; it’s not pretty…

Let’s take a step back. FAST requires that SSL is used for communication for search crawling between the FAST Server(s) and SharePoint Server(s). To communicate via SSL, a certificate needs to be generated for the FAST Sever and installed on the SharePoint server.

If you are getting errors connecting to the Content Distributor, it makes sense to first see if it is running, by running the following PowerShell in FAST: nctrl status

A very useful PowerShell command shows the connectivity and certificate status:

Ping-SPEnterpriseSearchContentService srv-fast01.[ReplaceWithYourDomain].com:13391

The below will show a list of the certificates. The timestamp below doesn’t lie; this is what I was doing at 5:30am this morning. When the problem existed, the highlighted entry showed an ExpiryDate of the day before and ConnectionSuccess of False. Note the port is 391 above your default port, which is 13000 unless you changed it on installation.

Here’s how to create a refreshed cert that will again celebrate its own birthday via expiration, On the FAST Server, open a PowerShell window in the D:YourFASTdirectoryInstallerScripts :

.ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

Microsoft helpfully provides a PowerShell script to load the certificate into the SharePoint server. Before we run that let’s configure things so we generate a certificate that’s good for longer than a single year. To do that, let’s edit the script in C:FASTSearchinstallerscriptsinclude called certificatesetup.ps1 and right after the line around line number 246 which says:

Add-Content -Path $infFile -Value "SuppressDefaults=true"
Add the following lines underneath it:
 
Add-Content -Path $infFile -Value "ValidityPeriod=Years" 
Add-Content -Path $infFile -Value "ValidityPeriodUnits=20"  #now we ar

You’ll want to be sure to get the SSA Name correct as well as the service account; this is the account under which the Application Pool is running that hosts FAST Search connector Service Application that is running in IIS. Note you’ll need to copy both the SecureFASTSearchConnector.ps1 script and the certificate itself (find the file by going to Certificates(Local Computer)Trusted Root Certification Authorities in MMC). You will also need to halt the FAST Search Service and FAST Search Monitoring in Servicesc.msc before being able to generate a new certificate. When exporting the certificate, make sure to export the private key, for which you will be prompted for the password. You’ll want in in PFX format, not in DER or CER format. If you try to use MMC, you may find you won’t be able to get the cert exported with the private key. The good news is that the FAST script already automatically exports the script in the right format, in this location: D:FASTSearchdatadata_securitycert

For the FAST SSA, use the Content SSA, and not the query SSA. You can determine the service account easily, by checking the Service Application in Central Admin and clicking “Properties”

.SecureFASTSearchConnector.ps1 –certPath “path of the certificatecertificatename.pfx” –ssaName “name of your content SSA” –username “domainusername”
If you do need to change the port for your Content Distributor, here are the PowerShell commands. Remember to replace the server/domain/port with your own:

$SearchSSA= Get-SPEnterpriseSearchServiceApplication -identity ‘FAST Search Connector’
$SearchSSA.extendedConnectorProperties[“ContentDistributor”]=”srv-fast01.YourDomain.com:13391″
$SearchSSA.update()