Error Message

Once the index rebuild process is stuck in the rebuild log, it shows errors below for multiple items:

3401 201511177729302~201508171249420000~Z~2004D7B9DACB7403322B15756A992901 http://xxxxxxxx.xx.local/EnterpriseVault/download.asp?VaultID=144CCD025FFC7E04196E497E204C8A1F71110000evarchive01&SavesetID=201511177729302~201508171249420000~Z~2004D7B9DACB7403322B15756A992901&Request=NativeItem               103        Elasticsearch encountered error whilst trying to index item

In Dtrace logs below, errors can be seen:

4878655             xx:24:04.538     [38292]          (EVIndexVolumesProcessor)                              EV-H               {ESNestClient.AddData} Exception: The underlying connection was closed: An unexpected error occurred on a send. Call: Status code unknown from: POST /144ccd025ffc7e04196e497e204c8a1f7_20345_15/_bulk?refresh=true Info:Elasticsearch, error while adding data into Index [144ccd025ffc7e04196e497e204c8a1f7_20345_15] Diag: Type:Elasticsearch.Net.ElasticsearchClientException ST:   at Elasticsearch.Net.Transport`1.HandleElasticsearchClientException(RequestData data, Exception clientException, IElasticsearchResponse response)|   at Elasticsearch.Net.Transport`1.FinalizeResponse[TResponse](RequestData requestData, IRequestPipeline pipeline, List`1 seenExceptions, TResponse response)|   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)|   at Veritas.EnterpriseVault.Indexing.EVElasticsearchClient.ESNestClient.AddData[T](IEnumerable`1 batch, String index, List`1& failedItemsActions, Boolean& isFloodStageReached) Inner:System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine|   at System.Net.Sockets.Socket.MultipleSend(BufferOffsetSize[] buffers, SocketFlags socketFlags)|   at System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[] buffers)|   — End of inner exception stack trace ---|   at System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[] buffers)|   at System.Net.Security._SslStream.StartWriting(SplitWritesState splitWrite, SplitWriteAsyncProtocolRequest asyncRequest)|   at System.Net.Security._SslStream.ProcessWrite(BufferOffsetSize[] buffers, SplitWriteAsyncProtocolRequest asyncRequest)|   at System.Net.TlsStream.MultipleWrite(BufferOffsetSize[] buffers)|   at System.Net.Connection.Write(ScatterGatherBuffers writeBuffer)|   at System.Net.ConnectStream.ResubmitWrite(ConnectStream oldStream, Boolean suppressWrite)|   — End of inner exception stack trace ---|   at System.Net.HttpWebRequest.GetResponse()|   at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)

In network trace, we see the Enterprise Vault process are trying to reach the Internet and fail:

0x8fd8  5:28:38 PM x/x/xxxx     213616               7:58:38 PM x/x/xxxx               448.3762378               EVIndexVolumesProcessor.exe               xx.xxx.xx.xx     212.252.126.72               TCP        TCP:SynReTransmit #204408Flags=......S., SrcPort=57289, DstPort=HTTP(80), PayloadLen=0, Seq=4107348604, Ack=0, Win=64240 ( Negotiating scale factor 0x8 ) = 64240               {TCP:3453, IPv4:3044}

In Fiddler trace we may observe that EVindexVolumesProcessor is trying to access the Windows Update CTDL URL for a certificate update:

79          502        HTTP               ctldl.windowsupdate.com /msdownload/update/v3/static/trustedr/en/authrootstl.cab?76390415f089c1c7         512        no-cache, must-revalidate   text/html; charset=UTF-8               evindexvolumesprocessor:33868               78                   

GET http://ctldl.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab?76390415f089c1c7 HTTP/1.1

Proxy-Connection: Keep-Alive
Accept: /
User-Agent: Microsoft-CryptoAPI/10.0
Host: ctldl.windowsupdate.com
HTTP/1.1 502 Fiddler - Connection Failed
Date: Wed, xx xxx xxxx 09:36:33 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Cache-Control: no-cache, must-revalidate
Timestamp: 12:36:33.438

[Fiddler] The connection to 'ctldl.windowsupdate.com' failed.
Error: TimedOut (0x274c).
System.Net.Sockets.SocketException A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xx.xxx.xx.xx:80

Starting Enterprise Vault 15.0.1 HTTPS is enabled for secure communication on elastic search, since HTTPS is enabled, hence Windows by default goes to ctldl.windowsupdate.com for certificate validation and updating the windows certificate trust list (CTL). However, this communication may fail due to restrictions while accessing the internet, which may lead to further failure of the Elastic search Rest API.

This issue is fixed in Enterprise Vault version 15.1.1 and later.

As a work around below, the registry key can be used to prevent indexing from going to internet for certificate validation.

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

1. Open Regedit.exe on the EV server.  
2. Open HKLM\Software\Policies\Microsoft\SystemCertificates\AuthRoot 
3. Create a new DWORD entry with the name DisableRootAutoUpdate and set a value of 1
4. Reboot the EV server 
5. Verify that the issue no longer occurs