Description

   In some virtualized environments when Enterprise Vault (EV) indexing may suffer due to either poor disk performance, or as the result of the disk that hosts the indexes being over-accessed.  Initial troubleshooting would include dtracing, scouring event logs, and analyzing relevant EV database tables.  However if all of those yield no results, the culprit could be slow disk access.

   In such a case, dtrace lines of EVIndexVolumesProcessor , would show this every 10 seconds:

734 09:11:38.301  [15556] (EVIndexVolumesProcessor)  EV-L {StatusChecker} Did not receive any Additions acknowledgments
737 09:11:39.570  [15556] (EVIndexVolumesProcessor)  EV-M {StatusChecker} No Additions acknowledgments received. Will check again in 10 seconds...

While very infrequently showing a small successful acknowledgement:

26374 09:12:33.914  [15556] (EVIndexVolumesProcessor)  EV-L {StatusChecker} Received [588] Additions acknowledgements. Updating status list...
26375 09:12:33.945  [15556] (EVIndexVolumesProcessor)  EV-L {StatusList} Acknowledged all [588] item actions.

   A Perfmon report showing the Physical Disk counters would also show an exceptionally high Avg. Disk Queue Length, while most other counters in that group would read normal. For reference, see Microsoft's published article here about Physical Disk counters:  https://technet.microsoft.com/en-us/library/cc938959.aspx. The screen below shows very minimal disk activity.

The slow access to the disk is caused by a long line of requests to read and/or write on the disk are stuck "waiting their turns."  This can stem from several sources, including:

• Too many services / applications acting on the same physical disk.  For example, in a virtualized EV environment, a single disk in a LUN may be hosting parittions for EV's indexing and EV's storage.
• Clustered disks replication modes set to "asynchronous."
• Poorly performing or slow disks.

Run Perfmon on the EV server whose indexes have the performance issue.  Including the EV Indexing counters is fine, but most importantly include the Physical Disk counters.  Analyze the data from the resulting Data Collector Set in "report" view as this is most legible. 

Look specifically at Avg. Disk Queue Length, and confirm that it is below 2.0.  Anything above indicates that the I/O load for that disk is too high to serve the normal amount of read/write requests from EV for Indexing purposes.  Likely the load for that disk would need to be split, or more physical disks should be introduced.