After upgrading to EV 12.5.1 or later, Metadata Store Build tasks take long time to complete or fail

book

Article ID: 100050303

calendar_today

Updated On:

Description

Error Message

When the job fails to complete: 

Metadata Store Build task report will show "The search did not complete in the required time"

24/03/2021 16:59:34 The metadata store build subtask encountered [2] errors.

    Failed to fetch batch from volume [VaultEntryId=[175C8BE852DCCB44DB3361957324332211110000evserver01] IndexVolumeIdentity=[1] StartISN=[1] EndISN=[60604] Offline=[False] Failed=[False] MissingItems=[0] Type=[64Bit] Number Of Ranges=[1]]. The search did not complete in the required time.

    Failed to fetch batch from volume [VaultEntryId=[175C8BE852DCCB44DB3361957324332211110000evserver01] IndexVolumeIdentity=[1] StartISN=[1] EndISN=[60604] Offline=[False] Failed=[False] MissingItems=[0] Type=[64Bit] Number Of Ranges=[1]]. The search did not complete in the required time.

24/03/2021 16:59:34 The metadata store build subtask is sleeping. Reason: [No items could be processed]. The subtask will try again after: [24/03/2021 17:29:34]

 

Event ID 41315 will be logged for the same archive in Enterprise Vault event logs

 

Log Name:      Veritas Enterprise Vault
Source:        Enterprise Vault
Event ID:      41315
Task Category: Index Query Server
Level:         Warning
Computer:      EVSERVER01.ev.local
Description:
A search failed with error "The search did not complete in the required time.".

Archive: Smith, John
Archive ID: 175C8BE852DCCB44DB3361957324332211110000evserver01
Index volume ID: 1
User: EV\evadmin
Maximum results: 40000
Timeout (seconds): 1800
Sort order: snum 

When the job completes: 

Depending on the size of the archive the job may complete but will take a long time to complete. The errors will not appear in the event log or the report as in the case when the job fails to complete. In this case the delay can be seen  in a Dtrace of EVIndexQueryServer as follows:

Note the TraceTimer for CalculatedFieldResultsTransformCommand 1348 seconds \ 22 minutes as an example:

403084    08:21:47.377     [4700]    (EVIndexQueryServer)    <6972>    EV-M    {SearchPipeline} TraceTimer [1348697ms] [15/19] Completed CalculatedFieldResultsTransformCommand, search 5, ID 68621d5b-ff66-4fc2-bba8-660f116f97c8, user JDL\vaultadmin, source SharedArchive1 (1013)

Cause

The issue is due to the time taken by CalculatedFieldResultsTransformCommand in the dtrace sample above. The time taken to complete this command will vary depending on the size of the archive in question. This behavior was unintentionally changed in (EV) 12.5.1 and is being investigated for a potential remedy. 

Resolution

Work Around:

We can workaround the issue by modifying EVMDSBuildTask.exe.config file in the \EntepriseVault program directory. 

There are four values that need to be changed in this file:

  1.  key="SearchTimeoutSeconds"   This manages the timeout for the thread that is processing EVMDS items
  2.  key="MaximumResultsPerItemQuery", Minimum amount of items allowed for a thread to process within the given timeout
  3.  key="MaximumResultsPerNRCPQuery",  Maximum amount of items allowed for a thread to process within the given timeout
  4. key="MinimumCompletionAfterCreationSeconds", Retry wait time for the Build process after completion. 

 

Note:

  • Prior to making any changes make a back up copy of the EVMDSBuildTask.exe.config file for each EV server.
  • Close out any inactive MDS Build jobs in the 'Monitor Indexing Tasks' status window. 
  • Changes must be made to each Enterprise Vault server in the environment. 

 

Modify the following keys as follows and save the config file:

 

Then restart IIS on each Enterprise Vault Server.

Resubmit any MDS Build jobs and monitor progress. 

 

 

 

 

Veritas has acknowledged that the above-mentioned issue is present in the version(s) of the product(s) referenced in this article.

This issue is currently under investigation by Veritas. Pending the outcome of the investigation, this issue may be resolved by way of a cumulative hotfix or service pack in the current or future versions of the software. However, this particular issue is not currently scheduled for any release.  If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Veritas Sales representative or the Veritas Sales group to discuss these concerns.  For information on how to contact Veritas Sales, please see http://www.Veritas.com .

 Please be sure to refer back to this document periodically as any changes to the status of the issue will be reflected here

Issue/Introduction

Enterprise Vault (EV) Metadata Store (MDS) Build Task takes a long time to complete or fails due to search timeout.

Additional Information

JIRA: CFT-3721