Accelerator Searches fail against ElasticSearch Index Volumes with error 0xc0041c0e

book

Article ID: 100063033

calendar_today

Updated On:

Description

Error Message

Error listed in the Accelerator Client for the affected Index Volume(s):
Index Search failed: %1 Index: %3 Internal reference: %2 (0xc0041c0e)

EV Event Log entry on the Accelerator Server:

Log Name: Veritas Enterprise Vault
Source: Enterprise Vault 
Event ID: 41315
Task Category: None
Level: Warning
Description:
A search failed with error "Elasticsearch failed to complete search request".
Archive:
Archive ID: 123ABC456DEF789GHI123ABC456DEF789GHI0000EVSiteAlias
Index volume ID: 123456
User: \
|Maximum results: 10000
|Timeout (seconds): 3600
Sort order: snum

V-437-41315

DTrace of the AcceleratorService process on the Accelerator Server:

[1234] (AcceleratorService) <5678> EV:M Response received. Result = <0xc0041c0e>, Status = 500
[1234] (AcceleratorService) <5678> EV:H CIndexQueryServerSearch::Search2. Request FAILED HR=0xc0041c0e.
[1234] (AcceleratorService) <5678> EV:L WinHttpRequest::Shutdown entry. Closing Winhttp handle
[1234] (AcceleratorService) <5678> EV:L WinHttpHandle: Closing handle.
[1234] (AcceleratorService) <5678> EV:L WinHttpRequest::Callback from WinHttp. Code=0x800, context=0x0FCDE254
[1234] (AcceleratorService) <5678> EV:L WinHttpRequest::OnCallback. WINHTTP_CALLBACK_STATUS_HANDLE_CLOSING.

DTrace of the EVIndexQueryServer process on the EV Index Servers (formatted for easier reading):

[1234]    (EVIndexQueryServer)    <123>    EV-H    {SearchInfoAccessor.GetValueFromSource} Loading search infos related to Archive(123ABC456DEF789GHI123ABC456DEF789GHI0000EVSiteAlias), SearcHash(X) and IndexVolume(1234ABCD12AB34CD56EF123456ABCDEF0_123456)

[1234]    (EVIndexQueryServer)    <123>    EV-H    {SqliteUtility} Exception while getting data from SQlite DB - Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 1: 'no such table: SearchInfo'.
|   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
|   at Microsoft.Data.Sqlite.SqliteCommand.d__64.MoveNext()
|   at Microsoft.Data.Sqlite.SqliteCommand.d__54.MoveNext()
|   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
|   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
|   at Symantec.EnterpriseVault.Indexing.Search.SqliteUtility.GetSearchInfos(String archiveId, String searchHash, String indexVolumeName)

[1234]    (EVIndexQueryServer)    <123>    EV-H    {ESSearch.ExecuteOptimizedSearch} Elasticsearch optimised search failed to complete search request with error: Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 1: 'no such table: SearchInfo'.
|   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
|   at Microsoft.Data.Sqlite.SqliteCommand.d__64.MoveNext()
|   at Microsoft.Data.Sqlite.SqliteCommand.d__54.MoveNext()
|   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
|   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
|   at Symantec.EnterpriseVault.Indexing.Search.SqliteUtility.GetSearchInfos(String archiveId, String searchHash, String indexVolumeName)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchInfoAccessor.GetValueFromSource(String[] keys)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchDataAccessor`1.GetValue(Boolean& fromCache, String[] keys)
|   at Symantec.EnterpriseVault.Indexing.Search.ESOptimizedSearch.GetSearchInfoMatchingToInputs(ESOptimizedSearchInputs searchInputs, List`1& searchInfoList)
|   at Symantec.EnterpriseVault.Indexing.Search.ESOptimizedSearch.GetLockObject(ESOptimizedSearchInputs searchInputs)
|   at Symantec.EnterpriseVault.Indexing.Search.ESSearch.ExecuteOptimizedSearch(ESQuery esQuery, Source source, String indexvolumeName, String index, String[] storedFields, Int32 startResultsFrom, Int32 resultsWindowSize, Boolean logQuery, Int32 timeoutInSec, String computerEntryId, Int64& totalHits, TruncationReason& truncationReason)

[1234]    (EVIndexQueryServer)    <123>    EV-M    {SearchPipeline} TraceTimer [Xms] [X/20] Completed EngineSearchCommand, search , ID , user \, source ...

[1234]    (EVIndexQueryServer)    <123>    EV-L    {SourcesInUse} Remove: Source 1234ABCD12AB34CD56EF123456ABCDEF0_123456 is no longer in use. 0 total sources in use.

[1234]    (EVIndexQueryServer)    <123>    EV-H    {SearchTask} Exception: Elasticsearch failed to complete search request Info:Search X failed, Attempt=1 Diag: Type:Symantec.EnterpriseVault.Indexing.Search.SearchExecutionFailedException ST:
   at Symantec.EnterpriseVault.Indexing.Search.ESSearch.ExecuteOptimizedSearch(ESQuery esQuery, Source source, String indexvolumeName, String index, String[] storedFields, Int32 startResultsFrom, Int32 resultsWindowSize, Boolean logQuery, Int32 timeoutInSec, String computerEntryId, Int64& totalHits, TruncationReason& truncationReason)
|   at Symantec.EnterpriseVault.Indexing.Search.ESSearch.PerformSearch(Search search, Source source)
|   at Symantec.EnterpriseVault.Indexing.Search.ESSearch.DoSearch(Search search, Source source)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchBase.Search(Search search, Source source)
|   at Symantec.EnterpriseVault.Indexing.Search.EngineSearchCommand.Execute(SearchPipelineContext context)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchPipeline.Execute(SearchPipelineContext context)|   at Symantec.EnterpriseVault.Indexing.Search.SearchTask.Execute() Inner:None

[1234]    (EVIndexQueryServer)    <123>    EV-L    {SearchTaskThread} TraceTimer [Xms] ThreadMain: Search execution has completed

[1234]    (EVIndexQueryServer)    <123>    EV-H    {SearchTaskProcessor} Search X failed, but not due to 'too busy' or intermittent error, so not re-queuing.

[1234]    (EVIndexQueryServer)    <567>    EV-H    {IndexQueryServerSearch} Exception: Elasticsearch failed to complete search request 
Info:Exception occurred during the execution of the pipeline for source 123ABC456DEF789GHI123ABC456DEF789GHI0000EVSiteAlias/X 
Diag: Type:Symantec.EnterpriseVault.Indexing.Search.SearchExecutionFailedException 
ST:   at Symantec.EnterpriseVault.Indexing.Search.ESSearch.ExecuteOptimizedSearch(ESQuery esQuery, Source source, String indexvolumeName, String index, String[] storedFields, Int32 startResultsFrom, Int32 resultsWindowSize, Boolean logQuery, Int32 timeoutInSec, String computerEntryId, Int64& totalHits, TruncationReason& truncationReason)
|   at Symantec.EnterpriseVault.Indexing.Search.ESSearch.PerformSearch(Search search, Source source)
|   at Symantec.EnterpriseVault.Indexing.Search.ESSearch.DoSearch(Search search, Source source)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchBase.Search(Search search, Source source)
|   at Symantec.EnterpriseVault.Indexing.Search.EngineSearchCommand.Execute(SearchPipelineContext context)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchPipeline.Execute(SearchPipelineContext context)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchTask.Execute() 
Inner:None

[1234]    (EVIndexQueryServer)    <567>    EV-H    {SearchService} Exception: Elasticsearch failed to complete search request 
Info:SearchService.Search failed. 
Diag: Type:Symantec.EnterpriseVault.Indexing.Search.SearchExecutionFailedException 
ST:   at Symantec.EnterpriseVault.Indexing.Search.IndexQueryServerSearch.HandleErrors(SearchTask task)
|   at Symantec.EnterpriseVault.Indexing.Search.IndexQueryServerSearch.Search(Search search, String callerLogin)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchService.Search(Search search) 
Inner:None

[1234]    (EVIndexQueryServer)    <567>    EV-L    {IndexVolumes} GetIndexVolumeInfo from cache: /123ABC456DEF789GHI123ABC456DEF789GHI0000EVSiteAlias/123456

[1234]    (EVIndexQueryServer)    <567>    EV~W    
|Event ID: 41315 A search failed with error "Elasticsearch failed to complete search request".
|Archive:
|Archive ID: 123ABC456DEF789GHI123ABC456DEF789GHI0000EVSiteAlias
|Index volume ID: 123456
|User: \
|Maximum results: 10000
|Timeout (seconds): 3600
|Sort order: snum

[1234]    (EVIndexQueryServer)    <567>    EV-M    {SearchTracking} [TRACKING] OUTGOING SEARCH RESPONSE ID=[1234ABCD-12AB-34CD-56EF-123456ABCDEF] DESTINATIONSERVER=unknown DESTINATIONPROCESS=unknown INDEXVOLUME=unknown

[1234]    (EVIndexQueryServer)    <567>    EV-H    {IndexServerSearch} Exception: Elasticsearch failed to complete search request 
Info:Search2 failed. 
Diag: Type:Symantec.EnterpriseVault.Indexing.Search.SearchExecutionFailedException 
ST:   at Symantec.EnterpriseVault.Indexing.Search.IndexQueryServerSearch.HandleErrors(SearchTask task)
|   at Symantec.EnterpriseVault.Indexing.Search.IndexQueryServerSearch.Search(Search search, String callerLogin)
|   at Symantec.EnterpriseVault.Indexing.Search.SearchService.Search(Search search)
|   at Symantec.EnterpriseVault.Indexing.Search.IndexServerSearch.Search2(String searchEntityId, String query, Int32 startResult, Int32 maximumResults, Int32 options, String sortBy, Int32 resultsPropertySet, String extraResultProperties, String resultsProcessingInstruction, Int32 resultsFormatOptionsNOTUSED, String authToken, Int32 timeoutSecs) 
Inner:None

[1234]    (EVIndexQueryServer)    <567>    EV-H    {IndexServerSearchService} Exception: Elasticsearch failed to complete search request 
Info:Search via HTTP/POST failed. 
Diag:HRESULT: c0041c0e Type:System.Runtime.InteropServices.COMException 
ST:   at Symantec.EnterpriseVault.Indexing.Search.IndexServerSearch.Search2(String searchEntityId, String query, Int32 startResult, Int32 maximumResults, Int32 options, String sortBy, Int32 resultsPropertySet, String extraResultProperties, String resultsProcessingInstruction, Int32 resultsFormatOptionsNOTUSED, String authToken, Int32 timeoutSecs)
|   at Symantec.EnterpriseVault.Indexing.Search.IndexServerSearchService.Search(Stream postBodyStream) 
Inner:None

 

Cause

From version 14.4 onwards, there is a new Accelerator Configuration Setting that optimizes Searches for ElasticSearch (ES) indices:

Name: Enable optimized searches on Elasticsearch index volumes
Section: Search
Description: If true, enabled the optimzed searches on ElasticSsearch Index Volumes in Enterprise Vault Archives.
Default Value: Enabled.

This Setting makes use of a temporary SQLite database created specifically for the Search. The error is caused by a missing table in the temporary database. Causes can include issues encountered when creating the database's schema, a corrupted database file, multi-threading issues whereby the thread that writes to the file runs before the thread that creates the database schema, antivirus, etc.

 

Resolution

To temporarily work around the issue and allow the Search to complete:

1. Open the Accelerator Client with an account having rights to edit Configuration Settings, such as the Vault Service Account (VSA), and navigate to Configuration | Settings | Search.

2. Disable the Enable optimized searches on Elasticsearch index volumes Setting.

3. Acknowledge the prompt to restart services, Customer Background Tasks, etc.

4. Restart the Enterprise Vault Accelerator Manager Service.

5. Run a new Search or Resubmit the Search.

 

To resolve the issue:

1. On the EV Indexing Server servicing the Archive in question:

1.1. Stop the EV Indexing Service.

1.2. Locate and delete the file SQLiteCache__ from the Cache location, typically at \EVStorage\Cache\Search\ESOptimizedSearchCache\\. It is safe to delete this file as it is just a cache file. Once the file is deleted, another file will get created when a Search is run on the Archive.

1.3. Start the EV Indexing Service.

2. On the Accelerator Server:

2.1. Open the Accelerator Client with an account having rights to edit Configuration Settings, such as the Vault Service Account (VSA), and navigate to Configuration | Settings | Search.

2.2. Check the Enable optimized searches on Elasticsearch index volumes Setting and enable it if disabled.

2.3. Acknowledge the prompt to restart services, Customer Background Tasks, etc.

2.4. Restart the Enterprise Vault Accelerator Manager Service.

2.5. Run a new Search or Resubmit the Search.

 

Issue/Introduction

Enterprise Vault (EV) Compliance Accelerator (CA) and Discovery Accelerator (DA) Accelerator searches fail with the error and Event Log entries listed below.

Additional Information

JIRA: CFT-6002
Powered by Wolken Software