Hotword Analysis processing does not start after new Hotword searches are accepted.

Description

Error Message

The log excerpts below were taken from a non-production test lab environment and are used for example purposes only.

A DTrace of the AcceleratorService64 process on the CA server will contain lines similar to the following:

[12345] (AcceleratorService64) EV-H {-} {CommonStarterEV} {C1} Failed to start AnalyticsIngester.ex. for hotword analysis.Thread was being aborted. [12345] (AcceleratorService64) EV-M {-} {AcceleratorEvent} {C1} Event logging initialised for 'Accelerator Manager'. Event log name - Veritas Enterprise Vault [12345] (AcceleratorService64) EV-H {-} Exception: Error Common_ThreadNoDelegate Info:{AcceleratorEvent} {C1} No delegate! System.Threading.ThreadAbortException: Thread was being aborted.| at KVS.Accelerator.Server.CommonStarterEV.StartProcessForHWAnalysisIfRequired()| at KVS.Accelerator.Common.ScheduledThread.WorkerThread() Diag: Type:System.Exception ST: Inner:None

Cause

A search term that is marked by the Hotword Analysis process as temporarily failed can cause the Hotword Analysis processing to not see new items to be processed. A restart of the Customer's Background Tasks (CBTs) or the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA server will allow new items to be processed for some time, but the item(s) marked as temporarily failed can cause new items to stop being processed in the near future.

Note: In the DTrace log snippets above and to follow

[12345] represents the Process ID value as seen in Task Manager
<6789> and <6789> represent the Thread ID of the process
{C1} represents the CA Customer ID number as found in the CA Configuration Database's tblCustomer table
DASQLServer represents the SQL Server name or SQL Always On Availability Group (AOAG) listener where the CA customer database is hosted
CACustomerDB represents the name of the CA Customer database

A DTtrace of the AcceleratorService64 process will have a set of the following 3 lines repeated every 3 minutes while waiting for a search with Hotwords to be accepted:

[12345] (AcceleratorService64) EV-L {CommonStarterEV} {C1} Checking if there are items available for hotword analysis [12345] (AcceleratorService64) EV-L {TraceTimer} {C1} [ 5ms] CheckIfItemsAvailableForHWAnalysis [12345] (AcceleratorService64) EV-L {CommonStarterEV} {C1} No items available for hotword analysis

When a search using Hotwords is accepted and fully into the Review Set, the DTrace will have lines similar to the following to indicate the Hotword Analysis process has been spawned:

[12345] (AcceleratorService64) EV-L {TraceTimer} {C1} [ 26ms] CheckIfItemsAvailableForHWAnalysis [12345] (AcceleratorService64) EV-L {CommonStarterEV} {C1} Items are available for hotword analysis [12345] (AcceleratorService64) EV-L {CommonStarterEV} {C1} Spawning AnalyticsIngesterTask.exe for hotword analysis [12345] (AcceleratorService64) EV-L {-} {CommonStarterEV} {C1} Customer DB Server = DASQLServer : [12345] (AcceleratorService64) EV-L {-} {CommonStarterEV} {C1} Customer Db = CACustomerDB: [12345] (AcceleratorService64) EV-L {-} {ProcessRunner} {C1} Spawning new process : C:\Program Files (x86)\Enterprise Vault Business Accelerator\AnalyticsIngesterTask.exe [12345] (AcceleratorService64) EV-H {-} {ConfigSettings} {C1} 31 configuration keys loaded for Customer 0 of type Undefined

In normal processing, the above lines will be followed by lines similar to the following:

[12345] (AnalyticsIngesterTask) <6789> EV-M {-} {AnalyticsIngesterApp} {C1} (TransactionId: -1) New Stop event was created [12345] (AnalyticsIngesterTask) <6789> EV-M {-} {AnalyticsIngesterApp} {C1} (TransactionId: -1) Old Stop event was returned [12345] (AnalyticsIngesterTask) EV-H {-} {TasksModuleInfo} {C1} Starting Customer Synchronisation thread

A successful completion of the Hotword Analysis processing will be followed by lines similar to the following:

[12345] (AnalyticsIngesterTask) <6789> EV-H {-} {AcceleratorEvent} {C1} INFORMATION EVENT: ID=Ingestion_Task_Completed_Info, Message=AnalyticsIngesterTask.exe (PID = 6548) with TransactionId -1 successfully completed. [12345] (AnalyticsIngesterTask) <6789> EV-M {-} {AnalyticsIngesterApp} {C1} (TransactionId: -1) Old Stop event was returned [12345] (AnalyticsIngesterTask) <6789> EV-M {-} {AnalyticsIngesterApp} {C1} (TransactionId: -1) Stop event was set [12345] (AnalyticsIngesterTask) EV-H {AnalyticsIngester} {C1} (TransactionId: -1) Exception in GetItemsToRetrieveFromDB. Exception: System.Threading.ThreadAbortException: Thread was being aborted.| at Symantec.Accelerator.Analytics.AnalyticsIngester.StopIngestion(IngestionStopCode stopCode)| at Symantec.Accelerator.Analytics.AnalyticsIngester.GetItemsToRetrieveFromDB(MultiQueuesProcessor tmQueue, Object customData) [12345] (AnalyticsIngesterTask) EV-H {-} {TasksModuleInfo} {C1} Stopping Customer Synchronisation thread [12345] (AnalyticsIngesterTask) <6789> EV-L {TraceTimer} {C1} [ 0ms] MultiQueuesProcessor: Shutting down 6 threads (RetrieveECMItem)

These AnalyticsIngesterTask lines will be followed by the repeating set of 3 lines noted above.

When an item is stuck in the Hotword Analysis processing, the AnalyticsIngesterTask process does not shut down as it loops on any items failing analysis and in a temporarily failed status, and will not see any new items to process.

The status of items to be processed can be seen in the CA Customer database in table tblIntHotwordAnalysisStatus using the following SQL query:

SELECT [Search_ID] = tdits.SearchID
, [Search_Run_Date] = tis.RunDate
, [Search_HWAnalysisStatusID] = tis.HWAnalysisStatusID
, [Search_HWAnalysisStatus] = ts0.Name
, [Item_Vault_ID] = tihas.VaultID
, [Item_Archive_Name] = tv.KVSVaultName
, [Item_Vault_Server_Name] = tvs.VaultServer
, [DiscoveredItemID] = tihas.DiscoveredItemID
, [Item_Saveset_ID] = tihas.KVSSaveSetID
, [Item_Hotword_Analysis_Status_ID] = tdits.HotwordAnalysed
, [Item_Hotword_Analysis_Retry_Count] = tihas.RetryCount
, [Item_Hotword_Analysis_Last_Update_Date] = tihas.LastUpdated
, [Item_HWAnalysisStatusID] = tihas.HWAnalysisStatus
, [Item_HWAnalysisStatus] = ts1.Name
, [Item_HWAnalysisStatusType] = ts1.Type
, [Item_HWAnalysisStatusName] = ts1.Name
, [Item_HWAnalysisStatusResourceID] = ts1.ResourceID
, [Item_Hotword_Analysis_Error_Code] = tihas.HWAnalysisErrorCode
, [Item_Hotword_Analysis_Error_Message] = tihas.HWAnalysisError
FROM tblIntHotwordAnalysisStatus (NOLOCK) AS tihas
INNER JOIN tblDiscoveredItemToSearch (NOLOCK) AS tdits ON tihas.DiscoveredItemID = tdits.DiscoveredItemID
INNER JOIN tblIntDiscoveredItems (NOLOCK) AS tidi ON tdits.DiscoveredItemID = tidi.DiscoveredItemID
INNER JOIN tblIntSearches (NOLOCK) AS tis ON tdits.SearchID = tis.SearchID
INNER JOIN tblVaults (NOLOCK) AS tv ON tihas.VaultID = tv.VaultID
INNER JOIN tblVaultServers (NOLOCK) AS tvs ON tv.VaultServerId = tvs.VaultServerId
INNER JOIN tblStatus AS ts0 ON tis.HWAnalysisStatusID = ts0.StatusID
INNER JOIN tblStatus AS ts1 ON tihas.HWAnalysisStatus = ts1.StatusID
--WHERE tdits.SearchID = X -- Use this line for a SearchID
--WHERE tihas.DiscoveredItemID = X -- Use this line for one DiscoveredItemID
--WHERE tihas.KVSSaveSetID = 'X' -- Use this line for one KVSSavesetID
ORDER BY tdits.SearchID DESC, tihas.HWAnalysisStatus;

Review the query output for the following:

- Search_HWAnalysisStatusID and Search_HWAnalysisStatus.
- Item_HWAnalysisStatusID, Item_HWAnalysisStatus, Item_HWAnalysisStatusType.

If these values indicate the Search Hotword Analysis is Completed or Completed with errors, and indicate the item's Hotword Analysis is TemporarilyFailed, this can cause the issue. Items found to be in the TemporarilyFailed state can cause the Hotword Analysis processing to attempt to process them repeatedly approximately every 3 minutes and prevent new items from being processed until the restart of the CBTs or the EVAMS.

Resolution

Items marked in the tblIntHotwordAnalyisStatus table in the CA Customer database as TemporarilyFailed must be set to the PermanentlyFailed status, followed by a restart of the CBTs or the EVAMS.

Contact Veritas Technical Support for assistance in confirming this issue is present and implementing the resolution.

Issue/Introduction

Hotword Analysis processing remains in the Pending status for new Hotword searches after being accepted within Compliance Accelerator (CA).

Welcome to "KB Articles"