Error Message

Log Name: Symantec Enterprise Vault
Source: Accelerator Service Processor
Event ID: 285
Task Category: None
Level: Error
Description: APP AT - Customer ID: X - An Error has occured while bulk inserting search items. System.Data.SqlClient.SqlException: Cannot insert duplicate key row in object 'dbo.tblSearchItems' with unique index 'IX_UNQ_CaseIDSearchIDKVSSavesetID'. The duplicate key value is (AA, BB, ).
ERROR_usp_SearchItem_Bulk_Add
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteScalar()
   at KVS.Accelerator.Common.TraceDLCommand.ExecuteScalar(IDbCommand command, Boolean RetryDLs)
   at KVS.Accelerator.Common.TraceDLCommand.ExecuteScalar(IDbCommand command)
   at KVS.Accelerator.Search.SearchItems.InsertBulkSearchItem(SearchResults results)
V-437-285

Log Name: Symantec Enterprise Vault
Source: Accelerator Service Processor
Event ID: 296
Task Category: None
Level: Error
Keywords: Classic
Description:
APP AT - Customer ID: X - Error processing bulk search items: Unrecoverable error bulk inserting search items. System.Data.SqlClient.SqlException: Cannot insert duplicate key row in object 'dbo.tblSearchItems' with unique index 'IX_UNQ_CaseIDSearchIDKVSSavesetID'. The duplicate key value is (AA, BB, ).
ERROR_usp_SearchItem_Bulk_Add
The statement has been terminated.
   at KVS.Accelerator.Search.SearchItems.InsertBulkSearchItem(SearchResults results)
   at KVS.Accelerator.Search.SearchQueues.SearchResultsProcessWork(ThreadSafeQueue& theQueue, Object queuedItem)
V-437-296

This issue can occur if either of the following conditions exist:

1. SQL maintenance is being performed on the DA Customer Database Transaction Logs without first Stopping the Enterprise Vault Accelerator Manager Service (EVAMS). This causes the Acceptance or Rejection process to be repeatedly processed while the original process is still in progress, causing the "Cannot insert duplicate key row in object" errors.
2. One or more malformed index entries exist in one or more index volumes where the KVSSavesetID value of the items either does not exist or is duplicated within each such item's entry.  This lack of or duplicated KVSSavesetID entry causes the DA search processing of the XML file containing the search result hits from the index volume to see an empty value for the KVSSavesetID of any such item.  Each item that is processed from the XML file is placed into the tblSearchItems table until such time as the search is accepted or rejected.  Each entry in the tblSearchItems table for a search must have a KVSSaveset that is unique from other entries for the search.  Having two or more entries with empty KVSSavesetID values violates a constraint setting, thus causing the Event ID 285 and 296 errors to be thrown.

Each set of Event ID 285 / Event ID 296 entries will indicate the CaseID, the SearchID, and the KVSSavesetID for the item causing the duplicate entry. Reviewing the "The duplicate key value is" lines in the error examples above, AA is the CaseID and BB is the SearchID.  The examples above do not contain a KVSSavesetID, which should follow the SearchID, as this entry is missing and is causing the errors to be thrown.

To stop the Event ID 285 / Event ID 296 entries:

1. The Acceptance and/or Rejection process for each SearchID listed in the events must be completed. This is accomplished by Restarting the EVAMS for each SearchID listed in the Events, as below:
   1. Review the oldest Event ID 285 / Event ID 296 entries and find the lines containing "The duplicate key value is". The first value will be the CaseID, the second value will be the SearchID. The oldest event entries should contain the first SearchID that triggered the events.
   2. Verify there is no user activity that could be disrupted by Restarting the EVAMS. For example, a user Reviewing items may experience errors during the EVAMS restart. Processes such as Legal Holds will be paused by DA during the EVAMS restarts and will automatically continue after the EVAMS is Started.
   3. Restart the EVAMS and monitor the Event Logs.
   4. If there are additional Event ID 285 / Event ID 296 entries after Restarting the EVAMS, review the errors to verify the SearchID is different from the SearchID noted above and continue Restarting the EVAMS and monitoring the Event Logs, one Restart per SearchID, until the Events are no longer logged. It is possible that multiple SearchIDs may be processed by a single EVAMS Restart, but this is not always the case.

To prevent this issue from recurring, the EVAMS must be Stopped during any SQL maintenance activities targeting any of the DA Databases: Configuration, Customer and Custodian (if present).

1. The index volume or volumes containing the items with the malformed KVSSavesetID entries must be rebuilt and the search or searches known to throw the errors must then be rerun.  To accomplish these actions, the following steps can be used to identify the affected items and the index volume or volumes in which they exist.
   1. Stop the search or searches that are causing the errors to be thrown.  If needed, the description field of the errors can be viewed to identify the SearchID value as noted in the Cause section able.  The SearchID can be used to lookin the DA Customer database's tblIntSearches table to identify the name of the search.  Likewise, the CaseID can be identified as noted above and the tblCase table queried to identify the name of the case in which the search was run.
   2. Enable the following 2 configuration settings by placing a check mark in the check boxes in their Value field in the CA Client's Configuration tab, Settings sub-tab, Diagnostics section:
      1. Save XML Search Items To Commit
      2. Save XML Search Results
   3. Restart the Enterprise Vault Accelerator Manager Service (EVAMS) on the DA server after saving these configuration changes.
   4. Run the search known to throw the errors again until after the errors start to be thrown, then stop and reject the search.
   5. Look on the DA server in the DA installation folder (default path is 'C:\Program Files (x86)\Enterprise Vault Business Accelerator') for the following folders:
      1. SearchItemToCommit
      2. SearchResults
   6. Look in the SearchItemToCommit folder for another folder that has the same name as the DA Customer (i.e., 'C:\Program Files (x86)\Enterprise Vault Business Accelerator\SearchItemsToCommit\DACustomer').
   7. Look in this customer folder and note the files that have the name format of 'Items_S#_V#_I#_C#_B{Archive Name}.xml', where S# is the letter S followed by the SearchID, V# is the letter V followed by VaultID containing the archive searched, I# is the letter I followed by the EVIndexVolumeSetID of the index volume searched, C# is the letter C followed by a sequencial number for the result set in the file (i.e, C2, C3, C4, etc), and B{ARCHIVENAME.EN_US} is the letter B followed by the name of the archive (i.e., Items_S1773_V32_I3_C2_BJournal Archive.xml).
   8. Search inside each of the Items_ files to find all instances of KVSSavesetID=""  (note this it the word KVSSavesetID followed by the equal sign and 2 double quotes with no spaces anywhere).
   9. Note all instances of the KVSSavesetID="" string as this is the number of malformed indexed items in that file.
   10. Look in the SearchResults folder for another folder that also has the same name as the DA Customer (i.e., 'C:\Program Files (x86)\Enterprise Vault Business Accelerator\SearchResults\DACustomer').
   11. Look in this customer folder and note the files that have the name format of '{NUMBER.EN_US}_Results_S#_C#_{Archive name}.xml', where the beginning number is a sequential identification number, S# is the letter S followed by the SearchID, C# is the letter C followed by the number 1 (i.e., 15539_Results_S1772_C1_Journal Archive.xml).
   12. Search inside each file associated with the archives noted in the names of the files containing the empty KVSSavesetID entries, looking for all instances of . 
   13. Note that every information segment on these lines is duplicated.  An example of such an entry is:
       1. Reading this on a mobile device? Try our optimized mobile version here: https://www.myapp.com/myappReading this on a mobile device? Try our optimized mobile version here: https://www.myapp.com/myapp201211082400817~201211081846550000~Z~C0D6960ECE2169C79EB9BC4931CAEE91201211082400817~201211081846550000~Z~C0D6960ECE2169C79EB9BC4931CAEE91InboxInbox2046563720465637SmartBrief on My MediaSmartBrief on My MediaJohn A. DoeJohn A. Doe110000johnadoe@myapp.comjohnadoe@myapp.com11Top five reasons to subscribe to another SmartBriefTop five reasons to subscribe to another SmartBrief48E607206536C74DBC442CAB3B577C5A48E607206536C74DBC442CAB3B577C5A112012-11-08T18:46:55Z2012-11-08T18:46:55Z2012-11-08T18:53:37Z2012-11-08T18:53:37Z6464MSGMSG00EXCHEXCH22
           
   14. Note the field value (20465637 in the above example) for all lines with the entries.  This is the Index Sequence Number (ISN) of each of those items in the index volume or index volumes.
   15. Look in the Vault Admin Console at the properties of the archive noted in the name of these files (i.e., Journal Archive) by right clicking on the archive name and selecting the Properties option.
   16. Click on the Index Volumes tab of the archive's properties, then click on the Details... button.
   17. Look in the Range column to see the start and end Index Sequence Number values for all items in each index volume and note all Index Volumes that have the ISN values noted in Step n above as any such Index Volume will need to be rebuilt.
   18. Close the Details window by clicking on the Close button near the window's lower right corner.
   19. Rebuild the appropriate index volume or volumes using the Manage Indexes wizard.  For more information on rebuilding indexes, see HowTo Article HOWTO57637 in the Related Articles section below. 
2. When the index volume rebuild action has completed, run new DA searches using the criteria previously known to cause the EventID 285 and 296 errors.
3. Upon completion of the new searches, complete the review and export processing as appropriate.