Error Message

The EV Event log shows the following warning:

Event ID:29061

"A problem was encountered while trying to process a saveset in the storage queue. The saveset will be requeued and one more attempt will be made to process it later. Details: Unspecified error  (0x80004005)

Storage Queue Location: X:\EVStorageQueue Storage Queue Batch File:

X:\EVStorageQueue\EVStorageQueue_1A88873545AD4CB4AA073F0CBC5BF23381210000EVSite.ev.local_20190107005604_10077.EVSQ Transaction Id: 1040218B71F861099692E6CAEE834461 Vault Store Entry Id: 1A88873545AD4CB4AA073F0CBC5BF23381210000EVSite.ev.local

A DTRACE against the StorageArchive process will show the following detail:-

uspu_Share10Fingerprints result: hrSPCall=0x80004005; retval=0x00000000; TxSuccessfulCount=0
CFingerprintCatalogueDatabase::ShareSISParts result: uspu_Share10Fingerprints failed=0x80004005.

Which will then trigger a rollback:

{CSavesetSISManager::RollbackSIS} (Entry)
Deleting unreferenced saveset...

This is caused by a SQL deadlock in the stored procedure called uspu_Share10Fingerprints which resides within the VSG database and is called as part of the archiving process. This has been seen in environments where the member table has millions of rows within it. Additional confirmation can be obtained by switching on SQL deadlock logging by running the following SQL command:

DBCC TRACEON (1204, 1222, -1)

This will send SQL Deadlock information to the SQL ERRORLOG.

Additionally, SQL Profiler can be configured to run against the vault Store Group or Fingerprint database with the Deadlock Graph Event selected. When the deadlock occurs a graph will be generated to provide further information. Below is an example:

Once analysis has been completed switch deadlock logging off with the following command:-

DBCC TRACEOFF (1204, 1222, -1)

The behavior is as follows:-

1. Item is archived.

2. A deadlock occurs in SQL server due to multiple writes occurring on the same Member Table.

3. SQL Server chooses a victim and kills the process.

4. This triggers EV to rollback archiving the item.

5. The item will be attempted again and will succeed.

Therefore, the issue does not cause any loss of data but creates a performance issue which creates the backlog on the storage queue.

Workaround

The following registry key may be implemented on each EV server in the environment. The ramifications of implementing the registry key are that when the item is retried as in Step 5 above it will no longer be eligible for Single Instance Storage. Therefore, the deadlocks will still be experienced and the event logs will still be generated but the backlog on the storage queue will be processed.

HKEY_LOCAL_MACHINE \SOFTWARE \Wow6432Node \KVS \Enterprise Vault \Storage
REG_SZ - InsertQueuedItemNonRecoverableErrors
Value (HEX) 0x80004005

Once the key is in place restart the storage service for it to take effect.

Veritas has acknowledged that the above-mentioned issue is present in the version(s) of the product(s) referenced in this article. This issue is currently under investigation by Veritas. Pending the outcome of the investigation, this issue may be resolved by way of a cumulative hotfix or service pack in the current or future versions of the software. However, this particular issue is not currently scheduled for any release.  If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Veritas Sales representative or the Veritas Sales group to discuss these concerns.  For information on how to contact Veritas Sales, please see http://www.Veritas.com.

Please be sure to refer back to this document periodically as any changes to the status of the issue will be reflected here.