Storage Service stops and logs Event ID 41584 after removing Compliance Accelerator

Description

Error Message

Log Name: Veritas Enterprise Vault Source: Enterprise Vault Event ID: 41619 Task Category: Compliance Sampler Level: Information Description: A transient error is preventing the execution of a SQL command. Enterprise Vault will try to run the command again. Command: 'EVSqlConnection: DB: Data Source=;Initial Catalog=;Integrated Security=True;Max Pool Size=200;Connect Timeout=120;Load Balance Timeout=30' Attempt: X of 40. Last Error: Cannot open database "" requested by the login. The login failed. Login failed for user 'domain\Vault Service Account'.

V-437-41619

Log Name: Veritas Enterprise Vault Source: Enterprise Vault Event ID: 41596 Task Category: Compliance Sampler Level: Warning Description: Failed to refresh the Compliance Sampling configuration from the Compliance Accelerator configuration database ''. Compliance Sampling will continue to use its existing configuration for a set period, if available (6 hours by default). After this period Enterprise Vault will stop the Storage service if any attempt is made to archive an item to ensure that no items are precluded from sampling. Ensure that the database is correctly configured in the Accelerator Manager website and is online and accessible to the Vault Service account. SQL Instance: Database: Error: Cannot open database "" requested by the login. The login failed. Login failed for user 'domain\Vault Service Account'.

V-437-41596

Log Name: Veritas Enterprise Vault Source: Enterprise Vault Event ID: 41583 Task Category: Compliance Sampler Level: Error Description: An attempt was made to sample an item to the Compliance Accelerator customer database ''. This database has been inaccessible for more than 6 hours, so the item has not been archived. Ensure that the database is correctly configured in the Accelerator Manager website and is online and accessible to the Vault Service account. Enterprise Vault will stop the Storage service to ensure that no items are precluded from sampling SQL Instance: Database: Error: Failed to initialise customer data in time.

V-437-41583

Log Name: Veritas Enterprise Vault Source: Enterprise Vault Event ID: 41584 Task Category: Compliance Sampler Level: Error Description: An attempt was made to sample an item, but Compliance Sampling cannot access a Compliance Accelerator configuration database. The item has not been archived. Enterprise Vault will stop the Storage service to ensure that no items are precluded from sampling. See previous events in the log for details. SQL Instance: Database:

V-437-41584

Log Name: Veritas Enterprise Vault Source: Enterprise Vault Event ID: 6222 Task Category: Storage Server Level: Information Description: Storage Service stopped

V-437-6222

Log snippets from EV server DTrace of the StorageOnlineOpns and JournalTask processes:

[1234] (StorageOnlineOpns) <5678> EV:H {AcceleratorSampling::ReadAcceleratorSamplingProps} (Exit) Status: [Exception] [1234] (StorageOnlineOpns) <5678> EV:H {CSimpleStore::ReadAcceleratorSamplingProps} (Exit) Status: [Exception] [1234] (StorageOnlineOpns) <5678> EV:H {CSimpleStore::PerformQueueSaveset} (Exit) Status: [Exception] [1234] (StorageOnlineOpns) <5678> EV:L CSavesetPersist::ResetStorage LB: 0x4756d88 Stg(LB): 0xb63e00 Stg(doc): 0 File: (null) [1234] (StorageOnlineOpns) <5678> EV:H {CEVByteStream::Flush} (Exit) Status: [Exception] [1234] (StorageOnlineOpns) <5678> EV:H {CSimpleStore::QueueSaveset3:#X} _com_error exception: [<0x80131600>] [1234] (StorageOnlineOpns) <5678> EV:H {CSimpleStore::QueueSaveset3} (Exit) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:H {QueueSavesetCommand::QueueSaveset} (Exit) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:H {QueueSavesetCommand::Execute:#X} _com_error exception: [<0x80131600>] [1235] (JournalTask) <5679> EV:H {QueueSavesetCommand::Execute} (Exit) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:L {EVCommandExecutor::ExecuteRecoverWithRetry} (Entry) [1235] (JournalTask) <5679> EV:H {QueueSavesetCommand::Recover:#X} Resetting item contents [1235] (JournalTask) <5679> EV:L {QueueSavesetCommand::QueueIsUnavailable} (Entry) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:L {QueueSavesetCommand::QueueIsUnavailable} (Exit) [1235] (JournalTask) <5679> EV:H {SavesetMarshaller::ResetSeekPointer:#X} If this is a saveset stream then seek to beginning. Variant type: X [1235] (JournalTask) <5679> EV:L {EVCommandExecutor::ExecuteRecoverWithRetry} (Exit) Status: [Success] [1235] (JournalTask) <5679> EV:L {QueueSavesetCommand::QueueIsUnavailable} (Entry) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:L {QueueSavesetCommand::QueueIsUnavailable} (Exit) [1235] (JournalTask) <5679> EV:H {EVCommandExecutor::ExecuteCommand} (Exit) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:L CAuthHelper::Reset Cancel registration? True CancelId: 0 [1235] (JournalTask) <5679> EV:H {CAutoStorageOnline::QueueSaveset:#X} _com_error exception: [<0x80131600>] [1235] (JournalTask) <5679> EV:H {CAutoStorageOnline::QueueSaveset} (Exit) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:H {CStorageArchive::QueueSaveset} Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:H {CStorageArchive::Store:#X} _com_error exception: [<0x80131600>] [1235] (JournalTask) <5679> EV:H {CStorageArchive::Store} (Exit) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV~E Event ID: 3433 Could not build a saveset for the message titled [] in Mailbox [...], function [IStorageArchiveV80::Store()] failed. |Error: 0x80131600 |Description: Unknown error 0x80131600 | [1235] (JournalTask) <5679> EV:L {CArchivingAgent::PI_FinishBuildingSaveset} (Exit) Status: [<0x80131600>] [1235] (JournalTask) <5679> EV:H {CArchivingAgent::ProcessItemInternal:#X} Call to PI_FinishBuildingSaveset Failed RetryCount[0] [0x80131600] [1235] (JournalTask) <5679> EV~I :CArchivingAgent::ProcessItem() |Something has gone wrong and this has been our last retry. We will undo the archive pending message in the users mailbox, so leaving it as a normal message again |

Cause

This behaviour is by design. CA uses the EV Storage Service to Department Tag items so they may be processed by CA Random Sampling. To do this, the EV Storage Service synchronises with CA at regular intervals to read the Department Tagging data. The starting point for this synchronisation is the CA Configuration database, information for which is stored in the EnterpriseVaultDirectory database's AcceleratorConfigEntry table. After the Storage Service connects to a CA Configuration database, it reads the available Customer database information, then contacts each CA Customer database to obtain each Customer's Department Tagging information (Departments, Monitored Employees and Policy percentages).

If the synchronisation process is not able to connect to the CA Configuration database to refresh the Department Tagging data, the previously known data is used. If CA still cannot be contacted after a certain number of consecutive unsuccessful synchronisation attempts, EV will shut down the Storage Service to prevent potentially outdated Department Tagging data from being used. This shutdown is designed to alert the administrator to investigate why EV Storage cannot communicate with the CA Configuration database, or to take the necessary manual actions to remove any non-existent CA environments' information from the EnterpriseVaultDirectory database's AcceleratorConfigEntry table.

Resolution

In order to properly remove (de-commission) CA from the environment, it is required to manually remove any corresponding CA synchronisation information from the AcceleratorConfigEntry table in the EnterpriseVaultDirectory database, as described in the Compliance Accelerator Installation Guide provided in the product documentation, under the section titled 'Uninstalling Compliance Accelerator'. The proper uninstallation/de-commissioning sequence and steps are as follows:

1. Uninstall all CA Client software.
2. Uninstall the CA Server and EV software from the CA Server.
3. Stop the Enterprise Vault Storage Services on all EV Storage Servers in the EV Site. Backup the EnterpriseVaultDirectory database and all Vault Store databases before proceeding to the next steps.
4. Manually remove the CA Configuration database synchronisation information from the EnterpriseVaultDirectory database.
4.1. Review the current information in the table by executing the following SQL query against the EnterpriseVaultDirectory database:

SELECT * FROM AcceleratorConfigEntry;

4.2. If only one CA environment exists and has been de-commissioned as above, remove its entry by executing the following SQL query against the EnterpriseVaultDirectory database:

DELETE FROM AcceleratorConfigEntry;

4.3. If multiple CA environments exist and one or more were de-commissioned, review the AcceleratorConfigEntry table information and determine the valid and the de-commissioned CA Configuration databases by reviewing the 'InitialCatalog=' section in the DSN column. Once confirmed, obtain the ConfigID for the de-commissioned CA Configuration database(s) and remove each associated entry by editing the ConfigID as indicated and executing the following SQL query against the EnterpriseVaultDirectory database (repeat for each de-commissioned CA Configuration database's ConfigID):

DELETE FROM AcceleratorConfigEntry WHERE ConfigID = ''; -- Edit ConfigID here

An example ConfigID looks like: ABCD1234-AB12-CD34-EF56-ABCDEF123456.

5. Manually remove the corresponding items' entries from the ItemSamplingStatusLog table in each Vault Store database.
5.1. Execute the following query against each Vault Store database:

SELECT DISTINCT CustomerGUID, COUNT(SavesetIdentity) AS 'Item Count' FROM ItemSamplingStatusLog GROUP BY CustomerGUID ORDER BY CustomerGUID;

5.2. If only one CA environment exists, clear the entries from the table by executing the following SQL query against each Vault Store database database:

DELETE FROM ItemSamplingStatusLog;

5.3. If multiple CA environments exist and one or more were de-commissioned, review the existing CustomerGUID entries and remove those corresponding to de-commissioned CA environments.
5.3.1. Identify the de-commissioned CA environments' CustomerGUID entries by comparing the above query's results with the following SQL query executed against each existing and de-commissioned (if available) CA Configuration Database:

SELECT CustomerID, Name, Server AS 'SQL: Server', InitialCatalog AS 'Customer Database', CustomerGUID FROM tblCustomer WHERE Type = 50 ORDER BY CustomerID;

5.3.2. Remove the ItemSamplingStatusLog information corresponding to the de-commissioned CA environments' CustomerGUIDs by editing the following SQL queries as indicated and executing against each Vault Store database:

For a single CustomerGUID:

DELETE FROM ItemSamplingStatusLog WHERE CustomerGUID = ''; -- Edit de-commissioned CustomerGUID here

For multiple CustomerGUIDs:

DELETE FROM ItemSamplingStatusLog WHERE CustomerGUID IN ( -- Edit de-commissioned CustomerGUIDs below (only the first line does not need a leading comma) '' , '' , '' );

An example CustomerGUID looks like: ABCD1234-AB12-CD34-EF56-ABCDEF123456.

6. Start the EV services, monitor the Storage Service and monitor the Event Logs.

7. Backup the de-commissioned CA Configuration and Customer databases in the event they may be needed in the future. Optional - The CA Configuration and/or Customer databases may then be removed from the SQL server as needed.

Issue/Introduction

After removing Enterprise Vault (EV) Compliance Accelerator (CA) from an EV environment, the EV Storage Service stops and logs warnings and errors in the EV Event Logs. Attempting to restart the service causes it to shut down again with the same entries. The EV Event Logs may also contain numerous informational entries prior to the warnings and errors. In some cases, EV may stop Archiving items and list Event ID 3433 in the Event Logs. A DTrace of the StorageOnlineOpns and JournalTask processes on the EV server may indicate exceptions referring to error 0x80131600, which is a COR_E_APPLICATION exception.

Welcome to "KB Articles"