Error Message

The application event log will show the following event over and over again in the log. The unique aspects of this event that are particular to this issue are bolded. 

Type :        Error
Event :        1000
Source :        Application Error
Category :    Application Crashing Events
Computer :    EVServer1.EV.Local
Description:
Faulting application name: StorageCrawler.exe, version: 12.0.0.1528, time stamp: 0x5674788d
Faulting module name: MSVCR120.dll, version: 12.0.21005.1, time stamp: 0x524f7ce6
Exception code: 0xc0000409
Fault offset: 0x000a46a9
Faulting process id: 0x2c60
Faulting application start time: 0x01d4a2b27ac79b13
Faulting application path: E:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe
Faulting module path: C:\Windows\SYSTEM32\MSVCR120.dll
Report Id: ba226f9b-0ea5-11e9-8162-005056814a22
Faulting package full name: ?
Faulting package-relative application ID: ?

Note:  Other errors in the Enterprise Vault log may include:

Event ID: 41352

The processing of the Index Volume has stopped following errors.  Archive name: Journal Archive Task: Index Volume ID: 1A587BE1BB65BDE4D8E93BE9463BDB11B1110000VAULT_1
Reason:
Error Type: NonCritical
Description: Storage Crawler is not accessible. Archive Id=[1A587BE1BB65BDE4D8E93BE9463BDB11B1110000VAULT]
HRESULT=0x800706BA
                                                                                                                                                                      
Event ID: 28998

( Symantec.Diag.Framework.Grid.Url ) Retrieval of a saveset failed.
Reason: The docfile has been corrupted.  (0x80030109)  Saveset Id: 201811303538608~201811292310280000~Z~00AC049B4E276EFD0DD6E0BD4A11ECB1  Vault Id: 1A587BE1BB65BDE4D8E93BE9463BDB11B1110000VAULT  Index Sequence Number: 1
Internal Reference: Fetching Saveset for indexing .

v-437-41352

v-437-28998

[5404]    (StorageCrawler)         EV:L    {CSavesetOnIStg2::get_ConvertedContentDisposition} (Entry)[5404]    (StorageCrawler)         EV:L    {CSavesetOnIStg2::OpenAttachmentMetadata} (Entry)[5404]    (StorageCrawler)         EV:L    {CSavesetOnIStg2::CommitAndReleaseCurrentAttachmentMetadata} (Entry)[5404]    (StorageCrawler)         EV:L    {CSavesetOnIStg2::CommitAndReleaseCurrentAttachmentMetadata} (Exit) Status: [Success][5404]    (StorageCrawler)         EV:L    {CSavesetOnIStg::OpenSubStg} (Entry) Element [1][5404]    (StorageCrawler)         EV:L    {CSavesetOnIStg::OpenSubStg} (Exit) Status: [Success][5404]    (StorageCrawler)         EV:L    {CSavesetOnIStg2::OpenAttachmentMetadata} (Exit) Status: [Success][5404]    (StorageCrawler)         EV:L    {CSavesetOnIStg2::get_ConvertedContentDisposition} (Exit) Status: [Success][5404]    (StorageCrawler)         EV:L    {CSaveset2::get_ConvertedContentDisposition} (Exit) Status: [Success]

If you scroll up to the dtrace you will see which part is causing the problem.

[5404]    (StorageCrawler)         EV:M    Reading part from: \\vjournal\journal2017\VaultStore\JournalVS\Ptrn1\2018\11-26\7\052\7052F00C68B02EC0BBFCEFBCCE2C2671.DVS[5404]    (StorageCrawler)         EV:M    Reading part from: \\vjournal\journal2017\VaultStore\JournalVS\Ptrn1\2018\11-26\7\052\7052F00C68B02EC0BBFCEFBCCE2C2671~83~192A99EB~00~1.DVSSP

These items can be renamed to old on disk so that storagecrawler will fail to find them and proceed on to other non-problematic items. Again this may have to be done to more than one item. 

Be sure to keep track of which items have been renamed as they will have to be dealt with later. 

This issue can occur is there is a minor corruption of the attachment metadata. Specifically the Converted Content Disposition and OriginalAttachmentSize for one or more attachments. 

If you are experiencing this issue it is recommended to upgrade to Enterprise Vault 12.4.2 or 12.5.2 to resolve the issue. These versions are available in the Downloads portal. 

If you are on Enterprise Vault 12.4 and earlier and are not able to upgrade immediately an temporary work around can be provided for each occurrence. Please follow the Preliminary Investigation Steps below: 

Preliminary Investigation Steps:

If you are experiencing this issue it is recommended to gather the recommended evidence and submit via a support case. To collect the recommended evidence please perform the following steps:

• Stop the Storage Service
• Configure Windows Error Reporting to generate full dump files per the following Tech Note:
  • TN: https://docs.microsoft.com/en-us/windows/desktop/wer/collecting-user-mode-dumps
  • Set DumpType = Full
  • Set the DumpFolder = To a folder on the system that has sufficient disk space
• Start a Dtrace of the StorageCrawler process see the following Tech note for detailed steps.
  • https://www.veritas.com/content/support/en_US/article.100038975.html
• Start the Storage Service and allow to run until Event ID shows up in the Application log.
• At that point collect the generated Crash dump file and Dtrace and submit to support for further investigation. 

If you need assistance with any of the steps provided above please feel free to open up a support case. 

Note: It has also been seen that malicious attachments on emails are causing the process to fault.  These attachments will typically have very long names or extensions.  If it is possible to identify the sender and block emails the issue should be avoided.