Error Message

The Enterprise Vault (EV) Event Logs will show the following: 

Log Name: Veritas Enterprise Vault
Source: Enterprise Vault
Event ID: 40966
Task Category: Index Admin Service
Level: Error
Keywords: Classic
Description:
A program fault has raised an exception.

Exception: The exception [search-Index Volume-invalid-name] was thrown.
Diagnostic:
Type: System.ServiceModel.FaultException`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]
Reference:

Command Line: "C:\Program Files (x86)\Enterprise Vault\\EVIndexAdminService.exe" -EntryID:123ABC456DEF789GHI123ABC456DEF789GHI0000EVSiteAlias
Application Domain: EVIndexAdminService.exe
Process Id: 11552
Thread Id: 3004
Stack Trace: at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.ReThrowAsTypedFaultExceptionIfAppropriate(FaultException faultEx)
at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.Invoke[TResult](Expression`1 method, String server)
at Symantec.EnterpriseVault.Indexing.Common.IndexingWCFProxy`1.Invoke[TResult](Expression`1 method)
at Symantec.EnterpriseVault.Indexing.IndexingEngine.VelocityProxy.SearchIndex VolumeXml(String Index VolumeName, Boolean bStaleOk)
at Symantec.EnterpriseVault.Indexing.IndexingEngine.VelocityIndexingEngine.GetIndexVolumeLocation(IIndexVolumeIdentifier indexVolumeIdentifier)
at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.FrozenColumnExists(VelocityProxy proxy, String Index VolumeName)
at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.HandleErrorsDueToOldSchemaVersion(ErrorInfo Error)
at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.HandleErrorBehaviour(ErrorInfo Error)
at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.EnumErrors()
at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.ErrorMonitoringTask()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

V-437-40966

DTrace of the EVIndexAdminService process will show the following: 

26720    10:08:38.223     [11552]    (EVIndexAdminService)    <3004>    EV-L    {IndexingWCFProxy`1} Calling SearchCollectionXml
 

26721    10:08:38.266     [11552]    (EVIndexAdminService)    <3004>    EV-L    {IndexingWCFProxy`1} TraceTimer [44ms] Called SearchCollectionXml
 

26722    10:08:38.271     [11552]    (EVIndexAdminService)    <3004>    EV-L    {IndexingWCFProxy`1} SOAP FAULT: detail xml = The collection 123ABC456DEF789GHI123ABC456DEF789_123 does not exist.

26723    10:08:38.271     [11552]    (EVIndexAdminService)    <3004>    EV-H    {IndexingWCFProxy`1} Exception: The exception [search-collection-invalid-name] was thrown. Info: Diag: Type:System.ServiceModel.FaultException`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] ST:   at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.ReThrowAsTypedFaultExceptionIfAppropriate(FaultException faultEx)|   at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.Invoke[TResult](Expression`1 method, String server)|   at Symantec.EnterpriseVault.Indexing.Common.IndexingWCFProxy`1.Invoke[TResult](Expression`1 method) Inner:None
 

26724    10:08:38.277     [11552]    (EVIndexAdminService)    <3004>    EV~E    |Event ID: 40966 A program fault has raised an exception.|Exception: The exception [search-Index Volume-invalid-name] was thrown.|Diagnostic: |Type: System.ServiceModel.FaultException`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]|Reference: |Command Line: "C:\Program Files (x86)\Enterprise Vault\\EVIndexAdminService.exe" -EntryID:123ABC456DEF789GHI123ABC456DEF789GHI0000EVSiteAlias|Application Domain: EVIndexAdminService.exe|Process Id: 11552|Thread Id: 3004|Stack Trace:    at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.ReThrowAsTypedFaultExceptionIfAppropriate(FaultException faultEx)|   at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.Invoke[TResult](Expression`1 method, String server)|   at Symantec.EnterpriseVault.Indexing.Common.IndexingWCFProxy`1.Invoke[TResult](Expression`1 method)|   at Symantec.EnterpriseVault.Indexing.IndexingEngine.VelocityProxy.SearchIndex VolumeXml(String Index VolumeName, Boolean bStaleOk)|   at Symantec.EnterpriseVault.Indexing.IndexingEngine.VelocityIndexingEngine.GetIndexVolumeLocation(IIndexVolumeIdentifier indexVolumeIdentifier)|   at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.FrozenColumnExists(VelocityProxy proxy, String Index VolumeName)|   at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.HandleErrorsDueToOldSchemaVersion(ErrorInfo Error)|   at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.HandleErrorBehaviour(ErrorInfo Error)|   at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.EnumErrors()|   at Symantec.EnterpriseVault.Indexing.Admin.IndexingEngineMonitor.ErrorMonitoringTask()|   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)|   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)|   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)|   at System.Threading.ThreadHelper.ThreadStart()

Note the index volume collection string in line 26722: 123ABC456DEF789GHI123ABC456DEF789_123

This occurs because the IndexAdminService is looking for the specific index volume listed in the DTrace in SQL in order to determine where it physically resides. However, there is no entry for that volume in the IndexVolume table. This index volume was upgraded from 64-bit to an ElasticSearch index volume. After the upgrade was completed, the original volume was deleted but all references were not removed from the EV server. As a result, the indexing start up process has a reference to a volume that no longer exists and the exception is logged as it's location cannot be found. 

The remediation is to repopulate the metadata for the EV Index server servicing the index volume in question.

1. Edit the collection string name(s) as indicated in the SQL query below. Execute the edited SQL query against the EnterpriseVaultDirectory database and note the ComputerName associated with the affected volumes in the output.

USE EnterpriseVaultDirectory;
SELECT
iv.ComputerName
, iv.VaultStoreEntryId
, vse.VaultStoreName
, iv.VaultEntryId
, iv.RootIdentity
, iv.ArchiveName
, iv.IndexVolumeIdentity
, iv.IndexVolumeSetId
, iv.IndexRootPathEntryId
, iv.IndexRootPath
, iv.BackupMode
, [FolderName (Collection)] = iv.FolderName
, iv.FirstItemSequenceNumber
, iv.HighestItemSequenceNumber
, iv.OldestArchivedDateUTC, iv.YoungestArchivedDateUTC
, OldestItemDateUTC, iv.YoungestItemDateUTC
, iv.IndexedItems
, iv.[Offline], iv.Failed, iv.Rebuilding
FROM IndexVolumeView AS iv
JOIN VaultStoreEntry AS vse ON iv.VaultStoreEntryId = vse.VaultStoreEntryId
WHERE iv.FolderName IN (
-- Edit collection string names below in a single-quote enclosed comma-separated list
'123ABC456DEF789GHI123ABC456DEF789_123'
, '234BCD567EFG890HIJ123ABC456DEF789_456'
-- Edit collection string names above in a single-quote enclosed comma-separated list
)
ORDER BY vse.VaultStoreName, iv.ArchiveName, iv.IndexVolumeIdentity, iv.IndexVolumeSetId;

2. Execute the following query against the EnterpriseVaultDirectory database and note the MetaDataLocation associated with the Computername.

USE EnterpriseVaultDirectory;
SELECT
SE.SiteName
, CE.VaultSoftwareVersion
, CE.ComputerEntryId
, CE.ComputerName
, CE.ComputerNameAlternate
, CE.ComputerRootPath
, CE.CacheLocation
, ISE.MetaDataLocation
FROM ComputerEntry AS CE
JOIN IndexingServiceEntry AS ISE ON CE.ComputerEntryId = ISE.ComputerEntryId
JOIN SiteEntry AS SE ON CE.ComputerSiteEntryId = SE.SiteEntryId
WHERE CE.ComputerEntryID = ISE.ComputerEntryID
ORDER BY SE.SiteName, CE.ComputerName;

3. Stop the Enterprise Vault Indexing Service on the EV Indexing server (Computername).

4. Browse to the IndexMetadata folder location on the EV Indexing server and rename the IndexMetaData folder to .old (default location: C:\Program Files (x86)\Enterprise Vault\EVIndexing\data\).

5. Start the Enterprise Vault Indexing Service on the EV Indexing server and monitor the Enterprise Vault Event Logs for the indexing startup events - See Related Articles below. Note that it may take some time for the Enterprise Vault Indexing Service to be fully functional - please monitor the Event Logs for Event ID 41376 to monitor the metadata repopulate progress.