Error Message

From the CA or DA Client view of the index volumes searched, Info column contents:

The RPC server is unavailable [0x800706ba]

From the Symantec Enterprise Vault Event Log:

Event Type: Error
Event Source: Accelerator Service Processor
Event Category: None
Event ID: 297
Description:
APP AT - Customer ID: 3 - Error processing Vault to Search off the
queue: Searching Vault Thread - worker thread for index server: 1 (0)
Vault ID: 123456
Number of tries: 5
Cause:The RPC server is unavailable. (0x800706ba)

V-437-297

Event Type: Error
Event Source: Accelerator Service Processor
Event Category: None
Event ID: 7181
Description:
Could not contact the Indexing Service |Please check the Indexing Service has been started on EVServer1 |VaultID: 1813451CA02D2D844AAD8ED7101FFDE011110000EVServer1 |Internal reference: |GIAO/cc : The RPC server is unavailable.  (0x800706ba) |

V-437-7181

From dtrace log of the AcceleratorService process on the CA or DA server:

(AcceleratorService) <7876> EV:H {CIndexServerPtr::CreateServer} (Exit) Status: [The RPC server is unavailable.

The Accelerator Service process used for all CA and DA Customer operations is unable to connect to the Enterprise Vault Indexing Service on the EV server associated with the index or specific index volumes of one or more archives to be searched.  Possible causes for this are, but are not limited to:

1. A DNS issue.  Try to ping the EV Indexing Server where the index is located to see if the basic network connectivity is working if all index volumes for an archive are showing the RPC error in the Info column of the Search Details pane when the Show: field is set to the Top 2000 archives by hits or 2000 failed/error archives option.
2. Each index volume reporting the issue has been moved to another EV Indexing Service server and the record for it in the CA or DA Customer database has not updated.
3. Each 32-bit index volume reporting the issue is fragmented and in need of a major compaction operation.

Another possible cause of the error is the use of wildcard characters in the domain section of SMTP address references used in the search criteria.  For example, searching for e-mail messages to or from *.EVLab.com could cause the error to be thrown due to the * wildcard character.

For a possible DNS issue causing the error:

1. Use the nslookup command to confirm the DNS entry for the EV Index Service server is correct.
   1. If the DNS entry is not correct, work with the appropriate networking personnel to correct the DNS entry.
   2. As an alternate, create an entry in the CA or DA server's hosts file for the EV Index Service server, then restart the CA or DA server to ensure the hosts file is read and used instead of DNS for that server.
2. Ping the Enterprise Vault Index Service server to see if the basic network card to network card connectivity is working.
   1. If the basic network card connectivity is not working, work with the appropriate personnel to troubleshoot and resolve.
   2. If the basic network card connectivity is working, check the Enterprise Vault Index Service on that EV server to ensure it is started.
   3. If the service is not started, start it.  Also take into account that the following services should also be running:
      1. Enterprise Vault Admin Service
      2. Enterprise Vault Directory Service
      3. Enterprise Vault Storage Service - this service needs to be running if new items are being added to the archive that must be indexed, or if an index volume is being repaired or rebuilt.

For an index volume that may have been moved to another EV Indexing Service server:

1. Using the Windows Services application on the CA or DA server, stop the Enterprise Vault Accelerator Manager Service (EVAMS).
2. Launch the the SQL Server Management Studio application on the SQL Server hosting the CA or DA customer database.
3. Connect to the SQL Server instance where the CA or DA customer database resides.
4. Open a New Query window focused on the CA or DA customer database.
5. Run the following SQL query to get a count of the Index Volumes in the tblIndexVolumeSet table that contains the reference to each index volume available from the Enterprise Vault site:
   1. SELECT COUNT (*) FROM tblIndexVolumeSet
6. Note the number of rows returned.
7. Run the appropriate SQL command to remove all entries in the tblIndexVolumeSet table from the CA or DA customer database.  Contact Veritas Enterprise Vault Support for assistance with this step.
8. Start EVAMS on the CA or DA server.
9. Launch to the CA or DA client using an account with Application level permission to modify the Archive selections, such as the Vault Service Account (VSA).
10. From the Application tab, click on the Archives sub-tab.
11. Click on the Synchronize button to repopulate the entries in the tblIndexVolumeSet table.
12. Run the following SQL query periodically to determine when the number of rows is close to or the same as to the original result previously noted in Step 6.
    1. SELECT COUNT (*) FROM tblIndexVolumeSet
13. Once the count has stopped increasing, the synchronization is done.
14. Reject the CA or DA search if it has not been manually accepted and run it again.

For a 32-bit index volume that may be fragmented and in need of a major compaction operation:

Note: These actions only work with Enterprise Vault versions 8.x and 9.x as they have only 32-bit index volumes.  For Enterprise Vault 10 or greater versions, an Update operation will convert the 32-bit index volume(s) into 64-bit index volume(s).

1. Identify the owner of the specified index.
   1. Using information from the Event Log Error or dtrace log:
      1. Access the archive search feature in the Vault Admin Console by holding the Ctrl and Shift keys while right clicking on the Archives folder in the tree view pane.
      2. Select the Find Archive or Folder option.
      3. Copy the ArchiveID or VaultID from the event details or dtrace log and paste it into the Entry ID: field in the Find Archive/Archive Folder pane.
      4. Click the Find button.  The result will contain the name of the archive to which the index is associated.
   2. Using information from the CA or DA search:
      1. Launch the CA or DA Client, if not already open, to the CA or DA Customer in which the error is being reported.
      2. If not already showing the search where the RPC error is displayed:
         1. Click on the Cases tab.
         2. In the left pane where all cases are listed, click on the case in which the error occurred.
         3. Click on the Searches sub-tab.
         4. Click on the search in which the error is reported.
      3. In the Search Details pane, select the Show: option of 2000 failed/error archives.
      4. Scroll through the list until the RPC error is displayed in the Info column.
      5. Note the name of the archive in the Archive column.
2. Open Windows Explorer and navigate to the folder containing the index volume reporting the error.
   1. To locate the folder containing the index volume on disk:
      1. Run SQL Query to Locate Index Volume Folder on Disk, shown below the final step for this solution, against the EnterpriseVaultDirectory database using SQL Server Management Studio.
      2. Look through the output of the query to locate the IndexVolumeID column entry of the index volume with the RPC error.
      3. Note the IndexRootPath column entry for the disk drive and folder path to the index volume's folder.
      4. Note the FolderName column entry for the name of the index volume's folder.
3. Look through the files in the folder to see if any of the files with the higher alphabet letters and numbers, such as 'ind_Dd.1' or 'ind_Ff.3', have any size.  Files at the lower 'buckets and tiers' of the index volume that have any size to them indicates the index is fragmented and could be improved by a major compaction operation.  If such files have any size, continue with Step 4.  Otherwise, the index volume may not need the major compaction, but will not be adversely affected if a major compaction is still run.
4. Create an empty file named compact.task in the specified index volume folder.
5. Perform an Update Index Volume operation on the index volume to trigger the index compaction process.  Note: Searches against the index volume will not run while the major compaction operation is running.
   1. Launch the Vault Admin Console (VAC), if not already open.
   2. Navigate to and expand the Archives folder.
   3. Under the Archives folder, expand the folder containing the archive with the index volume having the error.  For example, if an Exchange journal archive has the affected index volume, expand the Exchange Journal folder under the Archives folder.
   4. In the right panel, right click on the archive with the affected index volume.
   5. Select the Properties option.
   6. Select the Index Volumes tab.
   7. From the output of the SQL Query to Locate Index Volume Folder on Disk query run in Step 2.a.i. above, note the FirstItemSequenceNumber column entry of the affected index volume.
   8. If necessary, scroll through the list of index volumes for the archive until the Range column shows a line that begins with the FirstItemSequenceNumber column entry noted above.
   9. Right click on the line for the index volume matching the FirstItemSequenceNumber column entry.
   10. Select the Update Index Volume option.
   11. Click the Yes button on the pop-up alert to confirm the update operation is to be run.
   12. Click the OK button to acknowledge the pop-up alert confirming the update request has been successfully queued.
   13. Click the OK button to close the archive Properties option.
   14. Open the Event Viewer on the Enterprise Vault server running the Enterprise Vault Indexing Service that controls the affected index volume.
   15. Note the information Event ID 7294 indicating the index volume update started.
   16. Refresh the event log until information Event ID 7316 appears to indicate the index volume update has completed.
6. When all index volumes that need the major compaction operation have completed the compaction, run the search or searches again to confirm the compaction has resolved the RPC error.

/* SQL Query to Locate Index Volume Folder on Disk */
/*                                                                                         */
/* Replace "NameOfArchiveWithError" in the               */
/*   WHERE line with the name or partial name of the */
/*   archive with the index volume having the RPC       */
/*   error.                                                                           */

USE EnterpriseVaultDirectory
GO
SELECT [Root].[VaultEntryID]
     , [Archive].[Archive Name]
     , [IndexVolume].[IndexVolumeIdentity]
     , [IndexRootPathEntry].[IndexRootPath]
     , [IndexVolume].[FolderName]
     , [IndexVolume].[FirstItemSequenceNumber]
FROM [Archive]
INNER JOIN [Root]
  ON [Archive].[RootIdentity] = [Root].[RootIdentity]
INNER JOIN [IndexVolume]
  ON [Archive].[RootIdentity] = [IndexVolume].[RootIdentity]
INNER JOIN [IndexRootPathEntry]
  ON [IndexVolume].[IndexRootPathEntryID] = [IndexRootPathEntry].[IndexRootPathEntryID]
WHERE [Archive].[ArchiveName] LIKE '%NameOfArchiveWithError%'     -- Replace NameOfArchiveWithError with the name or partial name of the archive with the affected index volume.

For the use of a wildcard character in an SMTP address criteria:

1. Remove the wild card from the search criteria (i.e., instead of *.EVLab.com use EVLab.com as this will still obtain hits on any child domains such as us.evlab.com or emea.evlab.uk).
2. Test the search criteria to ensure the desired hits are being returned.
3. Adjust the search criteria as appropriate to limit the hits to only those desired.