A restart of the Enterprise Vault Storage Service may cause the StorageOnlineOpns.exe process to return STORAGE_E_SERVICE_NOT_RUNNING errors.
Article ID: 100011251
Updated On:
Description
Error Message
Events logged in Symantec EV event log during a restart of the EV Storage Service:
Type: Error
Event: 6656
Source: Enterprise Vault
Category: Storage Server
Description:
Error starting process
Reason: The Storage Service is not running (0xc0041ae9)
Process: Storage Online Process (1)
Reference: RunTheProcess/CCI
Type: Warning
Event: 6654
Source: Enterprise Vault
Category: Storage Server
Description:
Storage Online Process (1) - Restarting failed process
PID: 0
Exit Code: Unspecified error (0x80004005)
The process has failed abnormally and will be restarted automatically
Type: Error
Event: 6655
Source: Enterprise Vault
Category: Storage Server
Description:
Storage Online Process (1) - Process failed
PID: 0
Exit Code: Unspecified error (0x80004005)
The process has failed abnormally and will not be restarted. Either the process has exceeded
its restart count or automatic restart is disabled. The process can be restarted by stopping and
restarting the Storage Service
Events logged in the Application log on the EV server when the StorageOnlineOpns process crashes:
Type: Error
Event: 1000
Source: Application Error
Category: Application Crashing Events
Description:
Faulting application name: StorageOnlineOpns.exe, version: 10.0.2.1112, time stamp:
Faulting module name: RPCRT4.dll, version: 6.1.7601.18205, time stamp:
Exception code: 0xc00000fd
Fault offset: 0x00033f05
Faulting module name may vary between
RPCRT4.dll, ole32.dll, ntdll.dll, KERNELBASE.dll
Errors that may occur when using the Windows EV Outlook Add-In when retreieving items from the EV Server:
GetOnlineAttachmentfilesize 0xC0041AE9
Cause
The StorageOnlineOpns (SOPNs) process is used by the Storage Service and other processes to fetch information about/from Vault Stores. Remote Procedure Calls (RPC) from other EV processes can start an instance of the SOPNs process when the Storage Service is in a stopped or starting state. When called, the SOPNs process checks the status of the Storage Service. If the Storage Service is not in an online state the SOPNs process receives a return value of 0xC0041AE9 or STORAGE_E_SERVICE_NOT_RUNNING. Calls to the SOPNs process can originate from other EV servers.
The issue reveals itself when the following two conditions occur.
1. The SOPNs process has been invoked from another EV process and has received the Storage Service status of STORAGE_E_SERVICE_NOT_RUNNING, but has not yet exited.
2. At he same time the Storage Service is in the "Starting" state creating it's own instances of the SOPNs process.
When the two operations are happening at the same time, all instances of the SOPNs process receive the status value of STORAGE_E_SERVICE_NOT_RUNNING. This value is then returned for all subsequent instances of the SOPNs process until the EV server is rebooted.
Resolution
The Issue has been addressed in EV Versions 10.0.3/10.0.4 and above
Work around to Mitigate the frequency of the issue:
1. In large EV environments where Archives are located on different EV servers than the Archive Tasks:
a. Do not manually restart the Storage Service without scheduling a full EV server reboot.
2. In large EV environments where Archives are located on the same EV server as the Archive Tasks:
a. Stop all EV services on the affected EV server
b. Stop IIS as follows:
- Open a Command Prompt and type: iisreset /stop
Note: This will stop both IIS Admin Service and World Wide Web Publishing Service
c. Start the services in the following order
Enterprise Vault Admin Service
Enterprise Vault Directory Service
Enterprise Vault Storage Service
Start IIS as follows:
- Open a Command Prompt and type: iisreset /start
Note: This will start both IIS Admin Service and World Wide Web Publishing Service
Enterprise Vault Shopping Service
Enterprise Vault Indexing Service
Enterprise Vault Task Controller
Applies To
Windows Server 2008 R2
EV 10.0.1/10.0.2
Issue/Introduction
On Enterprise Vault (EV) servers, it is possible for a manual restart of the EV Storage Service to cause the StoreageOnlineOpns.exe process to return ‘STORAGE_E_SERVICE_NOT_RUNNING’ errors. Event IDs 6654, 6655, and 6656 will be in the Veritas EV event logs, indicating problems with the EV Storage Service. Retrieval of archived items using the Windows EV Outlook Add-In may return the error "GetOnlineAttachmentfilesize 0xC0041AE9", which will continue to occur until the EV server is rebooted. The Application Event log may include process crash errors, Event ID 1000 referencing the StoreageOnlineOpns process.
A Dtrace of the StorageOnlineOpns process on the affected EV server will contain repeated lines containing the string: "The Storage Service is not running (0xc0041ae9)"
Additional Information
UMI: V-437-6654
UMI: V-437-6655
UMI: V-437-6656
ETrack: 3350043
Was this article helpful?
Yes
No
Powered by
