StorageOnlineOPNS Fails to Stop/Start during Storage Service shutdown/startup causing The RPC server is too busy to complete this operation. (0x800706bb) errors.

book

Article ID: 100033268

calendar_today

Updated On:

Description

Error Message

The RPC server is too busy to complete this operation. (0x800706bb)

The error could manifest in different ways depending on the application or processing calling StorageOnlineOpns.
The error has been seen whilst users attempting to delete items from EV Search, whilst using ArchiveShuttle or Enterprise Vault Discovery Accelerator to retrieve items from the Storage.

A dtrace of the StorageOnlineOpns process:

10    15:15:18.332     [22672]    (StorageOnlineOpns)    <10584>    EV:L    {CEVAutoThreadModuleT::CreateInstance} (Entry)
11    15:15:18.332     [22672]    (StorageOnlineOpns)    <10584>    EV:L    {CEVAutoThreadModuleT::AllocateApartment} (Entry)
12    15:15:18.424     [22672]    (StorageOnlineOpns)    <33712>    EV:L    {CEVAutoThreadModuleT::AllocateApartment} (Exit)
13    15:15:18.424     [22672]    (StorageOnlineOpns)    <33712>    EV:H    {CEVAutoThreadModuleT::CreateInstance:#608} Unable to allocate an apartment returning server busy error.
14    15:15:18.424     [22672]    (StorageOnlineOpns)    <33712>    EV:H    {CEVAutoThreadModuleT::CreateInstance} (Exit) Status: [The RPC server is too busy to complete this operation.  (0x800706bb)]


In addition, the following DCOM System Event logs can be found:

Event : 10006
Source : DCOM
Category : None
Computer : EVSERVER1.testdomain.local
Description:
'2147944123', 'EVServer1.testdomain.local', '{630F3598-C563-4F73-AAA8-FC7A8F401873}'
2147944123 translates to 800706bb which is "SERVER TOO BUSY"
and 630F3598-C563-4F73-AAA8-FC7A8F401873 will be a StorageOnlineOpns instance classID.

Cause

Some StorageOnlineOpns instances are not correctly managed by the StorageService at the time of shutdown or startup, leaving orphaned processes running.

Resolution

Workaround:
Using TaskManager or ProcessExplorer tools, identify and terminate the orphaned StorageOnlineOpns processes with more than 500 threads.

There are no plans to address this issue by way of a patch or hotfix in earlier versions of the software at the present time.  However, the issue has been addressed in the revision of the product specified at the end of this article.  
Please contact your Veritas Sales representative or the Veritas Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.

The issue has been fixed in EV 12. Refer Download Center for upgrade.

 

Issue/Introduction

During a StorageService stop, several StorageOnlineOpns instances may remain running showing a thread leak if the EV API is being called by either a third party application or EV Discovery Accelerator to access archived items.
In a few minutes, this will cause the process or processes to reach the 500 COM Apartment object allocation limit.
At StorageService restart, these orphaned StorageOnlineOpns instances, will not be recreated and will be re-used.
At this point, no other process will be able to use the StorageOnlineOpns affected instances as the 500 threads limit has been reached. This will result in the " The RPC server is too busy to complete this operation. (0x800706bb) " error.

Additional Information

ETrack: 3842876 ETrack: 3781604 ETrack: 3672350