Error Message

The cluster log will show information similar to the following lines when the resource fails to come online:-

0000068c.00000be8::2015/09/17-00:42:01.901 INFO  [RCM] Issuing Online(EVVIR-EnterpriseVaultServerInstance) to RHS.
0000068c.00000be8::2015/09/17-00:42:01.902 WARN  [RCM] HandleMonitorReply: ONLINERESOURCE for 'EVVIR-EnterpriseVaultServerInstance', gen(0) result 193/0.
0000068c.00000be8::2015/09/17-00:42:01.902 INFO  [RCM] Res EVVIR-EnterpriseVaultServerInstance: OnlineCallIssued -> ProcessingFailure( StateUnknown )
0000068c.00000be8::2015/09/17-00:42:01.902 INFO  [RCM] TransitionToState(EVVIR-EnterpriseVaultServerInstance) OnlineCallIssued-->ProcessingFailure.
0000068c.00000be8::2015/09/17-00:42:01.902 ERR   [RCM] rcm::RcmResource::HandleFailure: (EVVIR-EnterpriseVaultServerInstance)
0000068c.00000be8::2015/09/17-00:42:01.903 INFO  [RCM] resource EVVIR-EnterpriseVaultServerInstance: failure count: 0, restartAction: 0 persistentState: 1.
0000068c.00000be8::2015/09/17-00:42:01.903 INFO  [RCM] Will queue immediate restart (15000 milliseconds) of EVVIR-EnterpriseVaultServerInstance after terminate is complete.
0000068c.00000be8::2015/09/17-00:42:01.903 INFO  [RCM] Res EVVIR-EnterpriseVaultServerInstance: ProcessingFailure -> WaitingToTerminate( DelayRestartingResource )
0000068c.00000be8::2015/09/17-00:42:01.903 INFO  [RCM] TransitionToState(EVVIR-EnterpriseVaultServerInstance) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
0000068c.00000be8::2015/09/17-00:42:01.903 INFO  [RCM] Res EVVIR-EnterpriseVaultServerInstance: [WaitingToTerminate to DelayRestartingResource] -> Terminating( DelayRestartingResource )
0000068c.00000be8::2015/09/17-00:42:01.903 INFO  [RCM] TransitionToState(EVVIR-EnterpriseVaultServerInstance) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
0000084c.00000a70::2015/09/17-00:42:01.903 INFO  [RHS] Waiting for Open call for EVVIR-EnterpriseVaultServerInstance to complete.
0000084c.00000a70::2015/09/17-00:42:01.903 WARN  [RHS] Attempting to call Open again for resource EVVIR-EnterpriseVaultServerInstance
0000068c.00000654::2015/09/17-00:42:01.905 INFO  [RCM-rbtr] giving default token to group EVVIR
0000068c.00000b7c::2015/09/17-00:42:01.905 WARN  [RCM] HandleMonitorReply: OPENRESOURCE for 'EVVIR-EnterpriseVaultServerInstance', gen(1) result 193/0.

The successful restart showing the separate monitor flag being automatically set:-

0000068c.00000b7c::2015/09/17-00:44:29.918 INFO  [RCM] Separate monitor flag has changed for resource 'EVVIR-EnterpriseVaultServerInstance'.  Now hosted by RHS process c6c
0000068c.00000b7c::2015/09/17-00:44:29.951 INFO  [NM] Received request from client address WS2012R2CLU1.
0000068c.00000b7c::2015/09/17-00:44:29.967 INFO  [RCM] HandleMonitorReply: OPENRESOURCE for 'EVVIR-EnterpriseVaultServerInstance', gen(4) result 0/0.
0000068c.00000654::2015/09/17-00:44:46.955 INFO  [RCM] Delay-restarting EVVIR-EnterpriseVaultServerInstance and any waiting dependents.
0000068c.00000654::2015/09/17-00:44:46.957 INFO  [RCM] Res EVVIR-EnterpriseVaultServerInstance: Failed -> OnlineCallIssued( StateUnknown )
0000068c.00000654::2015/09/17-00:44:46.957 INFO  [RCM] TransitionToState(EVVIR-EnterpriseVaultServerInstance) Failed-->OnlineCallIssued.
0000068c.00000654::2015/09/17-00:44:46.957 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (EVVIR, Failed --> Pending)
0000068c.00000d4c::2015/09/17-00:44:46.957 INFO  rcm::RcmResource::OnlineWorker[CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint Software\KVS\Enterprise Vault for resource EVVIR-EnterpriseVaultServerInstance.
0000068c.00000d4c::2015/09/17-00:44:46.985 INFO  [RCM] Issuing Online(EVVIR-EnterpriseVaultServerInstance) to RHS.
00000c6c.00000834::2015/09/17-00:44:47.009 INFO  [RES] Enterprise Vault Server Instance : Online request.
0000068c.00000d4c::2015/09/17-00:44:47.015 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'EVVIR-EnterpriseVaultServerInstance', gen(4) result 997/0.
0000068c.00000d4c::2015/09/17-00:44:47.015 INFO  [RCM] Res EVVIR-EnterpriseVaultServerInstance: OnlineCallIssued -> OnlinePending( StateUnknown )
0000068c.00000d4c::2015/09/17-00:44:47.015 INFO  [RCM] TransitionToState(EVVIR-EnterpriseVaultServerInstance) OnlineCallIssued-->OnlinePending.
00000c6c.000007e4::2015/09/17-00:44:47.015 INFO  [RHS] Resource EVVIR-EnterpriseVaultServerInstance has come online. RHS is about to report status change to RCM
0000068c.00000654::2015/09/17-00:44:47.016 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'EVVIR-EnterpriseVaultServerInstance', gen(4) result 0/0.
0000068c.00000654::2015/09/17-00:44:47.016 INFO  [RCM] Res EVVIR-EnterpriseVaultServerInstance: OnlinePending -> Online( StateUnknown )
0000068c.00000654::2015/09/17-00:44:47.016 INFO  [RCM] TransitionToState(EVVIR-EnterpriseVaultServerInstance) OnlinePending-->Online.

Cause

This issue is due to the 32-bit resource DLL that is currently in use for the EnterpriseVaultServerInstance resource.

Workaround

One possible workaround for this is to select the "Run this resource in a separate Resource Monitor" option for the resource 'EVVIR-EnterpriseVaultServerInstance'.

To do this follow the below steps:-

1. In Failover Cluster Manager, in the console tree, click the Role that contains the resource that you want to configure.
2. In the details pane, right-click the resource that you want to configure, and then click Properties.
3. On the Advanced Policies tab in the resource's Properties dialog box, select Run this resource in a separate Resource Monitor, click OK.

Note: This workaround does not work in every environment. Veritas could not identify the difference between environments where this workaround worked and did not work. If this workaround does not work, utilize the solution of upgrading to EV 12.1 and applying the steps in the related article or upgrading to EV 12.2 or above.

Solution

This issue has been addressed in the following release:
 
Enterprise Vault 12.1
 
The following link contains information about Enterprise Vault 12.1
https://downloads.enterprisevault.com/enterprise-vault/REL897076

Refer to the related article for the steps post installing or upgrading Enterprise Vault.
 
Note: For information on how to obtain Enterprise Vault 12 see: https://downloads.enterprisevault.com/enterprise-vault/REL897076
 

Workaround

One possible workaround for this is to select the "Run this resource in a separate Resource Monitor" option for the resource 'EVVIR-EnterpriseVaultServerInstance'.

To do this follow the below steps:-

1. In Failover Cluster Manager, in the console tree, click the Role that contains the resource that you want to configure.
2. In the details pane, right-click the resource that you want to configure, and then click Properties.
3. On the Advanced Policies tab in the resource's Properties dialog box, select Run this resource in a separate Resource Monitor, click OK.

Note: This workaround does not work in every environment. Veritas could not identify the difference between environments where this workaround worked and did not work. If this workaround does not work, utilize the solution of upgrading to EV 12.1 and applying the steps in the related article or upgrading to EV 12.2 or above.

This issue is due to the 32-bit resource DLL that is currently in use for the EnterpriseVaultServerInstance resource.

This issue has been addressed in the following release:
 
Enterprise Vault 12.1
 
The following link contains information about Enterprise Vault 12.1
https://downloads.enterprisevault.com/enterprise-vault/REL897076

Refer to the related article for the steps post installing or upgrading Enterprise Vault.
 
Note: For information on how to obtain Enterprise Vault 12 see: https://downloads.enterprisevault.com/enterprise-vault/REL897076
 

The issue manifests itself as follows:-

1. The current Windows 2012 R2 active cluster node running EV fails unexpectedly.
2. The cluster resource 'EnterpriseVaultServerInstance' is automatically failed over to the passive node, which is now becoming active.
3. The EnterpriseVaultServerInstance goes into a status of failed.
4. The EnterpriseVaultServerInstance is eventually brought online once all of the retries have been exhausted as set on the Properties - Policies tab of the resource. This is due to the fact that once all retries have been exhausted an automatic flag is set to run the resource in a separate resource monitor.

When performing a manual failover this issue does not occur.

ETrack: 3851784