Error Message

Enterprise Vault Event log contains

Event : 6654
Source : Enterprise Vault
Category : Storage Server
Description:
Storage Restore Server (1)  -  Restarting failed process
PID: 7324
Exit Code: 0
The process has failed abnormally and will be restarted automatically

Event : 6654
Source : Enterprise Vault
Category : Storage Server
Description:
Storage Deletion Server (1)  -  Restarting failed process
PID: 7384
Exit Code: 0
The process has failed abnormally and will be restarted automatically

Event : 6654
Source : Enterprise Vault
Category : Storage Server
Description:
Storage File Watch Server (1)  -  Restarting failed process
PID: 7760
Exit Code: 0
The process has failed abnormally and will be restarted automatically

Followed by:

Event : 6952
Source : Enterprise Vault
Category : Storage File Watch
Description:
A EMC Centera Collector thread encountered an error and has stopped.
Reason: Catastrophic failure  (0x8000ffff)
PartitionEntryID: 19C70BD4EA7FB724B9FBFC1514B3E23D61q10000evserver
Method: CCollectorEMCCentera::CollectStore

This issue occurs when the StorageFileWatch is traversing the original item to determine if it contains any components that are valid for sharing on Centera; in rare circumstances the StorageFileWatch thread might hang during the function {NoteRenderer::RenderNoteAsTextMIME} breaking the notes mutex connection between Enterprise Vault and Domino and not allowing the acquisition of new mutexes.

The dtrace for the StorageFileWatch contains the following call in hang where the {NoteRenderer::RenderNoteAsTextMIME} begins but never ends:

[7760]    (StorageFileWatch)    <10948>    EV:L    {NoteRenderer::RenderNoteAsTextMIME} (Entry) --> begin
[7760]    (StorageFileWatch)    <10948>    EV:M    NoteRenderer::RenderNoteAsTextMIME - allocating [331] bytes
[7760]    (StorageFileWatch)    <10948>    EV:M    NoteRenderer::RenderNoteAsTextMIME - too small - allocating [662] bytes
[7760]    (StorageFileWatch)    <10948>    EV:M    NoteRenderer::RenderNoteAsTextMIME - too small - allocating [1324] bytes

This is an example of a StorageFileWatch dtrace where the {NoteRenderer::RenderNoteAsTextMIME}  begins and ends:

[7760]    (StorageFileWatch)    <10980>    EV:L    {NoteRenderer::RenderNoteAsTextMIME} (Entry)  --> begin
[7760]    (StorageFileWatch)    <10980>    EV:M    NoteRenderer::RenderNoteAsTextMIME - allocating [726] bytes
[7760]    (StorageFileWatch)    <10980>    EV:L    {NoteRenderer::RenderNoteAsTextMIME} (Exit) Status: [Success]  --> end

A further analysis of the aforementioned thread in hang would reveal the latest .DVS file being processed such as:

[7760]    (StorageFileWatch)    <10948>    EV:M    CSavesetPersist::Load File: D:\CENTERA_COLLECTION\2016\11\05\12\201611058308857~201611051206240000~Z~01550196ACAF855F067F61E4737B6C11.DVS Mode:0

The RenderNotesAsTextMIME function is hanging on the IBM C API MIMEGetText, in order to fix this issue it is necessary to install the latest Lotus Notes Feature Pack available and restart the Enterprise Vault Server.

 

Users are no longer able to retrieve any archived items from Lotus Notes or iNotes after the Event IDs 6654 containing 'Restarting failed process'

ETrack: 3905364