Error Message

For example, if the Vault Admin Console (VAC) throws a critical alert for items awaiting backup, likely 28944 events would be thrown by StorageFileWatch (SFW).  A Dtrace of the StorageFileWatch process will show:

(StorageFileWatch)     <99200>     EV:M     OST Streamer: [TID:xxxxx] [Plugin] <= Recv header, 0000000024 bytes (0x00000018)|0000: HTTP/1.1 404 Not Found|

(StorageFileWatch)     <99200>     EV:M     OST Streamer: [TID:xxxxx] [Plugin] generic: Error: no xml data received
(StorageFileWatch)     <99200>     EV:M     OST Streamer: [TID:xxxxx] [Plugin] generic: Error: http 404 Not Found
(StorageFileWatch)     <99200>     EV:M     OST Streamer: [TID:xxxxx] [Plugin] generic: Error getting image properties, HTTP code 404, returning(2060013) ...">
(StorageFileWatch)     <99200>     EV:M     OST Streamer: [TID:xxxxx] [Plugin] generic:xxxxF56128C0A1448CD3DAxxxxxxxxxx: Error: failed to get metadata for image 7c160ecf-d136-xxxx-xxxx-e7c9a5372b9hostname1 (rv = 2060013)
(StorageFileWatch)     <99200>     EV:M     OST Streamer: [TID:xxxxx] OST API sts_get_image_prop_byname() failed with error: 2060013 - 'no more entries'.
(StorageFileWatch)     <99200>     EV~E     Event ID: 28944 The 3rd party storage system application 'OST Streamer' has logged the following message: |CStreamerObject::Info(): method failed|Reason = 0x80070103|Description = OST API sts_get_image_prop_byname() failed with error: 2060013 - 'no more entries'. |

This is caused by a combination of an http error at the time of writing the Cab to the S3 device, and OST Streamer's lack of reporting/acknowledging that error. 

S3 devices dictate that a standard set of metadata files are written with Cab's, such as "Image_Properties" and "Block_Map_File."  A situation may occur in which the Cab file is successfully written, but an http error, such as a 503, occurs upon writing the metadata files with it. 

Enterprise Vault (EV) storage reports no error, as the Cab file itself was successfully written.  But when the Cab is accessed later for saveset retrieval during indexing, SFW, or other EVSVR processes (like dumpsaveset), the OST Streamer does require those metadata files to properly retrieve the file, and thus fails. 

This has been resolved in EV 12.4.2, by improving error reporting of the metadata files that accompany Cab's written to S3 devices by the OST Streamer.  If an error occurs in writing those metadata files, the process is rolled back and continued to try again until successful writing of the accompanying metadata files. 

NOTE: In the event that an upgrade is impossible, there is a work around:  Where the issue exists, increase the age at which to migrate to secondary storage, and increase the time to delete local Cab files after migration.  This will put a strain on local primary storage usage space, but keeps data available.  This is only meant to be temporary until an upgrade can occur. 

If a local version of a Cab exists as an ".archcab," then that ".archcab" can be renamed to ".Cab" and will be attempted to be migrated again, providing it meets age requirements.  It should be noted that while this causes no problems for EV, it will double up space used on the S3 for that Cab, and no clean up is in place for the original that was already written improperly without its metadata files.