Using Dtrace.exe to collect debugging information shows that when trying to recreate the Archive Point with FSAUtility.exe, it results in error 3, which translates to "The system cannot find the path specified"
{NTFSStream.Save} Creating NTFS stream: \\?\UNC\svmsrv-cifs.domain.ext\vol1:EVArchivePoint.xml Err: 3
When using the Vault Admin Console to add a root volume, a Dtrace of EVFilesvrArcMngr and mmc will show the following:
2393 13:40:39.229 [12740] (EVFileSvrArcMngr) <4192> EV-M {NTFSStream.Load} Opening NTFS stream: \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml Err: 3
2394 13:40:39.229 [12740] (EVFileSvrArcMngr) <4192> EV-L {FolderAttributes.SaveXml} EVFOLDERPOINT: Trying to save folder point xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\'
2395 13:40:39.229 [12740] (EVFileSvrArcMngr) <4192> EV-L {FolderAttributes.SaveXml} Folder time stamp Before processing for folder :\\svmsrv-cifs.domain.com\volume1\\ is : 3/16/2024 12:23:13 PM
2396 13:40:39.229 [12740] (EVFileSvrArcMngr) <4192> EV-M {NTFSStream.Save} Creating NTFS stream: \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml Err: 3
2397 13:40:39.229 [12740] (EVFileSvrArcMngr) <4192> EV-L {Attributes.CheckRetryCount} FolderAttributes:Retrying to save xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\' : try:(1/3)
2398 13:40:39.338 [12740] (EVFileSvrArcMngr) <4192> EV-L {FolderAttributes.SaveXml} EVFOLDERPOINT: Trying to save folder point xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\'
2399 13:40:39.338 [12740] (EVFileSvrArcMngr) <4192> EV-L {FolderAttributes.SaveXml} Folder time stamp Before processing for folder :\\svmsrv-cifs.domain.com\volume1\\ is : 3/16/2024 12:23:13 PM
2400 13:40:39.338 [12740] (EVFileSvrArcMngr) <4192> EV-M {NTFSStream.Save} Creating NTFS stream: \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml Err: 3
2401 13:40:39.338 [12740] (EVFileSvrArcMngr) <4192> EV-L {Attributes.CheckRetryCount} FolderAttributes:Retrying to save xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\' : try:(2/3)
2402 13:40:39.448 [12740] (EVFileSvrArcMngr) <4192> EV-L {FolderAttributes.SaveXml} EVFOLDERPOINT: Trying to save folder point xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\'
2403 13:40:39.448 [12740] (EVFileSvrArcMngr) <4192> EV-L {FolderAttributes.SaveXml} Folder time stamp Before processing for folder :\\svmsrv-cifs.domain.com\volume1\\ is : 3/16/2024 12:23:13 PM
2404 13:40:39.448 [12740] (EVFileSvrArcMngr) <4192> EV-M {NTFSStream.Save} Creating NTFS stream: \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml Err: 3
2405 13:40:39.448 [12740] (EVFileSvrArcMngr) <4192> EV-L {Attributes.CheckRetryCount} FolderAttributes:Will NOT retry to save xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\'
2406 13:40:39.448 [2592] (mmc) <25448> EV-L {FileServerManagementStub.KVS.EnterpriseVault.FileServerArchive.IFileServerManagement.MarkRootFolder} Exception: The file path \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml: is invalid Info: Diag: Type:System.IO.FileNotFoundException ST:|Server stack trace: | at KVS.EnterpriseVault.Common.NTFSStream.Save(String filePath, Stream data, UInt32 flagsAndAttributes)| at KVS.EnterpriseVault.FileServerArchive.Attributes.GetXmlWriter(String folderPath, String propsFileName, Boolean writeAsAlternateStream, MemoryFileStream& alternateStreamOut, String& filenameOut)| at KVS.EnterpriseVault.FileServerArchive.FolderAttributes.SaveXml()| at KVS.EnterpriseVault.FileServerArchive.Attributes.Save()| at KVS.EnterpriseVault.FileServerArchive.FileServerArchiveManager.KVS.EnterpriseVault.FileServerArchive.IAdminClientManagement.MarkRootFolder(String fileServerEntryId, String volumeId, String folderPath, String rfEID, Boolean remove, String reserved)| at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)| at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)|Exception rethrown at [0]: | at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)| at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)| at KVS.EnterpriseVault.FileServerArchive.IAdminClientManagement.MarkRootFolder(String fileServerEntryId, String volumeId, String folderPath, String rfEID, Boolean remove, String reserved)| at KVS.EnterpriseVault.FileServerArchive.FileServerManagementStub.KVS.EnterpriseVault.FileServerArchive.IFileServerManagement.MarkRootFolder(String fileServerEntryId, String volumeId, String folderPath, String rfEID, Boolean remove) Inner:None
2407 13:40:39.448 [2592] (mmc) <25448> EV~I DirectoryConnection: Function call: CVersionUpdate::NewInstallation |
2408 13:40:39.448 [2592] (mmc) <25448> EV:L {CBaseDirectoryServiceWrapper::CreateDirectoryService} Directory Name [usfsa], Try Local Service [True]
2409 13:40:39.463 [2592] (mmc) <25448> EV:L {VaultCreateTrustedInstanceRequest::CreateLocalInstance} Successfully connected to local service
2410 13:40:39.463 [2592] (mmc) <25448> EV:L {VaultCreateInstanceRequest::CreateInstance} CLSID [{F4D3EB5B-C7C5-11D1-90DB-0000F879BE6A} (EnterpriseVault.DirectoryService.1)] Server Name [usfsa] Used Server Name [usfsa] Num of attempts [1] Total elapsed [0.001s] Result [Success (0)]
2443 13:40:39.730 [2592] (mmc) <25448> EV:H {IAUIBaseImpl
2444 13:40:39.730 [2592] (mmc) <25448> EV:H {IAUIBaseImpl
Further investigation showed that the environment was using the "Symlinks" option for the Shares defined at the Volume level.
Symlinks is an option that can be used to allow UNIX type access to CIFS Shares.
The issue started following an upgrade of ONTAP to a new minor version, where the "symlink-properties" option sets a different default value.
In earlier versions of ONTAP the default value was "enable", and after the upgrade, the default value for the option is "symlinks".
Volumes created prior to the upgrade would show the default value as "enable", whereas new Volumes where the CIFS shares are created with the "Symlinks" option would be set with a default value of "symlinks".
Changing the "symlink-properties" from enable to symlinks allows the hidden streams to be created / displayed, therefore allowing the FSA Task to correctly process the content within the identified Archive Point.
Note: This may require a reboot of the Enterprise Vault server.