Event ID 199: Searches do not run and the Accelerator Manager Service stops intermittently

book

Article ID: 100018631

calendar_today

Updated On:

Description

Error Message

Pop-up Error in the Accelerator Client:

Only one usage of each socket address (protocol/network address/port) is normally permitted.

Enterprise Vault Event Log entries:

Event Type:  Error
Event Source:  Accelerator Manager
Event Category:  None
Event ID:  199
Description:
APP ATM - Error during remoting.  System.Runtime.Remoting.RemotingException: Remoting configuration failed with the exception 'System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocal/network address/port) is normally permitted.

Event Type:  Error
Event Source:  .NET Runtime 2.0 Error Reporting
Event Category:  None
Event ID:  5000
Description:
EventType clr20r3, P1 acceleratorservice.exe, P2 7.5.2.0, P3 475edacb, P4 mscorlib, P5 2.0.0.0, P6 471ebc5b, P7 416b, P8 27, P9 system.runtime.remoting.remoting, P10 NIL.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type:  Error
Event Source:  Accelerator Service Provider
Event Category:  None
Event ID:  141
Description:
TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

Event Type:  Error
Event Source:  Accelerator Service Provider
Event Category:  None
Event ID:  177
Description:
TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

Event Type:  Error
Event Source:  Accelerator Service Provider
Event Category:  None
Event ID:  223
Description:
TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

Event Type:  Error
Event Source:  Accelerator Service Provider
Event Category:  None
Event ID:  288
Description:
TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

Event Type:  Error
Event Source:  Accelerator Service Provider
Event Category:  None
Event ID:  291
Description:
TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

Event Type:  Error
Event Source:  Accelerator Service Provider
Event Category:  None
Event ID:  293
Description:
TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

Event Type:  Error
Event Source:  Accelerator Service Provider
Event Category:  None
Event ID:  312
Description:
TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

Event Type:  Error
Event Source:  Accelerator Service Provider
Event Category:  None
Event ID:  323
Description:
TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

Source:        Accelerator Service Processor
Event ID:      372
Task Category: None
Level:         Error
Description:
APP AS - Customer ID: -1 - Error When Starting the Accelerator Service. System.Net.Sockets.SocketException (0x80004005): Only one usage of each socket address (protocol/network address/port) is normally permitted
   at KVS.Accelerator.Server.CommonStarter.InitRemoting()
   at KVS.Accelerator.Server.CommonStarter.Start()
   at KVS.Accelerator.Server.AcceleratorServiceProcessor.MainThread()

Source:        Accelerator Service Processor
Event ID:      364
Task Category: None
Level:         Error
Description:
APP AS - Customer ID: -1 - Remoting: Failed to initiliase remoting. System.Net.Sockets.SocketException (0x80004005): Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at System.Net.Sockets.TcpListener.Start(Int32 backlog)
   at System.Runtime.Remoting.Channels.ExclusiveTcpListener.Start(Boolean exclusiveAddressUse)
   at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel.StartListening(Object data)
   at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel.SetupChannel()
   at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel..ctor(IDictionary properties, IServerChannelSinkProvider sinkProvider, IAuthorizeRemotingConnection authorizeCallback)
   at KVS.Accelerator.Server.CommonStarter.ListenForWebClientOnIPv4()
   at KVS.Accelerator.Server.CommonStarter.InitRemoting()

Cause

When a client initiates a TCP/IP socket connection to a server, the client typically connects to a specific port on the server and requests that the server respond to the client over an ephemeral, or short lived, TCP or UDP port. On Windows Server 2003 and Windows XP the default range of ephemeral ports used by client applications is from 1025 through 5000. Under certain conditions it is possible that the available ports in the default range will be exhausted.

Compliance & Discovery Accelerator are an example of such clients.  For every customer database, an instance of AcceleratorService.exe process will be created (this can be seen in Task Manager under processes).  For every instance of AcceleratorService.exe, a TCP/IP socket connection is made with the SQL Server.  If a large number of AcceleratorService.exe processes exist, subsequent searches across multiple customers can cause one or all of the following:

  • Searches will not run.
  • Accelerator Service will stop.
  • Events 199 & 5000 will occur with the information shown in the Errors section above.

TCP/IP port exhaustion can occur on a client computer if the client computer is engaging in an unusually high number of TCIP/IP socket connections. This can occur if many client applications are initiating connections.  With Compliance & Discovery Accelerator, this would occur when multiple searches are being performed against multiple customers.

If all of the available ephemeral ports are allocated to client applications then the client experiences a condition known as TCP/IP port exhaustion. When TCP/IP port exhaustion occurs, client port reservations cannot be made and errors will occur in client applications that attempt to connect to a server via TCP/IP sockets.

TCP/IP port exhaustion is more likely to occur under high load conditions than under normal load conditions.

The below event id's may also be seen on a Compliance or Discovery Accelerator server when port exhaustion occurs.  NOTE: Each of these events shows the same error:  TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.

EventID's:  141, 177, 223, 288, 291, 293, 312, 323

 

Resolution

Adjust the web server TCP/IP socket parameters to provide the ASP.NET environment with adequate network sockets at a sufficient reusable rate. To do this:

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry.  Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

Windows 2003 Server:
1.  On each of the Accelerator, Enterprise Vault and SQL server(s), locate the following key in the Windows registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2.  Update the following values, or create them if they do not already exist:

 
Name Type Default Recommended (decimal)
MaxUserPort DWORD 5,000 64,512
TCPTimedWaitDelay DWORD 240 120
MaxFreeTcbs DWORD 2,000 65,536
MaxHashTableSize DWORD 512 16,384



3.  For changes to take effect, restart the server. 

 

Windows Server 2008 and later machines:
  On Windows 2008 and later servers, by default the operating system allows socket connections to be established between the ports 49152 – 65535; this equates to a little over 13,000 user ports. 

To display the current ports for the TCP protocol use the netsh command

netsh int ipv4 show dynamicport tcp

- To increase the ports use the net shell command

netsh int ipv4 set dynamicport tcp start=1500 num=63000
 
This will provide 63,000 TCP user ports.

 

Note:  
• When increasing the MaxUserPort, Microsoft recommends that port 1434 be reserved for use by the SQL Server Browser service (sqlbrowser.exe).

• On a Windows 2003 server add the following registry key
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Add a Multi-string Value key.
Rename the new key to ReservedPorts
Add the values:  1434-1434
• On a Windows 2008 server start the port allocation at 1500 as shown in the previous Windows 2008 netsh command.
 

 

Issue/Introduction

Enterprise Vault Compliance & Discovery Accelerator searches will not run and the Accelerator Manager Service stops intermittently. Event log shows event error id 199.
Powered by Wolken Software