The remote server returned an error: (429) Too Many Requests

book

Article ID: 100064868

calendar_today

Updated On:

Description

Error Message

 

Event ID: 41309 The following error message occurred in the core indexing engine: Failed to communicate with Elasticsearch indexing engine. Try restarting the Indexing Service to fix the issue.

Elasticsearch JVM heap Usage Thread:32984> EV-H{ESNestClient.GetJVMHeapUsage}

Exception: The remote server returned an error: (429) Too Many Requests. Call: Status code 429 from: GET /_cat/nodes?master_timeout=30s. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [20119824728/18.7gb], which is larger than the limit of [19732368588/18.3gb], real usage: [20119824728/18.7gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=9542280/9.1mb, model_inference=0/0b, eql_sequence=0/0b, accounting=9795536/9.3mb]" Info:Failed to get Elasticsearch JVM heap Usage Diag: Type:Elasticsearch.Net.ElasticsearchClientException ST: at Elasticsearch.Net.Transport`1.HandleElasticsearchClientException(RequestData data, Exception clientException, IElasticsearchResponse response)| at Elasticsearch.Net.Transport`1.FinalizeResponse[TResponse](RequestData requestData, IRequestPipeline pipeline, List`1 seenExceptions, TResponse response)| at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)| at Nest.NamespacedClientProxy.DoCat[TRequest,TParams,TCatRecord](TRequest request)| at Veritas.EnterpriseVault.Indexing.EVElasticsearchClient.ESNestClient.GetJVMHeapUsage(Int16 esJVMHeapUsageTimeoutInSeconds) Inner:System.Net.WebException: The remote server returned an error: (429) Too Many Requests.| at System.Net.HttpWebRequest.GetResponse()| at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)

V-437-41309

 

Cause

The issue is caused by the ElasticSearch JVM memory usage exceeding its limit. This is due to the way the search criteria is translated and executed by the ElasticSearch indexing engine. 

 

Resolution

In some instances, increases the JVM heap size may resolve the issue. 

1. Edit the EV Server's Properties in the Vault Admin Console (VAC).

2. Select the Indexing dropdown in the Advanced tab.

3. Edit the Elasticsearch JVM heap size (MB).

image.png

4. Save the change

Note: The maximum size allowed is 50% of RAM, capped at 30720 MB.

For Compliance Accelerator and Discovery Accelerator Searches, the issue can be worked around by doing the following: 

  • Compliance Accelerator Searches
    • Reduce the number of hotwords/phrases/keywords being searched.
    • Create additional searches based on date ranges and spread hotwords being searched.
  • Discovery Accelerator Searches:
    • Reduce the number of keywords being searched.
    • Reduce the number of Custodians being searched.
    • Create additional searches based on date ranges and/or divide Custodians being searched.

 

The log excerpts, information and/or screenshot(s) above were taken from a non-production test lab environment and are used for example purposes only.

 

Enterprise Vault has acknowledged that the above-mentioned issue is present in the version(s) of the product(s) referenced in this article.

This issue is currently under investigation by Enterprise Vault. Pending the outcome of the investigation, this issue may be resolved by way of a cumulative hotfix or service pack in the current or future versions of the software. However, this particular issue is not currently scheduled for any release.  If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Sales representative or the Sales group to discuss these concerns.  For information on how to contact Sales, please see https://supportenterprisevault.cloud.com .

Please be sure to refer back to this document periodically as any changes to the status of the issue will be reflected here.

 

 

Issue/Introduction

When performing searches with ElasticSearch indexes, Searches, searches may fail and the Enterprise Vault Server indexing engine may require a restart to initiate additional searches.

Additional Information

JIRA: CFT-5907

Powered by Wolken Software