How to enable Search Query Logging for EV Indexing

book

Article ID: 100039313

calendar_today

Updated On:

Description

Enabling Search Query Logging:

To enable Search Query Logging, follow these steps within the Enterprise Vault Admin Console (VAC):

Expand Enterprise Vault Servers under the site container in the left pane.
Right-click the Enterprise Vault server that hosts the Indexing service and select Properties.
On the Advanced tab, select Search Log Queries and click Modify.
In the drop-down menu, select On, and click OK.
Select Search Logs Folder and click Modify.
Enter a path to a local folder on the Index server, and click OK.
On the Properties page click OK, and then on the warning dialog about changes to the advanced settings, click OK.
Restart the Enterprise Vault server’s Indexing Service.
Perform searches using any Search application.

To disable Search Query Logging, follow the same steps but select OFF in step 4.

Locating the Search Query Logs:

After enabling logging and running a search, navigate to the local path as set in Step 5 to review the log files generated. Depending on the number and type of index volumes that are targeted by the search, different files are generated for each search query.

A file beginning with squery contains the search request in StructuredQuery format, suitable for 32-bit index volumes (EV 9 and earlier). This will be an XML file.
A file beginning with velreq contains the search request in Velocity format, suitable for 64-bit index volumes (EV 10 through EV 14.1). This will be an XML file.
A file beginning with elasticsearchreq contains the search request in Elasticsearch format, suitable for Elasticsearch index volumes (EV 14.2 and later). This will be a JSON file.

Note that for search requests that target multiple index volumes, the StructuredQuery and Velocity query logs will create a separate XML file for each index volume, while the Elasticsearch query logs will create just one JSON file.

Note also that a single search may target an archive with several index volumes of different types, and therefore it is not alarming for many different index query logs to be generated from a single search.

How to interpret a Search Query Log in thesqueryform:

Once the Search Query Logging is enabled, squery_*.xml files are generated in the configured location when a search is performed. This file contains the basic structure of a search query in a format internal to Enterprise Vault. It has the search terms, the search fields and conditions specified in the search criteria.

Sample Search Criteria 1:

FIELD	OPERATOR	SEARCH TERMS
Subject	contains all of	ratify eclipse suspend
From	contains any of	bill
Content	contains phrase	snowy adventure

Search Query Log generated in the squery_*.xml:

true true false and and subj "ratify" any subj "eclipse" any subj "suspend" any auth "bill" any cont "snowy adventure" any

Sample Search Criteria 2:

FIELD	OPERATOR	SEARCH TERMS
Date	is between	From 01/05/2005 To 01/01/2012
Subject or Content	contains any of	good old days

Search Query Log generated in the squery_*.xml:





    true

    true

    false

    

        and

        

            

                date

                2005-01-05T05:00:00Z

                2012-01-02T04:59:59Z

            

            

                or

                

                    

                        text

                        "good"

                        any

                    

                    

                        text

                        "old"

                        any

                    

                    

                        text

                        "days"

                        any

The following examples, contained within the log file, can be used to understand the search query better:

. The operand i:type indicates the type of search terms like StringTerm for String, NumericRangeTerm for numeric or DateRangeTerm for Dates.
The node indicates the field specified in search criteria.
The node indicates the search terms specified.
The node indicates the logical operator applied on the search terms as specified.
For search by Ranges, the node indicates the ‘From’ value of the range.
For search by Ranges, the node indicates the ‘To’ value of the range.

How to interpret a Search Query Log in thevelreqform:

Once the Search query logging is enabled the velreq_*.xml files are generated in the configured location when a search is performed. This file contains the complete structure of a search query that is passed to the 64-bit indexing engine. The following are some of the major information that can be found in the velreq file,

search terms specified
search fields used
search conditions (like any of or phrase, etc)
sorting order
search timeout duration
collection ID (the id of an index volume)

Sample Search Criteria 1:

FIELD	OPERATOR	SEARCH TERMS
Subject	contains all of	ratify eclipse suspend
From	contains any of	bill
Content	contains phrase	snowy adventure

Search Query Log generated in the velreq_*.xml:

17845AF6064C96B4C80B01C41EC244919_136 0 100 100 list msgc natc auth subj date size tags ppdn dtyp pvid snum vlid _Vault.MsgType ItemExpiry keys _evtag.category _evtag.inclusion _evtag.exclusion false true limited off 600000

Sample Search Criteria 2:

FIELD	OPERATOR	SEARCH TERMS
Date	is between	From 01/05/2005 To 01/01/2012
Subject or Content	contains any of	good old days

Search Query Log generated in the velreq_*.xml:

17845AF6064C96B4C80B01C41EC244919_76 0 100 100 list msgc natc auth subj date size tags ppdn dtyp pvid snum vlid _Vault.MsgType ItemExpiry keys _evtag.category _evtag.inclusion _evtag.exclusion false true limited off 600000

The main lines to focus in the log and understand the search query are:

The value in the node indicates the index volume Id or the collection id on which the search is performed on.
The node indicates the type of condition like AND, OR, PHRASE, RANGE, etc.
Field in node indicates the field specified in search criteria.
String in node indicates the search terms specified.
The prefix x_ is meant to search on both top-level and attachment-level of the item. If the search field is not prefixed by x_ then it is meant to search only on the top-level of the item.

How to interpret a Search Query Log in theelasticsearchreqform:

Once the Search query logging is enabled the elasticsearchreq_*.xml files are generated in the configured location when a search is performed. This file contains the complete structure of a search query that is passed to the Elasticsearch indexing engine. The following are some of the major information that can be found in the velreq file,

search terms specified
search fields used
search conditions (like any of or phrase, etc)
sorting order
search timeout duration

Sample Search Criteria 1:

FIELD	OPERATOR	SEARCH TERMS
Subject	contains all of	ratify eclipse suspend
From	contains any of	bill
Content	contains phrase	snowy adventure

Search Query Log generated in the elasticsearchreq_*.json:

{ "_source": false, "from": 0, "query": { "bool": { "filter": [ { "term": { "indexVolumeName": { "value": "17845af6064c96b4c80b01c41ec244919_153" } } } ], "must": [ { "bool": { "must": [ { "bool": { "must": [ { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "fields.text.subj": { "query": "ratify" } } }, { "match_phrase": { "fields.text.a_subj": { "query": "ratify" } } }, { "match_phrase": { "fields.textsearchable.a_subj": { "query": "ratify" } } } ] } } ] } }, { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "fields.text.subj": { "query": "eclipse" } } }, { "match_phrase": { "fields.text.a_subj": { "query": "eclipse" } } }, { "match_phrase": { "fields.textsearchable.a_subj": { "query": "eclipse" } } } ] } } ] } }, { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "fields.text.subj": { "query": "suspend" } } }, { "match_phrase": { "fields.text.a_subj": { "query": "suspend" } } }, { "match_phrase": { "fields.textsearchable.a_subj": { "query": "suspend" } } } ] } } ] } } ] } }, { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "fields.textsearchable.wrdn": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.wrsm": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.wrot": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.frdn": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.frsm": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.frot": { "query": "bill" } } }, { "match_phrase": { "fields.text.ppdn": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.ppsm": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.ppot": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.jrfm": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.jrpp": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.jaen": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_wrdn": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_wrdn": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_wrsm": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_wrsm": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_wrot": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_wrot": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_frdn": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_frdn": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_frsm": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_frsm": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_frot": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_frot": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_ppdn": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_ppdn": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_ppsm": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_ppsm": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_ppot": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_ppot": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_jrfm": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_jrfm": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_jrpp": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_jrpp": { "query": "bill" } } }, { "match_phrase": { "fields.text.a_jaen": { "query": "bill" } } }, { "match_phrase": { "fields.textsearchable.a_jaen": { "query": "bill" } } } ] } } ] } }, { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "fields.textsearchable.originalcont": { "query": "snowy adventure" } } }, { "match_phrase": { "fields.textsearchable.a_originalcont": { "query": "snowy adventure" } } } ] } } ] } } ] } } ] } }, "size": 100, "sort": [ { "fields.datetime.date": { "order": "desc", "unmapped_type": "date" } }, { "fields.integer.snum": { "order": "asc", "unmapped_type": "integer" } } ], "stored_fields": [ "fields.text.msgc", "fields.integer.natc", "fields.text.auth", "fields.text.subj", "fields.datetime.date", "fields.integer.size", "fields.textretrievable.tags", "fields.text.ppdn", "fields.text.dtyp", "fields.text.pvid", "fields.integer.snum", "fields.textretrievable.vlid", "fields.text._Vault.MsgType", "fields.intretrievable.itemexpiry", "fields.text.keys", "tags.text._evtag.category", "tags.text._evtag.inclusion", "tags.text._evtag.exclusion" ], "timeout": "600s", "track_total_hits": true }

Sample Search Criteria 2:

FIELD	OPERATOR	SEARCH TERMS
Date	is between	From 01/05/2005 To 01/01/2012
Subject or Content	contains any of	good old days

Search Query Log generated in the elasticsearchreq_*.json:

{ "_source": false, "from": 0, "query": { "bool": { "filter": [ { "term": { "indexVolumeName": { "value": "17845af6064c96b4c80b01c41ec244919_153" } } } ], "must": [ { "bool": { "must": [ { "bool": { "should": [ { "range": { "fields.datetime.date": { "gte": "2005-01-05T05:00:00Z", "lte": "2012-01-02T04:59:59Z" } } }, { "range": { "fields.datetime.a_date": { "gte": "2005-01-05T05:00:00Z", "lte": "2012-01-02T04:59:59Z" } } }, { "range": { "fields.datetimesearchable.a_date": { "gte": "2005-01-05T05:00:00Z", "lte": "2012-01-02T04:59:59Z" } } } ] } }, { "bool": { "should": [ { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "fields.textsearchable.originalcont": { "query": "good" } } }, { "match_phrase": { "fields.text.subj": { "query": "good" } } }, { "match_phrase": { "fields.textsearchable.a_originalcont": { "query": "good" } } }, { "match_phrase": { "fields.text.a_subj": { "query": "good" } } }, { "match_phrase": { "fields.textsearchable.a_subj": { "query": "good" } } } ] } } ] } }, { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "fields.textsearchable.originalcont": { "query": "old" } } }, { "match_phrase": { "fields.text.subj": { "query": "old" } } }, { "match_phrase": { "fields.textsearchable.a_originalcont": { "query": "old" } } }, { "match_phrase": { "fields.text.a_subj": { "query": "old" } } }, { "match_phrase": { "fields.textsearchable.a_subj": { "query": "old" } } } ] } } ] } }, { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "fields.textsearchable.originalcont": { "query": "days" } } }, { "match_phrase": { "fields.text.subj": { "query": "days" } } }, { "match_phrase": { "fields.textsearchable.a_originalcont": { "query": "days" } } }, { "match_phrase": { "fields.text.a_subj": { "query": "days" } } }, { "match_phrase": { "fields.textsearchable.a_subj": { "query": "days" } } } ] } } ] } } ] } } ] } } ] } }, "size": 100, "sort": [ { "fields.datetime.date": { "order": "desc", "unmapped_type": "date" } }, { "fields.integer.snum": { "order": "asc", "unmapped_type": "integer" } } ], "stored_fields": [ "fields.text.msgc", "fields.integer.natc", "fields.text.auth", "fields.text.subj", "fields.datetime.date", "fields.integer.size", "fields.textretrievable.tags", "fields.text.ppdn", "fields.text.dtyp", "fields.text.pvid", "fields.integer.snum", "fields.textretrievable.vlid", "fields.text._Vault.MsgType", "fields.intretrievable.itemexpiry", "fields.text.keys", "tags.text._evtag.category", "tags.text._evtag.inclusion", "tags.text._evtag.exclusion" ], "timeout": "600s", "track_total_hits": true }

The log excerpts, information and/or screenshot(s) above were taken from a non-production test lab environment and are used for example purposes only.

IMPORTANT: Do not edit this content in Information Manager

Internal Notes

The content of this document is referenced in the Enterprise Vault 10.0 Indexing Key Topics. Do not make changes to this article. If you would like changes to be made to this content, use the Information Manager feedback feature. Alternatively, you can send an e-mail to the owner or the Enterprise Vault Knowledge Management team DL-ARC-KM.

Issue/Introduction

How to enable Search Query Logging for EV Indexing

Was this article helpful?

thumb_up Yes

thumb_down No

Welcome to "KB Articles"