Searches and Jobs are failing after installing 8.3 Cumulative Hotfix 2

book

Article ID: 100041158

calendar_today

Updated On:

Description

Error Message

The Following Error messages may be present:

On Search Page:

ServletException in '/jsp/search/emailSearchResults.jsp': com.teneo.esa.asyncqueryengine.cachecomponent.CacheComponentException: [#80007] Unexpected exception encountered: com.teneo.esa.admin.system.ComponentException: [#71000] Admin Services Error : - [#71000] Admin Services Error : Reached maximum restart = 1 -

Image

Discovery and Collection Job Status Logs:

11/16/2017 13:22:35   Starting component remotejob and waiting for resources

Note: Job Status logs hang here and do not get any additional Entries

Server Log (x:\cw\v83\logs\server.YYYY-MM-DD.log):

2017-11-15 18:24:46,212 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:) com.teneo.esa.admin.system.runtime.ProtocolException: [#71000] Admin Services Error :  - [#71000] Admin Services Error : Abnormal termination: eDP-App1:49136:0   processpool_component(Process Pool component) 0.2.1.0 "" -
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.admin.system.runtime.AbstractRuntimeContext.start(AbstractRuntimeContext.java:836)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.admin.system.runtime.RuntimeManager.attemptStart(RuntimeManager.java:793)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.admin.system.runtime.RuntimeManager.start(RuntimeManager.java:552)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.admin.system.ComponentService.startReservedComponent(ComponentService.java:409)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at java.lang.reflect.Method.invoke(Method.java:498)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.admin.system.ComponentHelper$ComponentServiceHandler.invoke(ComponentHelper.java:50)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.sun.proxy.$Proxy28.startReservedComponent(Unknown Source)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.admin.system.ComponentBuilder.start(ComponentBuilder.java:247)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.common.processpool.PPCUtil.startProcessPoolComponent(PPCUtil.java:91)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.common.processpool.ProcessPoolComponentManagementService.runService(ProcessPoolComponentManagementService.java:72)
2017-11-15 18:24:46,213 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.admin.service.AbstractService.doRun(AbstractService.java:1179)
2017-11-15 18:24:46,214 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at com.teneo.esa.admin.service.AbstractService.run(AbstractService.java:1099)
2017-11-15 18:24:46,214 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      at java.lang.Thread.run(Thread.java:745)
2017-11-15 18:24:46,214 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:) Caused by: com.teneo.esa.admincli.SuddenDeathException: [#71000] Admin Services Error : Abnormal termination: eDP-App1:49136:0   processpool_component(Process Pool component) 0.2.1.0 "" -
2017-11-15 18:24:46,214 WARN  [STDERR] (ProcessPoolCompService.ProcessPoolCompService:)      ... 16 more

Discovery Remote Job Log (x:\cw\v83\data\\case-logs\\):

ProcessPool Component Log (X:\cw\v83\logs\component-logs\processpool_component\processpool_component@#####-YYYYMMDDHHmmss-0.log):
2017-11-15 19:03:51,325 ERROR [esa.cluster.ClusterManager] (clustermanager.clustermanager:) [#71000] Admin Services Error : create() failed:  - [#80007] Unexpected exception encountered: loading JDBC driver: com.mysql.jdbc.Driver
2017-11-15 19:03:51,326 WARN  [esa.cluster.ClusterManager] (clustermanager.clustermanager:) Service creation error:
com.teneo.esa.common.exception.TeneoRuntimeException: [#80007] Unexpected exception encountered: loading JDBC driver: com.mysql.jdbc.Driver
     at com.teneo.esa.common.database.PersistenceManagerFactory.getJDBCConnectionThroughDriverManager(PersistenceManagerFactory.java:209)
     at com.teneo.esa.common.database.PersistenceManagerFactory.getJDBCConnectionThroughDriverManager(PersistenceManagerFactory.java:183)
     at com.teneo.esa.common.database.PersistenceManagerFactory.getJDBCConnection(PersistenceManagerFactory.java:164)
     at com.teneo.esa.cluster.BootInfo.getConnection(BootInfo.java:260)
     at com.teneo.esa.cluster.BootInfo.getBootInfo(BootInfo.java:123)
     at com.teneo.esa.cluster.ClusterManager.createServiceInternal(ClusterManager.java:94)
     at com.teneo.esa.cluster.ClusterServiceBase.createService(ClusterServiceBase.java:90)
     at com.teneo.esa.admin.service.AbstractService$1.call(AbstractService.java:1245)
     at com.teneo.esa.admin.service.AbstractService.ensureExclusive(AbstractService.java:1404)
     at com.teneo.esa.cluster.ClusterServiceBase.ensureExclusive(ClusterServiceBase.java:254)
     at com.teneo.esa.admin.service.AbstractService.create(AbstractService.java:1240)
     at com.teneo.esa.admin.service.AbstractService.doRun(AbstractService.java:1145)
     at com.teneo.esa.admin.service.AbstractService.run(AbstractService.java:1099)
     at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
     at java.lang.Class.forName0(Native Method)
     at java.lang.Class.forName(Class.java:264)
     at com.teneo.esa.common.database.PersistenceManagerFactory.getJDBCConnectionThroughDriverManager(PersistenceManagerFactory.java:206)
     ... 13 more

Search Cache Log (X:\cw\v83\data\\case-logs\\-remotejob@#####.log:

2017-11-16 13:09:17,481 ERROR [esa.cluster.ClusterManager] (clustermanager.clustermanager:) [#71000] Admin Services Error : create() failed:  - [#80007] Unexpected exception encountered: loading JDBC driver: com.mysql.jdbc.Driver

     
Collection Remote Job log (x:\cw\v83\logs\icp-logs\icp-remoteicpjob@#####.log):

2017-11-16 13:32:24,408 WARN  [esa.cluster.ClusterManager] (clustermanager.clustermanager:) Service creation error:
com.teneo.esa.common.exception.TeneoRuntimeException: [#80007] Unexpected exception encountered: loading JDBC driver: com.mysql.jdbc.Driver
     at com.teneo.esa.common.database.PersistenceManagerFactory.getJDBCConnectionThroughDriverManager(PersistenceManagerFactory.java:209)
     at com.teneo.esa.common.database.PersistenceManagerFactory.getJDBCConnectionThroughDriverManager(PersistenceManagerFactory.java:183)
     at com.teneo.esa.common.database.PersistenceManagerFactory.getJDBCConnection(PersistenceManagerFactory.java:164)
     at com.teneo.esa.cluster.BootInfo.getConnection(BootInfo.java:260)
     at com.teneo.esa.cluster.BootInfo.getBootInfo(BootInfo.java:123)
     at com.teneo.esa.cluster.ClusterManager.createServiceInternal(ClusterManager.java:94)
     at com.teneo.esa.cluster.ClusterServiceBase.createService(ClusterServiceBase.java:90)
     at com.teneo.esa.admin.service.AbstractService$1.call(AbstractService.java:1245)
     at com.teneo.esa.admin.service.AbstractService.ensureExclusive(AbstractService.java:1404)
     at com.teneo.esa.cluster.ClusterServiceBase.ensureExclusive(ClusterServiceBase.java:254)
     at com.teneo.esa.admin.service.AbstractService.create(AbstractService.java:1240)
     at com.teneo.esa.admin.service.AbstractService.doRun(AbstractService.java:1145)
     at com.teneo.esa.admin.service.AbstractService.run(AbstractService.java:1099)
     at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
     at java.lang.Class.forName0(Native Method)
     at java.lang.Class.forName(Class.java:264)
     at com.teneo.esa.common.database.PersistenceManagerFactory.getJDBCConnectionThroughDriverManager(PersistenceManagerFactory.java:206)
     ... 13 more

Cause

The MySQL Upgrader not only upgrades the MySQL server version, but it also replaces the MySQL Java Connector Library with a new version. The issue described in this technote occurs due to the CHF2 installer not preserving the value of a property that points to the upgraded MySQL Java Connector Library. The property value is reverted to the default that shipped with 8.3, so that it points to a libray that no longer exists.

Resolution

Update the affected property in default.properties to point to the correct library. This needs to be done on each system that CHF2 was installed on.

  1. Stop all eDP Services
  2. Open the default.properties file in a text editor such as notepad.
    • x:\CW\V83\config\configs\default.properties
  3. Locate the following properties:
    • esa.mysql.dbmsConnectorJar
    • esa.common.dbms.connector
  4. Edit the value of esa.common.dbms.connector to match the value of esa.mysql.dbmsConnectorJar
    • Before:
      • esa.mysql.dbmsConnectorJar = mysql-connector-java-commercial-5.1.42-bin.jar
      • esa.common.dbms.connector = mysql-connector-java-5.1.40-bin.jar
    • After:
      • esa.mysql.dbmsConnectorJar = mysql-connector-java-commercial-5.1.42-bin.jar
      • esa.common.dbms.connector = mysql-connector-java-commercial-5.1.42-bin.jar
  5. Save and close default.properties
  6. Run Option 7 in Clearwell Utility to deploy the updated properties.
  7. Start Services and verify searches and jobs can now run.

NOTE: This issue has been corrected and will not occur when installing any future 8.3 CHF releases.

Issue/Introduction

After installing 8.3 CHF2, an issue can occur that prevents any searches from running and most jobs from starting properly.
This issue only appears when the 8.3 MySQL Upgrader has been run previous to installing 8.3 CHF2
Powered by Wolken Software