Distributed Architecture nodes become disabled with SystemClockValidator errors in the server log

book

Article ID: 100033752

calendar_today

Updated On:

Description

Error Message

DETAIL [cluster.node.state] (clusterupdateservice.clusterupdateservice:) attempting to connect to cluster master at :2595
INFO  [common.database.CWDate] (clusterupdateservice.clusterupdateservice:) Skipping propagation of timezone to shared db
INFO  [cluster.node.state] (clusterupdateservice.clusterupdateservice:)  
INFO  [cluster.node.state] (clusterupdateservice.clusterupdateservice:) System time of local appliance differs from that of master by at least seconds (network delay is milliseconds).  The system clock time of local appliance will be adjusted to cluster masters time.
INFO  [cluster.node.state] (clusterupdateservice.clusterupdateservice:) 
ERROR [cluster.node.state] (clusterupdateservice.clusterupdateservice:) [#156077] Unable to add appliance :2595 to cluster with master appliance :2595. Its environment is not compatible with the environment of this Clearwell cluster.
java.lang.RuntimeException: 
    at com.teneo.esa.cluster.SystemClockValidator.triggerRestartService(SystemClockValidator.java:237)
    at com.teneo.esa.cluster.SystemClockValidator.updateEnvironment(SystemClockValidator.java:228)
    at com.teneo.esa.cluster.ClusterMasterService.validateEnvironment(ClusterMasterService.java:3820)
    at com.teneo.esa.cluster.ClusterMasterService.validateVersions(ClusterMasterService.java:3796)
    at com.teneo.esa.cluster.ClusterUpdateService.connectToMaster(ClusterUpdateService.java:277)
    at com.teneo.esa.cluster.ClusterUpdateService.createServiceInternal(ClusterUpdateService.java:196)
    at com.teneo.esa.cluster.ClusterServiceBase.createService(ClusterServiceBase.java:74)
    at com.teneo.esa.admin.service.AbstractService$1.call(AbstractService.java:1229)
    at com.teneo.esa.admin.service.AbstractService.ensureExclusive(AbstractService.java:1388)
    at com.teneo.esa.cluster.ClusterServiceBase.ensureExclusive(ClusterServiceBase.java:238)
    at com.teneo.esa.admin.service.AbstractService.create(AbstractService.java:1224)
    at com.teneo.esa.admin.service.AbstractService.doRun(AbstractService.java:1129)
    at com.teneo.esa.admin.service.AbstractService.run(AbstractService.java:1083)
    at java.lang.Thread.run(Thread.java:745)
ERROR [esa.cluster.ClusterUpdateService] (clusterupdateservice.clusterupdateservice:) [#71000] Admin Services Error : create() failed:  - [#156077] Unable to add appliance :2595 to cluster with master appliance :2595. Its environment is not compatible with the environment of this Clearwell cluster.

 

Cause

It is possible for worker node system clocks to get out of sync with the Cluster Master if all systems are not getting their system clock times from an authoritative network time source. 

 

Resolution

On all members of the cluster, including the Cluster Master, confirm the W32Time registry key is not set to "NoSync".

1. Use the Registry Editor and browse to HKLM\SYSTEM\CurrentControlSet\Services\W32time\Parameters.
2. Confirm the Type is not set to "NoSync"; it should either be set to "NTP" (a manually configured time source, set using the w32tm Windows command), or "Nt5DS" (synchronize with domain hierarchy).

Note: A server restart will be required in the event that this is changed.

Issue/Introduction

In a Distributed Architecture implementation, worker nodes repeatedly become disabled or otherwise inaccessible to case users with SystemClockValidator errors being logged in the server log file. Timezones are correctly set on all members of the cluster.