Retrieving Job Status from Job History - NullPointerException: HistoryClientService$HSClientProtocolHandler.getCounters


DM: 4.5 and higher, OS: Linux, DIST: CDH5.1, COM: Kerberos, YARN

This issue has been observed with CDH 5.1 clusters, but may also pertain to other cluster environments as well. 


After configuring Datameer to connect to a Kerberos enabled Hadoop cluster with YARN, the user attempts to run a job on the Datameer host (i.e. the Cluster Health job). The job completes successfully according to the YARN Job Tracker, but the job fails according to the Datameer Server. When reviewing the job logs, the following stack trace is logged:

WARN [2014-01-01 00:00:00.000] [MrPlanRunnerThread-0] ( - PriviledgedActionException as:datameer@DATAMEER.COM (auth:KERBEROS) org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(
	at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(
	at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
	at org.apache.hadoop.ipc.RPC$
	at org.apache.hadoop.ipc.Server$Handler$
	at org.apache.hadoop.ipc.Server$Handler$
   	at Method)
	at org.apache.hadoop.ipc.Server$


The cause is a settings issue between the Datameer Server and the Hadoop cluster.


To resolve this issue, follow these steps to add a custom Hadoop property to the Datameer settings.

  1. Check the current value of the property on the Hadoop cluster.
  2. Using the value, add a line to the Custom Property settings of the Hadoop Cluster page on the Administration tab.
  3. Save these settings and then re-run the affect job to verify that the getCounters method no longer generates a NullPointerException.