Datameer Package for Cloudera 5.8 Running Against 5.8.3 Shows a Warning: No TokenRenewer defined for token kind kms-dt

Problem

A Datameer package for Cloudera 5.8, such as Datameer-6.1.18-cdh-5.8.0, running against a cluster with Cloudera 5.8.3 displays the following warning message during job execution:

No TokenRenewer defined for token kind kms-dt

Symptoms

When a Datameer environment is configured to use Key Management Server (KMS) to issue delegation tokens, jobs might fail when a token needs to be renewed. The following stack trace is logged in job.log:

WARN [2017-02-10 15:05:05.438] [MrPlanRunnerV2] (Token.java:365) - No TokenRenewer defined for token kind kms-dt
WARN [2017-02-10 15:05:05.438] [MrPlanRunnerV2] (Token.java:124) - Cannot find class for token kind kms-dt
WARN [2017-02-10 15:05:05.439] [MrPlanRunnerV2] (SecurityUtil.java:124) - Failed to cancel token Kind: kms-dt, Service: 192.168.225.172:16000, Ident: 00 0f 73 65 72 76 69 63 65 64 61 74 61 6d 65 65 72 04 79 61 72 6e 00 8a 01 5a 28 69 a9 48 8a 01 5a 4c 76 2d 48 8d 03 5a ad 61 because Token cancel is not supported  for kms-dt tokens
WARN [2017-02-10 15:05:05.467] [MrPlanRunnerV2] (Client.java:1455) - interrupted waiting to send rpc request to server
java.lang.InterruptedException
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
    at java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1055)
    at org.apache.hadoop.ipc.Client.call(Client.java:1450)
    at org.apache.hadoop.ipc.Client.call(Client.java:1408)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy145.cancelDelegationToken(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.cancelDelegationToken(ApplicationClientProtocolPBClientImpl.java:336)
    at sun.reflect.GeneratedMethodAccessor607.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy146.cancelDelegationToken(Unknown Source)
    at org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.cancel(RMDelegationTokenIdentifier.java:129)
    at org.apache.hadoop.security.token.Token.cancel(Token.java:395)
    at datameer.dap.common.graphv2.hadoop.SecurityUtil.finaliseDelegationTokenFile(SecurityUtil.java:122)
    at datameer.plugin.spark.cluster.SparkLauncherClusterJob.runImpl(SparkLauncherClusterJob.java:135)
    at datameer.dap.common.graphv2.ClusterJob.run(ClusterJob.java:125)
    at datameer.dap.common.graphv2.ClusterSession.execute(ClusterSession.java:206)
    at datameer.dap.common.graphv2.mixedframework.MixedClusterSession.execute(MixedClusterSession.java:48)
    at datameer.dap.common.graphv2.ClusterSession.runAllClusterJobs(ClusterSession.java:343)
    at datameer.dap.common.graphv2.MrPlanRunnerV2.run(MrPlanRunnerV2.java:129)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:109)
    at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:186)
    at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:34)
    at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:30)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at datameer.dap.common.filesystem.Impersonator.doAs(Impersonator.java:31)
    at datameer.dap.common.security.RunAsThread.run(RunAsThread.java:30)

Cause

Cloudera made fixes related to token renewal to address underlying defects in maintenance release such as CDH 5.8.3. HADOOP-13155 is one of them. 

Datameer packages such as Datameer-6.1.18-cdh-5.8.0 are bundled with JAR files from 5.8 distribution. Those JAR files don't have the latest implementation of DelegationTokenRenewer.

Solution

Implement the following steps:

  1. Check Cloudera package version 
    [datameer@<host> ~]$ cat /usr/lib/hadoop/cloudera/cdh_version.properties
    
    # Autogenerated build properties
    version=2.6.0-cdh5.8.3
    git.hash=<hash>
    cloudera.hash=<hash>
    cloudera.cdh.hash=<hash>
    cloudera.cdh-packaging.hash=<hash>
    cloudera.base-branch=cdh5-base-2.6.0
    cloudera.build-branch=cdh5-2.6.0_5.8.3
    cloudera.pkg.version=2.6.0+cdh5.8.3+<ver>
    cloudera.pkg.release=1.cdh5.8.3.p<release>
    cloudera.cdh.release=cdh5.8.3
    cloudera.build.time=<buildTime>
    
    cloudera.pkg.name=hadoop
  2. Stop the Datameer service.
  3. Locate the following JAR files in <Datameer>/webapps/conductor/WEB-INF/lib directory and move it outside of the Datameer directory structure as a backup:
    hadoop-common-2.6.0-cdh5.8.0.jar
    avro-1.7.6-cdh5.8.0.jar
    hadoop-hdfs-2.6.0-cdh5.8.0.jar
    hadoop-mapreduce-client-shuffle-2.6.0-cdh5.8.0.jar
    hadoop-mapreduce-client-app-2.6.0-cdh5.8.0.jar
    hadoop-openstack-2.6.0-cdh5.8.0.jar
    hadoop-annotations-2.6.0-cdh5.8.0.jar
    hadoop-mapreduce-client-common-2.6.0-cdh5.8.0.jar
    hadoop-yarn-api-2.6.0-cdh5.8.0.jar
    hadoop-auth-2.6.0-cdh5.8.0.jar
    hadoop-mapreduce-client-core-2.6.0-cdh5.8.0.jar
    hadoop-yarn-client-2.6.0-cdh5.8.0.jar
    hadoop-aws-2.6.0-cdh5.8.0.jar
    hadoop-mapreduce-client-hs-2.6.0-cdh5.8.0.jar
    hadoop-yarn-common-2.6.0-cdh5.8.0.jar
    hadoop-common-2.6.0-cdh5.8.0.jar
    hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.8.0.jar
    hadoop-yarn-server-common-2.6.0-cdh5.8.0.jar
    hadoop-distcp-2.6.0-cdh5.8.0.jar
    hadoop-mapreduce-client-jobclient-2.6.0-cdh5.8.0.jar
  4. Locate all the corresponding JAR files from CDH 5.8.3 distribution and copy it to the above location.
  5. Start the Datameer service.