How to Enable Tez History/UI for HDP/MapR?

We've found the following setup working

Hadoop Cluster

  • Since your Hadoop cluster needs a running Timeline Server, check that the following properties are enabled in yarn-site.xml
    ...
    <property>
    <name>yarn.timeline-service.enabled</name>
    <value>true</value>
    </property>
    ...
    <property>
    <name>yarn.timeline-service.hostname</name>
    <value>localhost</value><!-- configure a proper hostname that is available for all nodes -->
    </property>
    ...
    <property>
    <name>yarn.timeline-service.http-cross-origin.enabled</name>
    <value>true</value>
    </property>
    ...
    <property>
    <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
    <value>true</value>
    </property>
    ...
  • It runs per default on port 8188
  • If it is not running, start it with ./sbin/yarn-daemon.sh start timelineserver

On Datameer Application Server

  • Download and unpack Apache Tomcat 8.5.20
  • Configure the HTTP port in conf/server.xml e.g. 8280
  • Download and unzip Apache Tez 0.7.1 binaries
  • Delete everything under TOMCAT_HOME/webapps/*
  • Unzip the tez-ui-0.7.1.war into TOMCAT_HOME/webapps/ROOT
  • Edit TOMCAT_HOME/webapps/ROOT/scripts/configs.js
    • timelineBaseUrl must point to the Timeline Server (ATS)
    • RMWebUrl must point to the Resource Manager (RM)
  • Double check that JAVA_HOME is set, (It should be set as it's a pre-requirement for Datameer)
  • Start the server with TOMCAT_HOME/bin/catalina.sh start
  • Configure a test job with Hadoop Custom Properties and Debug Logging Implemented

    das.execution-framework=Tez
    das.debug.tasks.logs.collect.force=true
    tez.task.log.level=DEBUG
    tez.am.log.level=DEBUG
    tez.allow.disabled.timeline-domains=true
    tez.am.history.logging.enabled=true
    tez.dag.history.logging.enabled=true tez.history.logging.service.class=org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService
    tez.tez-ui.history-url.base=http://<thisHost>:8080/#/main/view/TEZ/tez_cluster_instance
    yarn.timeline-service.enabled=true
    yarn.timeline-service.hostname=<ATS> 
  • If you have a Kerberos Secured cluster you should configure the cluster in the yarn-site.xml as described in section Security Configuration

    yarn.timeline-service.principal=datameer@EC2.INTERNAL
    yarn.timeline-service.keytab=/home/datameer/datameer.keytab 

Test

  • Execute test job
  • Gather and review logs, yarn-site.xml, Tomcat/catalina.log, full job trace
  • Check that everything is working properly