How to Interpret the Contents of a Job Trace
Understand the individual files captured by a job trace from the perspective of the job-trace-creation.log, which is part of a job trace.
While a job is executing, the job-trace-creation.log is created to track logging activity. This log gives an overall view of what is successfully captured.
job.log - Logging job execution activity. This file is of the most interest to the support team.
job-plan-original.dot - Original definition of the job which is sent to the Hadoop Cluster.
job-plan-complied.dot - Modified definition of the job which is sent to the Hadoop Cluster. This file contains the reordering of job sequences to be processed, by understanding the dependencies.
job-definition.json - Job definition in JSON format, compatible with the Datameer REST API.
job-definition-<xxx>.json - xxx denotes the original file name of the job which is incorporated as a part of this file name. This file defines job specifics in JSON format.
job-conf.xml - Job configuration used when running jobs locally.
job-conf-cluster.xml - When the execution framework is Tez or SparkClient, this file is logged. It merges the Datameer configuration with the Hadoop configuration.
tasklog-spark-submit.log - When the execution framework is SparkCluster, this file is logged and it contains an account of all activities for the tasks executed on this particular job.
When there are exceptions, a different error log file with a different name is created.