How to Save Hadoop Task Logs in a Job Trace

Goal

This article will explain how to force Datameer to save Hadoop task logs so that they are accessible by a Job Trace. 

This is a common practice when troubleshooting a Datameer job failure. 

Learn

Follow these steps to enable task collection for a Datameer job:

  • In the Datameer GUI, configure the desired job. 
  • Navigate to the Custom Properties section of the configuration
    • For Import Jobs, Data Links and Export Jobs this is found on the Schedule tab of the configuration in the Advanced section
    • For Workbooks, this is found in the Advanced section of the configuration.
  • Add the following line to the Custom Properties:
    das.debug.tasks.logs.collect.force=true
  • Re-run the job to activate the new parameter.

Once these steps have been completed, the Hadoop task logs will automatically be saved by Datameer and will be available in a Job Trace. The steps to collect a Job Trace are documented here: How To Collect a Job Trace.