How to Implement Debug Logging for sFTP Imports

Goal

In case of execution errors or a long runtime of file imports using sFTP, it might be necessary to implement additional logging to debug and to find out what is preventing the import from a successful run. 

Learn

First, configure the artifact in question for a specific execution framework. Then implement the enhanced logging. 

For MapReduce v2, related jobs should be set within the import-specific Custom Properties:

das.debug.tasks.logs.collect.force=true
das.execution-framework=MapReduce

For Tez, related jobs should additionally be set within the import-specific Custom Properties:

das.execution-framework=Tez
tez.task.log.level=DEBUG
tez.am.log.level=DEBUG

Set the Default log severity to:

TRACE

And Logging Customization of:

log4j.category.datameer=TRACE
log4j.category.datameer.awstasks=DEBUG
log4j.category.awstasks.com.jcraft=DEBUG
log4j.category.org.apache.hadoop=DEBUG

Further Information

Include Hadoop Task Logs