How to Implement Debug Logging for sFTP Imports and Exports
Troubleshoot problematic Import Jobs and Export Jobs using an sFTP connection by implementing additional logging to debug and to determine what is preventing a successful job run.
First, occasionally there is a caching problem depending on the version of SSH/sFTP running on the host. Attempt re-running the job with the following Custom Property.
If this does not resolve the issue, remove the above parameter.
To begin debug troubleshooting, configure the artifact in question for a specific execution framework. Then implement the enhanced logging.
For Tez, related jobs should additionally be set within the import-specific
log4j.category.datameer=TRACE log4j.category.datameer.awstasks=DEBUG log4j.category.awstasks.com.jcraft=DEBUG log4j.category.org.apache.hadoop=DEBUG
Often, comparing the sshd_config and ssh_config files from the Datameer Host, Data Nodes, and sFTP Host can be a quick path to resolving sFTP issues. Notably, supported authentication and encryption mechanisms should be identical on all machines.