How to Adjust Hadoop Memory Usage Settings in Datameer 5.6 and later

Goal

This guide will explain the available Custom Properties in Datameer 5.6 and later for adjusting the memory usage of Hadoop tasks. 

Learn

The following parameters were introduced in Datameer 5.6 and are mentioned in the New and Noteworthy section of this release:

Property name
Default value
Description
das.job.map-task.memory 2048 Controls the container size of map-like tasks across execution frameworks.
das.job.reduce-task.memory 2048 Controls the container size of the reduce-like tasks across execution frameworks.
das.job.application-manager.memory 2048 Controls the container size of the Application Master across execution frameworks.

These parameters are to be used universally in Datameer to update the memory usage for Map tasks, Reduce tasks and YARN Application Manager. 

The value of each of these parameters is defined in MB. It is worth noting that the value is the container size, not the JVM memory sizing.

These parameters affect all execution frameworks of Datameer. No longer do separate custom properties need to be defined for MapReduce jobs and TEZ jobs.

Lastly, it is worth noting that these new settings override the execution framework existing settings.