How to Configure the Number of Mappers and Reducers

Goal

Go beyond Finding the optimal split size / split count and learn how to configure the number of mapper and reducers.

Learn

The property das.splitting.max-split-count applies to mappers only. 

As Datameer Custom Properties outlines, it is possible to configure the number of reducers. If you want to enforce a specific reduce count, set the property das.job.reduce.tasks to greater then zero (0).

If the reducer count should be adapted based on the split-count (map-task count), set the property

das.job.reduce.tasks.correct=true

Be aware that setting either property can influence the performance impact on job execution.