MapBufferTooSmallException: Record too large for in-memory buffer


Recommendations under Container is Running Beyond Physical Memory Limits were followed and increasing Mapper and Reducer memory partitions to 16G did not improve the issue. The following error message is received:

task:org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTooSmallException: Record too large for in-memory buffer. Exceeded buffer overflow limit, bufferOverflowRecursion=2, bufferList.size=1, blockSize=104857600


1. First add the following to the Custom Properties of the job:


2. If the job still fails, remove the above property and add the following:

  • tez.runtime.sorter.class=LEGACY

3. If execution also fails, attempt the job with both properties added.

Background Information

Regarding the property, you have a look into Hortonworks documentation about Configuring Tez and Demystify Apache Tez Memory Tuning - Step by Step