Application Master Connectivity Issue


Datameer isn't installed on an edge node and there is a firewall between the Datameer host and the cluster. All jobs fail without any progress. Job logs report the following errors, but the cluster application runs fine and gets completed successfully.

2018-02-01 11:01:17 -  Retrying connect to server: Already tried 0 time(s); maxRetries=45
2018-02-01 11:15:58 - Retrying connect to server: Already tried 44 time(s); maxRetries=45

2018-02-01 11:16:18 - Failed to retrieve AM Status via proxy Call From <datameer hostname>/<datameer IP> to failed on socket timeout exception: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending]; For more details see:


As soon as a Resource Manager accepts the application, it starts the Application Master at one of the DataNodes. Further communication happens between Datameer and the Application Master.

2018-02-01 11:00:55,849 [INFO] [IPC Server Responder] |ipc.Server|: IPC Server Responder: starting
2018-02-01 11:00:55,849 [INFO] [IPC Server listener on 45823] |ipc.Server|: IPC Server listener on 45823: starting
2018-02-01 11:00:55,850 [DEBUG] [IPC Server handler 0 on 45823] |ipc.Server|: IPC Server handler 0 on 45823: starting
2018-02-01 11:00:55,850 [INFO] [ServiceThread:DAGClientRPCServer] |client.DAGClientServer|: Instantiated DAGClientRPCServer at

Datameer tries to connect to this address using the port provided by the Resource Manager. In case this port is closed for incoming connections from the Datameer host, the connection attempt will fail with socket timeout exception.



There are Hadoop properties that control the port range a Resource Manager uses for the Application Master processes. If at the cluster these properties are not set/blank, any available port could be used.

In case you want to dedicate a certain port range to start AM for Datameer executions, it is required to set using the next Hadoop custom properties on the Datameer Admin page. for MR jobs and for Tez jobs.

For example.