Error Connecting to Hadoop Cluster (DM Admin)
When attempting to connect a freshly upgraded Datameer to Cloudera 5.9.0, we received the following error message...
java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed
When Datameer is attempting to connect to a secured Cluster environment, it is required to have all security layers declared in Datameer's custom properties. Once Datameer has connected to Hadoop, it can harvest the rest of the configuration details in order to build applications to be processed by Hadoop.
In this particular case, we were required to investigate the settings on the cluster and explicitly set the following in Datameer to match the cluster's configuration:
Please note that starting in Datameer 6, we have an auto configuration option available that is dependent upon having a valid Hadoop client installed on the Datameer host.
Datameer will have direct access to all pertinent configurations through this channel and will require much less configuration from an Admin's point of view.