Workbook fails after Upgrade from 6.3.3 to 6.4.4

3 followers
0
Avatar

I have a workbook which is automatically starte every hour. After upgrading Datameer to version 6.4.4 it fails:

 

ERROR [2018-02-07 14:02:10.711] [JobExecutionPlanRunner] (ClusterSession.java:252) - Failed to run cluster job 'Workbook job (3720106): tweet_analysis#evaluated_tweets(Disconnected record stream), TweetExportNew(' [23 sec]
datameer.com.google.common.base.VerifyException: Finished DAG 'Workbook job (3720106): tweet_analysis#evaluated_tweets(Disconnected record stream), TweetExportNew( (2d554b4c-1ac2-4fbe-b69d-109010cd3858)' (application_1517996577677_0640) with state FAILED and diagnostics: [Vertex failed, vertexName=Map for sheets:[Map side join processor, evaluated_tweets, TweetExportNew, Group by step 1, PartitioningRecordFunction{}] (4ce86072-9690-4c19-84cc-91f9ee7d2c79), vertexId=vertex_1517996577677_0640_9_03, diagnostics=[Task failed, taskId=task_1517996577677_0640_9_03_000001, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.ArrayIndexOutOfBoundsException: 3
	at datameer.dap.sdk.common.Record.getFieldContent(Record.java:132)
	at datameer.dap.common.graphv2.RecordDistributor$SelectedColumnsDistributor.distribute(RecordDistributor.java:131)
	at datameer.plugin.tez.processing.TezPartitioner.getPartition(TezPartitioner.java:26)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:284)
	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163)
	at datameer.plugin.tez.processing.VertexOutput$Partitioned.write(VertexOutput.java:65)
	at datameer.plugin.tez.processing.CombinedVertexOutput.write(CombinedVertexOutput.java:22)
	at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:38)
	at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:27)
	at datameer.dap.sdk.sequence.Sequence.forEach(Sequence.java:1025)
	at datameer.plugin.tez.processing.SimpleVertexProcessor.run(SimpleVertexProcessor.java:121)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.ArrayIndexOutOfBoundsException: 3
	at datameer.dap.sdk.common.Record.getFieldContent(Record.java:132)
	at datameer.dap.common.graphv2.RecordDistributor$SelectedColumnsDistributor.distribute(RecordDistributor.java:131)
	at datameer.plugin.tez.processing.TezPartitioner.getPartition(TezPartitioner.java:26)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:284)
	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163)
	at datameer.plugin.tez.processing.VertexOutput$Partitioned.write(VertexOutput.java:65)
	at datameer.plugin.tez.processing.CombinedVertexOutput.write(CombinedVertexOutput.java:22)
	at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:38)
	at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:27)
	at datameer.dap.sdk.sequence.Sequence.forEach(Sequence.java:1025)
	at datameer.plugin.tez.processing.SimpleVertexProcessor.run(SimpleVertexProcessor.java:121)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.ArrayIndexOutOfBoundsException: 3
	at datameer.dap.sdk.common.Record.getFieldContent(Record.java:132)
	at datameer.dap.common.graphv2.RecordDistributor$SelectedColumnsDistributor.distribute(RecordDistributor.java:131)
	at datameer.plugin.tez.processing.TezPartitioner.getPartition(TezPartitioner.java:26)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:284)
	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163)
	at datameer.plugin.tez.processing.VertexOutput$Partitioned.write(VertexOutput.java:65)
	at datameer.plugin.tez.processing.CombinedVertexOutput.write(CombinedVertexOutput.java:22)
	at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:38)
	at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:27)
	at datameer.dap.sdk.sequence.Sequence.forEach(Sequence.java:1025)
	at datameer.plugin.tez.processing.SimpleVertexProcessor.run(SimpleVertexProcessor.java:121)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.ArrayIndexOutOfBoundsException: 3
	at datameer.dap.sdk.common.Record.getFieldContent(Record.java:132)
	at datameer.dap.common.graphv2.RecordDistributor$SelectedColumnsDistributor.distribute(RecordDistributor.java:131)
	at datameer.plugin.tez.processing.TezPartitioner.getPartition(TezPartitioner.java:26)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:284)
	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163)
	at datameer.plugin.tez.processing.VertexOutput$Partitioned.write(VertexOutput.java:65)
	at datameer.plugin.tez.processing.CombinedVertexOutput.write(CombinedVertexOutput.java:22)
	at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:38)
	at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:27)
	at datameer.dap.sdk.sequence.Sequence.forEach(Sequence.java:1025)
	at datameer.plugin.tez.processing.SimpleVertexProcessor.run(SimpleVertexProcessor.java:121)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:2, Vertex vertex_1517996577677_0640_9_03 [Map for sheets:[Map side join processor, evaluated_tweets, TweetExportNew, Group by step 1, PartitioningRecordFunction{}] (4ce86072-9690-4c19-84cc-91f9ee7d2c79)] killed/failed due to:OWN_TASK_FAILURE], Vertex killed, vertexName=Reduce for sheets:[Group by operation, CompactTweetExport] (ca31e534-516c-430e-bd2f-ef8a16ba0795), vertexId=vertex_1517996577677_0640_9_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:400, Vertex vertex_1517996577677_0640_9_04 [Reduce for sheets:[Group by operation, CompactTweetExport] (ca31e534-516c-430e-bd2f-ef8a16ba0795)] killed/failed due to:OTHER_VERTEX_FAILURE], DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1]
	at datameer.com.google.common.base.Verify.verify(Verify.java:125)
	at datameer.plugin.tez.TezJob.runTezDag(TezJob.java:178)
	at datameer.plugin.tez.TezJob.runImpl(TezJob.java:152)
	at datameer.dap.common.graphv2.ClusterJob.run(ClusterJob.java:115)
	at datameer.dap.common.graphv2.ClusterSession.execute(ClusterSession.java:229)
	at datameer.dap.common.graphv2.ClusterSession.runAllClusterJobs(ClusterSession.java:343)
	at datameer.dap.common.graphv2.JobExecutionPlanRunner.run(JobExecutionPlanRunner.java:122)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:120)
	at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:197)
	at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:34)
	at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:30)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at datameer.dap.common.impersonation.ClusterAwareUgiImpersonator.doAs(ClusterAwareUgiImpersonator.java:43)
	at datameer.dap.common.impersonation.ConfigurableImpersonator.doAs(ConfigurableImpersonator.java:34)
	at datameer.dap.common.security.RunAsThread.run(RunAsThread.java:30)
 INFO [2018-02-07 14:02:10.711] [JobExecutionPlanRunner] (ClusterSession.java:255) - -------------------------------------------
Michael Ahn

3 comments

  • Avatar
    Alan Mark

    Hi Michael,

    Please open up a ticket for this and attach the full job trace, as well as the YARN log for the job in question.

    Thanks,

    Alan Mark

    0
Please sign in to leave a comment.