Cannot create a data link

2 followers
0
Avatar

When I try to create a datalink with Datameer 6.1.5 I get an exception when I press the final Save button (after entering the file name):

[anonymous] ERROR [2016-10-13 14:51:18.932] [qtp563628874-34] (DasExceptionFilter.java:43) - Unhandled exception handling /import-job/create/save
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.StringIndexOutOfBoundsException: String index out of range: -7
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:979)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at datameer.dap.conductor.webapp.filter.MinifyJsCssFilter.doFilter(MinifyJsCssFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.HeaderInjectFilter.doFilter(HeaderInjectFilter.java:24)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.CsrfFilter.doHttpFilter(CsrfFilter.java:87)
at datameer.dap.conductor.webapp.filter.CsrfFilter.doFilter(CsrfFilter.java:66)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.SessionFilter.doFilter(SessionFilter.java:53)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.RestAuthenticationOriginFilter.doFilter(RestAuthenticationOriginFilter.java:52)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.OptimisticLockRetryFilter.doFilter(OptimisticLockRetryFilter.java:32)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.PermissionDeniedExceptionFilter.doFilter(PermissionDeniedExceptionFilter.java:29)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.PersistenceFilter.doFilter(Unknown Source)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:157)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:48)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at datameer.dap.conductor.authentication.CompositeAuthenticationFilter$InternalFilterChain.doFilter(CompositeAuthenticationFilter.java:64)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205)
at datameer.dap.conductor.authentication.CompositeAuthenticationFilter$InternalFilterChain.doFilter(CompositeAuthenticationFilter.java:68)
at datameer.dap.conductor.authentication.CompositeAuthenticationFilter.doFilter(CompositeAuthenticationFilter.java:49)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at datameer.dap.conductor.webapp.controller.fte.AccessTokenBasedAuthenticationProcessingFilter.doFilter(AccessTokenBasedAuthenticationProcessingFilter.java:94)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at datameer.dap.conductor.webapp.filter.AutoAdminAuthenticationFilter.doFilter(AutoAdminAuthenticationFilter.java:65)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.ClickJackingPreventionFilter.doFilter(ClickJackingPreventionFilter.java:40)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.web.multipart.support.MultipartFilter.doFilterInternal(MultipartFilter.java:118)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.OptimisticLockRetryFilter.doFilter(OptimisticLockRetryFilter.java:32)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.PermissionDeniedExceptionFilter.doFilter(PermissionDeniedExceptionFilter.java:29)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.DeprecatedRestUrlsFilter.doFilter(DeprecatedRestUrlsFilter.java:90)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.BrowserExceptionFilter.doFilter(BrowserExceptionFilter.java:31)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.RestExceptionFilter.doFilter(RestExceptionFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.NoResultExceptionFilter.doFilter(NoResultExceptionFilter.java:24)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.DasExceptionFilter.doFilter(DasExceptionFilter.java:29)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.RequestContextFilter.doFilter(RequestContextFilter.java:43)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.authentication.FailedLoginServletFilter.doFilter(FailedLoginServletFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at com.jamonapi.http.JAMonJettyHandlerNew.handle(JAMonJettyHandlerNew.java:36)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -7
at java.lang.String.substring(String.java:1875)
at datameer.dap.common.partition.filerange.StaticPartitionDefinition.createPatternPathAppendix(StaticPartitionDefinition.java:61)
at datameer.dap.common.partition.filerange.StaticPartitionDefinition.<init>(StaticPartitionDefinition.java:49)
at datameer.dap.common.partition.filerange.StaticPartitionDefinition.forDataSource(StaticPartitionDefinition.java:201)
at datameer.dap.common.partition.filerange.FileRangePartitionProvider.<init>(FileRangePartitionProvider.java:36)
at datameer.dap.common.partition.filerange.FileRangePartitioning.createPartitionProvider(FileRangePartitioning.java:23)
at datameer.dap.common.partition.filerange.FileRangePartitioning.createPartitionProvider(FileRangePartitioning.java:8)
at datameer.dap.common.partition.extension.DatameerPartitioningExtension.partitionProvider(DatameerPartitioningExtension.java:31)
at datameer.dap.conductor.webapp.commandObject.DataSourceCommand.createPartitionDefinition(DataSourceCommand.java:420)
at datameer.dap.conductor.webapp.commandObject.DataSourceCommand.fillDataSource(DataSourceCommand.java:412)
at datameer.dap.conductor.webapp.controller.data.SaveController.save(SaveController.java:80)
at datameer.dap.conductor.webapp.controller.data.importjob.DataSourceW7SaveController.saveDatasource(DataSourceW7SaveController.java:79)
at datameer.dap.conductor.webapp.controller.data.importjob.DataSourceW7SaveController.submitSave(DataSourceW7SaveController.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:177)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:446)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:434)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
... 125 more

Michael Ahn

18 comments

  • Avatar
    Konsta Danyliuk

    Hello Michael.

    Mentioned error might be caused by some mismatches in format of dataset you are truing to load.

    • Could you please clarify what exactly data are you trying to ingest into Datameer by this DataLink?

    • Are you able to get correct Data Preview while setting up the DataLink?

    • Please try to check Drop record strategy under How to handle invalid data section and check if this helps the job to run without the error.

    0
  • Avatar
    Michael Ahn

    I try to link to AVRO data I exported from a Workbook through a HDFS connection. I see the correct data preview in the data link wizard. Changing invalid data handling makes no difference.

    0
  • Avatar
    Konsta Danyliuk

    Michael, thank you for additional information.

    What if you will create a base line test for this activity, to narrow down the root cause:

    • create a very simple dataset (lets say 5 or 10 records).
    • export it into HDFS in avro format.
    • ingest it back via the DataLink.

    0
  • Avatar
    Michael Ahn

    I tried with a very simple table:

    Almost the same error:

    [anonymous] ERROR [2016-10-13 15:37:55.508] [qtp563628874-15226] (DasExceptionFilter.java:43) - Unhandled exception handling /import-job/create/save
    org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.StringIndexOutOfBoundsException: String index out of range: -2

    0
  • Avatar
    Konsta Danyliuk

    Thank you, Michael.

    • Do you face the same issues with any other files export from HDFS?
    • Could you save this tiny avro file at local machine and try to upload it directly?
    0
  • Avatar
    Michael Ahn

    Doing an import instead of DataLink works with the HDFS connection. But I need the DataLink.

    0
  • Avatar
    Konsta Danyliuk

    Just to confirm if I properly understand - you are able to ingest mentioned avgo file from HDFS over an ImportJob, but DataLink gives you the error, right?

    0
  • Avatar
    Konsta Danyliuk

    Michael.

    Can you try the following, please

    • Create new folder in HDFS
    • Create new HDFS connection and point it into this folder
    • Copy mentioned avro file (or just the test one) into this folder
    • Create new Data Link and give it another try to ingest the file.

    0
  • Avatar
    Michael Ahn

    I tried it with a new HDFS connection, did not work. I tried the same file with an ssh connection, did not work. I tried other file format (Excel), did not work. I tried another user (admin), did not work. I just cannot create any datalink, I always get that exception on save.

    0
  • Avatar
    Konsta Danyliuk

    Hello Michael.

    Could you please let me know what Datameer version you are using and what is your Hadoop distribution?

    0
  • Avatar
    Konsta Danyliuk

    Michael,

    Let me arrange few tests at my environment.

    btw, have you tried to restart Datameer?

    0
  • Avatar
    Konsta Danyliuk

    Just made few tests at fresh installed 6.1.5 (but for another Hadoop distribution HDP 2.4) - everything works fine, and I was able to create and use DataLinks for HDFS and Hive.

    To get some more inputs, could you please try to switch Datameer to embedded cluster mode, setup connection to Datameer server file system and try to create a DataLink.  

     

     

    0
  • Avatar
    Michael Ahn

    I found the reason for the exception: The file paths.

    I used "/data" as the path in the connection and "/" as path in the datalink. It does also not work the other way round. But now I'm using "/data" in the connection and "/mydir" in the datalink and it works. So I have a workaround, but you should be able to reproduce the bug and open a ticket.

    0
  • Avatar
    Konsta Danyliuk

    Hello Michael.

    Good to hear that you were able to find workaround. Thank you for pointing us to this issue, I'll check the problem with engineers.

     

     

    0
Please sign in to leave a comment.