Custom Function that performs lookup

3 followers
0
Avatar

Can I write a custom function that takes in some columns as parameters and then does an hbase lookup to translate this to a different value?

Guhan Kashyap

5 comments

  • Avatar
    Guhan Kashyap

    Thanks Gido. I'm still confused about what the runtime support is. Can I connect to any HBase, HDFS source? Can I connect to other sources like Vertica? Are there limitations to what servers I can connect to or can I pretty much use any Java functionality?

    Also - are there other restrictions on the functions such as side effect restrictions?

    0
  • Avatar
    Gido

    In respect to your most recent question, if you could use pretty much any Java functionality, the answer is yes.

    About your your question what the runtime support is, I am not sure. Can you provide more context regarding this?

    0
  • Avatar
    Guhan Kashyap

    In terms of context - I was thinking of something like using Spark SQL in the custom functions - can that be done?

    0
  • Avatar
    Joel Stewart

    Guhan - I'd be cautious about implementing a custom function that requires feedback from an outside system. This may cause a lot of lag when using the function since the result must be computed and displayed on the sample data. This may provide a poor user experience.

    0
Please sign in to leave a comment.