How do I retrieve additional pages from the REST API?

2 followers
0
Avatar

Hi

Just having a first go at using the API. Generally going well, but I am trying to retrieve data from a workbook sheet that has >100000 rows in it, and so far it cuts off at 100000 rows.

I was unsure about how to get additional pages from it; I see on this questions site that it suggests modifying configuration to put more rows on each page... while that could work, it's not an ideal solution for me. On some other APIs I use I can specify something like ?offset=100000 to fetch the second hundred thousand - is there any equivalent to this for Datameer?

Thanks

Saurabh Agashe

Official comment

  • Avatar
    Saurabh Agashe

    If you take a look in conf/default.properties, there is a value for the maximum rest download records. The default for this is 100000. You can adjust the following property to raise this limit:

    rest.download-data.records-max=100000

    I have a relatively standard Datameer install, and it's located on line 230, but the exact line may vary.

    Hope this helps!

    0

2 comments

  • Avatar
    Brian Junio

    I would just like to add that increasing this value over the 100k default configuration can result in slowness felt throughout Datameer. The 'conductor' is responsible for not only managing API calls, but for every process that Datameer must manage.

    It is recommended that you create an ExportJob and offload that work to your cluster.

    0
Please sign in to leave a comment.