Graphics

Does anyone have any tricks to receiving the entire payload from a paginated REST API? 

I have an API that only returns 25 rows per scrape and I need all 60 rows. It accepts the argument ?page=2  but I have not found a way to have Datalinq server scrape all pages and present me with the full data sample.

Off the cuff ...

Sounds like you are getting data from a third party service.

Communicate with the service (or fuzz it {don't}) to see if you can get more data per page.

Or,

you need an internal server that scrapes data that takes multiple queries and then formats the data for Datalinq. This is the sort of thing Raspberry PI was born for.

Does anyone have any tricks to receiving the entire payload from a paginated REST API? 

I have an API that only returns 25 rows per scrape and I need all 60 rows. It accepts the argument ?page=2  but I have not found a way to have Datalinq server scrape all pages and present me with the full data sample.

Thats what I was afraid of. For this event I can probably just spin up a Node server that grabs all pages and then spits out the full JSON (or hosts it for the RSS provider to grab). I was hoping I was missing something simple to force DLS to scrape all pages of a result but sounds like this is more of an api-side issue than it is something DLS can solve natively. 

Off the cuff ...

Sounds like you are getting data from a third party service.

Communicate with the service (or fuzz it {don't}) to see if you can get more data per page.

Or,

you need an internal server that scrapes data that takes multiple queries and then formats the data for Datalinq. This is the sort of thing Raspberry PI was born for.

Does anyone have any tricks to receiving the entire payload from a paginated REST API? 

I have an API that only returns 25 rows per scrape and I need all 60 rows. It accepts the argument ?page=2  but I have not found a way to have Datalinq server scrape all pages and present me with the full data sample.

What is the whole query string? Is it just page?

Thats what I was afraid of. For this event I can probably just spin up a Node server that grabs all pages and then spits out the full JSON (or hosts it for the RSS provider to grab). I was hoping I was missing something simple to force DLS to scrape all pages of a result but sounds like this is more of an api-side issue than it is something DLS can solve natively. 

Off the cuff ...

Sounds like you are getting data from a third party service.

Communicate with the service (or fuzz it {don't}) to see if you can get more data per page.

Or,

you need an internal server that scrapes data that takes multiple queries and then formats the data for Datalinq. This is the sort of thing Raspberry PI was born for.

Does anyone have any tricks to receiving the entire payload from a paginated REST API? 

I have an API that only returns 25 rows per scrape and I need all 60 rows. It accepts the argument ?page=2  but I have not found a way to have Datalinq server scrape all pages and present me with the full data sample.