API calls crashing on higher max row parameter

Bugs
1

API requests with a higher maximal results parameter (rows) crashing sometimes with a timeout error.
Example:

https://www.natbib-lod.org/publications.json?q=publications+whose+contributor+is+https%3A%2F%2Fwww.natbib-lod.org%2Fagents%2F601704984733604+as+rpbl&rows=5000

2

Thank you @sascha , our technicians will also look into this.
Please bear with us while we dive into your reports.
CC @a.gazzarini

3

Hi @sascha I will create a ticket for better handling this scenario and have the system to return a proper error message.
In the meantime, the short answer is: this is expected. You should “paginate” the request and fetch the results page by page. Search systems are not supposed to retrieve a huge number of results (that interaction is usually considered an anti-pattern). Instead of fetching 5000 rows at a time, you should move forward page by page (as it happens in search pages) by asking a small number of items (10,20, even 50).
Other options that come in my mind are:

  • an export API which could use a different approach under the schema and would allow a higher “row” number
  • an offline, on-demand, request for huge exports
4

It is ok, that the system denies to high row amounts. And of course, I can implement paginated requests. But it is not good, that the system answers in this error case with plain unformatted HTML. There should be better a JSON formated error message delivered.
Btw I used value rows=5000 in my post only for demonstration purpose. I’ve set the rows value in my queries to 500 and apply offset parameter in case of higher totalAmount. Do You think 500 is acceptable for the VDE system resources? Otherwise I can adjust the value to a lower/higher amount.

5

I guess that you issued the request using a browser. In that case, that is expected.
The REST paradigm requires the server to return a proper HTTP code, which in this case, I guess, it’s a 5xx meaning “some internal error occurred, I cannot handle the request”
The browser interprets that response and provides a default error page.

Shortly, what is important for the client is the HTTP error that informs it about the error. Specifically

  • is that something that has to do with the client-side (e.g., malformed request)? → 4xx error codes
  • is that something that has to do with the server-side (e.g., timeout, internal error)? → 5xx error codes

While in the first scenario, a message in the response body could be useful to inform the client about the “wrong” behaviour, the 5xx category tells something like “there’s an error and you, as a client, can’t do anything about that”

6