Irregular search result counts (they are both incorrect and they change!)

[rtollert]

I have a particular database search which returns 1105 results (as claimed at the top of the CGI result page). However, this also reproduces in XML.

• At 50 results per page, 23 pages of results are claimed, but I can only go to the 20th page, to result 1000. There are no pages beyond that.
• At 51 results per page, page 20 is still visible, going to result 1004 (!). But it is not accessible from the HTML results - I have to manually change the URL to access that page. The normal HTML results end at page 19.
• Page 20 at 52 results per page still ends at result 1004; page 20 at 53 results per page is not accessible.

I was unable to find documentation for this behavior in the user's guide, the forum or the FAQs, so I wanted to ask about it here. We're resizing a single scrolled window to the length of the total result count, so it really is a big deal if the result count is overestimated or the end of the results are otherwise unavailable. Underestimating the number of results would be preferrable to overestimation in this event.

[David]

For the CGI script, there is some optimisation in the code, which means at most 1000 results are returned. There are the most relevent 1000 results.

There is some discussion about this in this old forum post.

But surely you don't want to display a 1000 results in a single window. Most people don't look beyond the first 3 results.

[rtollert]

While I agree that it is an unlikely situation for the user, it is not inconceivable. Moreover, if results were limited to at most 1000, isn't it a bug that higher result counts are returned? Even if I don't display all 1000 results in a single window, the behavior as it currently stands means that the user will encounter an error of some kind if paging to a result range that the CGI reports is valid.

I'll work around this for now by clamping the result count to 1000.

[David]

It is common practice to limit the results for large sets of index files (and necessary as you don't want to be sorting millions of results that no one will ever see). Google does the same thing and no one seems to mind. Try to navigate to 1100th result in Google and you'll also get an error.

I agree it is not correct that you can manipulate the URL to get 1004 results. It should never return more than 1000. So this might be a (rather minor) bug.