Website Unavailable, moving on

[jdenman]

I just ran a test index, using 10 threads, over a few websites, with a 50 page limit. One website was down, and as a result, Zoom tried over and over to download a page...with no happy result.

What I'd like, is if Zoom sees that 20 (variable?) pages have gone through the queue but none were able to download, it will abort trying to download add'l pages from that base URL.

Can this be accomplished as the product stands now, without creating multiple custom configurations?

Thanks!

[Ray]

Just to confirm, do you mean that Zoom _did_ index a few pages from that website at first, and then during indexing, the server became unavailable, and the other pages that were queued would now fail to download? It would attempt all the remaining URLs in the queue for that website, but should stop and move on once those queued URLs have all been attempted.

The above would be expected behaviour. It is somewhat rare since it would only occur when a site goes down during indexing. If it was down before indexing, then the whole site would have been bypassed.

But I'm not entirely sure if the above is what you mean or if you are saying that Zoom is attempting to download the same page, over and over again? That would not be expected behaviour. If this is the case, is there some redirection happening, and what sort of messages are you seeing in the index log?

Nonetheless, for the former, there is currently no automatic way to ask Zoom to give up on a site after x pages have failed to download. You can only limit the number of files per start point to be indexed. We can consider this for a future version.