Connection timed out

[chriscsanyi]

I have notice that we are getting a few Connection timed out errors in log like the one below. Should we change how soon the spider starts on the next thread? Will that help reduce the amount of timeouts the indexer is receiving maybe?

11|01/05/12 12:15:47|Queued URL: http://www.ncsl.org/default.aspx?tabid=22336
14|01/05/12 13:34:23|DL Thread #1, got URL (http://www.ncsl.org/default.aspx?tabid=22336) off queue
04|01/05/12 13:34:23|Downloading file http://www.ncsl.org/default.aspx?tabid=22336
09|01/05/12 13:35:23|Could not download file:
http://www.ncsl.org/default.aspx?tabid=22336 (Connection timed out)

[David]

Hard to know the solution without knowing the cause.

Possible causes are,
- A unreliable network connection. Wireless is notorious for this.
- A server that is overloaded some of the time.
- A bug in the ASPX code that generates the page in question on your site.
- Overactive security software (e.g. some type of firewall that starts blocking after a certain number of requests per second).
- Throttling on the server itself. e.g. maybe the site only serves 1 page per sec per IP address as a way to stopping to much load on the server.
- A bug in the web server software (as apposed to the web page). For example we have seen IIS leak resources and then crash from time to time.

Depending on the cause reducing the threads and increasing spider throttling in Zoom might help.

Deeper investigation is really required. E.g. looking at error logs on the server, taking TCP/IP traces, looking at Zoom logs to see if just a specific subset of pages are effected or if it is random or seeing if the problem occurs periodically, e.g. each 500 pages indexed.

[chriscsanyi]

I changed it to 1 sec delay and it did help but still had a few timeouts. Going to look at make a little longer to see if that will help out. Thanks for all the ideas to look at. Chris

[David]

A modern server should be able to serve up more than 10 pages per second. So if there are any failures at 1 page / sec I think there is something else going wrong.

[chriscsanyi]

It has been better at 8 threads and 1/2 sec delay. I am going to push that to a full 1 sec to see if I can caputre the last few documents. The Previous settings I had it at 10 threads with no delay.

[chriscsanyi]

Finally figured out the problem is with the session login options. I would setup IE with a search login and it seems to be what created the problem with timeouts.