Indexer Lags Site

[Flipflop]

I've been using the professional edition with no problems since about April this year. A couple of weeks ago my host moved me to new faster web server they had commissioned and since then when ever I try to index my site it lags out the site to all other visitors while it's being indexed. As soon as you stop crawling the sites performance returns, within a minute. The indexer itself runs for 30 secs or so then returns 404 errors in blocks or from every other thread.

I'm using the CGI version, my web server is IIS 6.0, the site is dynamically driven (portal/forum) and uses an MS SQL DB back end.

I've not had any issues what so ever until the move, I've tried changing it from multi threading to single thread to see if that made a difference but it has not.

Any help would be very much appreciated.

Thank you.

[Ray]

That doesn't sound much like an improvement on your web host's side. Needless to say, while they may have moved you to a "faster new server", you are probably now sharing this "faster server" with twice as many websites, in which case, its probably a slower performing server from a practical point of view.

While a multi-threaded indexer can place a certain amount of load on a server, it should never really lag the server out unless the server is significantly under powered. And considering that you also tried this in single threaded mode, there really is no other explanation besides the fact that the server is inadequate.

The indexer, when running in single threaded mode, is equivalent to a single web user who is quick to follow every link that appears on each web page. This means that, based on the above reported performance, should more than a few users be browsing your website at once, your server will lag (or even worse, return 404). I would say that this is not acceptable performance for your web server, and I would recommend contacting your web host about this.

[Flipflop]

Thanks for the reply.

I have a really good working relationship with my hosts, this new server is not over subscribed, in fact it has barely been populated at the moment as the old servers are still in use as far as I know.

It's obviously some obscure setting or oversight within the new hardware configuaration. The site does get busy but even with a combination of for example 30 visitors and SE bots active this behaviour is not replicated but as soon as you start the crawler it brings the site down. I know this sounds crazy but it does, I've tested and tested and tested. I'm saying it's a fault with Zoom, like I said it is obviously an issue with the new setup but I was wondering if you had seen anything similar before.

I'll see if I can organise some co-ordinated testing between myself and the host and report back if I discover anything useful.

Thanks again.

[Flipflop]

This has been resolved, thank you. There was a restriction on the number of open ports for MS SQL that had not been set correctly.

[Ray]

Glad to hear the problem has been resolved.

For the record, Zoom does not make any direct connections to MS SQL. It will only crawl pages via HTTP so the problem should not have been unique to Zoom at any rate, and instead, was most likely triggered by how the server-side scripts were coded/setup to communicate with the SQL database.