Tweet
We have been investigating this surprising scenario recently, where users have found that the Indexer is crashing while downloading the first URL on a Windows Server 2003 machine.
This does not always happen, only on seemingly random occasions.
First thing to note was that we could not reproduce this problem using an older Server 2003 SP2 machine. And we never had any reports of such a behaviour.
It was only after we ran Windows Update and installed all the latest updates that the behaviour changed. So we believe it is a problem with a recent update.
We have narrowed down the crash to happen within the Microsoft WinInet API. This is a component within Windows and it is used by Internet Explorer and many other third party applications that accesses the Internet.
From extensive testing (at a line by line level in the code), the most likely explanation is that this is a Microsoft bug in the WinInet.dll module introduced in a recent Windows Update.
The scenario seems to be limited to Windows Server 2003, and only with the latest updates installed. There must have been a recent change in one of the updates that is evoking this behaviour. This behaviour never occured on Server 2003 for many years. We only started hearing of this problem in the last couple months.
We have also not received reports of the problem (nor been able to replicate the problem) on any other version of Windows besides Server 2003.
In our testing, we found that the crashing seemed to happen less when we disable the need to reload and retrieve "robots.txt". But this probably only reduced the crash rate from 1 out of 5 to 1 out of 10. You may want to try this,
"Configure"->"Spider options":
Uncheck "Reload all files (do not use cache)"
And uncheck "Enable 'robots.txt' support"
The problem however, persists. We will continue investigation as more information become available to us. Please do let us know if you have any further information.
In the meantime, if you wish to truly avoid the problem, you should run the Indexer on a machine that is NOT running Server 2003.
Note that the Indexer can be run on any machine, and you simply upload or copy over the index files to your web server which will be hosting the web site.
This does not always happen, only on seemingly random occasions.
First thing to note was that we could not reproduce this problem using an older Server 2003 SP2 machine. And we never had any reports of such a behaviour.
It was only after we ran Windows Update and installed all the latest updates that the behaviour changed. So we believe it is a problem with a recent update.
We have narrowed down the crash to happen within the Microsoft WinInet API. This is a component within Windows and it is used by Internet Explorer and many other third party applications that accesses the Internet.
From extensive testing (at a line by line level in the code), the most likely explanation is that this is a Microsoft bug in the WinInet.dll module introduced in a recent Windows Update.
The scenario seems to be limited to Windows Server 2003, and only with the latest updates installed. There must have been a recent change in one of the updates that is evoking this behaviour. This behaviour never occured on Server 2003 for many years. We only started hearing of this problem in the last couple months.
We have also not received reports of the problem (nor been able to replicate the problem) on any other version of Windows besides Server 2003.
In our testing, we found that the crashing seemed to happen less when we disable the need to reload and retrieve "robots.txt". But this probably only reduced the crash rate from 1 out of 5 to 1 out of 10. You may want to try this,
"Configure"->"Spider options":
Uncheck "Reload all files (do not use cache)"
And uncheck "Enable 'robots.txt' support"
The problem however, persists. We will continue investigation as more information become available to us. Please do let us know if you have any further information.
In the meantime, if you wish to truly avoid the problem, you should run the Indexer on a machine that is NOT running Server 2003.
Note that the Indexer can be run on any machine, and you simply upload or copy over the index files to your web server which will be hosting the web site.
Comment