Updating index files fails

[chinkle]

I've got the indexer set up to run directly on the web server itself and to deposit its index files right in the root directory of the site. It ran perfectly well in testing this way, but after a move to production, the indexing process (running nightly) fails with these kinds of errors at the end of the log:

08|11/01/10 01:52:31|Failed to delete existing file E:\Inetpub\[..]\zoom_dictionary.zdat (File may be held open by search engine - attempt 3 of 3)
08|11/01/10 01:52:32|Failed to delete existing file E:\Inetpub\[..]\zoom_dictionary.zdat (File is held open by another application)

It says this for four of the .zdat files. The site is running on IIS 6.0 in its own application pool. In order to release the hold on the .zdat files that is causing the problem, I have to stop the website AND recycle the application pool. After that, I can delete the files and replace them with new ones.

I want to automate this process without having to shut down the site and recycle the app pool on a daily basis. Is there something else that can be done to release the hold on these files?

chinkle

[Ray]

Are you using Classic ASP, ASP.NET or the CGI/Win32 version on your IIS server?

[chinkle]

Are you using Classic ASP, ASP.NET or the CGI/Win32 version on your IIS server?

I'm using ASP.NET.

[Ray]

Can you confirm that you have the latest ASP.NET Server Control installed? This means you have uninstalled the previous control on your web server, and downloaded and installed the latest from here, onto your web server:
http://www.wrensoft.com/zoom/aspdotnet.html

I can't reproduce this behaviour here, and I'm using the latest version of the server control. Do you know if there's any caching settings in place which might be changing behaviour? What version of Windows are you using?

[chinkle]

I had actually corresponded with you by email on 25 OCT 2010 about another issue wherein you advised me to update my server control. I did so, and it resolved the issue I had... so I'm sure that my software is up-to-date.

I don't believe any caching settings are in place on the web server. We're using IIS 6.0 on Windows 2003 Server (fully patched).

[Ray]

Is there another ASP.NET application running in the same application pool? It's hard to imagine how it might affect it, but I'm just looking at anything related. If so, try creating a separate application pool for Zoom.

I'm guessing you're saying that right after you restart the application pool (and make a request), immediately after that, the files are locked, and you cannot rename or delete any these ZDAT files from the folder (until you stop the pool). Or is it only after a number of searches have been made? Or is it intermittent and sometimes the files are locked and sometimes they're not?

Just so you know, we still can't reproduce this problem, and we just tried it on IIS 6.0 on Windows 2003 Server.

[chinkle]

The application is the only one in its pool. I haven't explicitly tried testing the locked-ness (as opposed to Loch Ness!) of the files because this is a production site. It didn't seem to me that the files were locking immediately because this wasn't a problem during testing or, indeed, in the initial few days the site was online.

I wonder if the problem may be related to high volume... since clearly the test site isn't going to get the same traffic as the production site, and we launched on a Friday, so the first couple days were the weekend (our traffic is higher during the week than on the weekends).

[David]

Can you turn on search word logging in the Zoom configuration. At least then you can see how many searches are being done on the site and the duration of the search (which will more of less correspond to the duration that the files are being locked).

If you have near continuous overlapping searches being done, then this might be linked to the problem. It also might highlight other issues, like searches taking too long to perform on your server.

[chinkle]

Apologies for the very late reply. My boss turned my attention elsewhere, and I'm not getting email when this thread updates.

I actually did have the search logging turned on. We don't get that many searches, especially not at 1:00am, which is when the scheduled task runs. We get a few searches every 5 or 10 minutes. Most people search on a couple things, so there are clusters of 3 or 4 searches at a time by the same user, but the clusters are separated by about 5 to 10 minutes.

[David]

Doesn't sound like this is the cause of the locking then.
How long does each search take?

Does the user who is running the scheduled job have permission to delete files from this folder. Might be just permissions related.

[chinkle]

I've never seen a query take longer than 0.7s. Most queries run faster than 0.1s, with the majority taking around 0.06s. Also, I don't think it's a permissions problem because the process was running fine for a while, and then it intermittently ran into problems. However, when it does run into problems, the files become corrupted (because only a couple of the files get copied), and the search mechanism fails until I can regenerate the files manually and copy them over.

Something else I'm seeing... occasionally the indexing will halt right in the middle of what it's doing. I have the indexer outputting a log file, and successful runs have log files of about 1,418KB. Occasionally (the 16th, 19th, and 20th of this month) the log files were 9 to 13KB with them spontaneously ending in the middle of a word, e.g.:

14|11/16/10 01:50:06|Index Thread got ready buffer for http://my.domain.c

01|11/19/10 01:50:06|Skipping http://my.domain.com/images/News/select_tab.gif (Blocked by extensi

14|11/20/10 01:50:06|Index Thread got ready buffer for h

Since I've been outputting the index files to a different directory and manually copying them over, it's not the same problem I had before.

[Ray]

If it is a scheduled task, make sure you do not have any setting enabled which may terminate the scheduled task if it does not complete within a certain amount of time (or other conditions such as if the computer ceases to be idle, etc.). Check this via Windows Control Panel -> Administrative Tools -> Task Scheduler.

Otherwise we have not had any other reports for freeze/crashes with the current build. Make sure you are using the latest build available (V6.0 build 1023 as of time of post) at this page:
http://www.wrensoft.com/zoom/whatsnew.html

With the updating index files problem -- just reviewing your original post, it has to be a difference in configuration between your development server and your production server. Since you said it never exhibited this problem until you moved to your production server. So if you can find out what is different (especially in terms of policies applied, user accounts used, etc.) then it will help.

[chinkle]

If it is a scheduled task, make sure you do not have any setting enabled which may terminate the scheduled task if it does not complete within a certain amount of time (or other conditions such as if the computer ceases to be idle, etc.). Check this via Windows Control Panel -> Administrative Tools -> Task Scheduler.

Otherwise we have not had any other reports for freeze/crashes with the current build. Make sure you are using the latest build available (V6.0 build 1023 as of time of post) at this page:
http://www.wrensoft.com/zoom/whatsnew.html

The scheduled tasks aren't being terminated... that's not the issue. ZoomIndexer.exe is crashing. I just found some entries in the Application event log:

Event Type: Information
Event Source: Application Error
Event Category: (100)
Event ID: 1004
Date: 11/24/2010
Time: 1:52:45 AM
User: N/A
Computer: SERVER
Description:
Reporting queued error: faulting application ZoomIndexer.exe, version 6.0.0.0, faulting module ntdll.dll, version 5.2.3790.4455, fault address 0x0004cde2.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 46 61 69 6c ion Fail
0010: 75 72 65 20 20 5a 6f 6f ure Zoo
0018: 6d 49 6e 64 65 78 65 72 mIndexer
0020: 2e 65 78 65 20 36 2e 30 .exe 6.0
0028: 2e 30 2e 30 20 69 6e 20 .0.0 in
0030: 6e 74 64 6c 6c 2e 64 6c ntdll.dl
0038: 6c 20 35 2e 32 2e 33 37 l 5.2.37
0040: 39 30 2e 34 34 35 35 20 90.4455
0048: 61 74 20 6f 66 66 73 65 at offse
0050: 74 20 30 30 30 34 63 64 t 0004cd
0058: 65 32 e2

and

Event Type: Information
Event Source: Application Error
Event Category: (100)
Event ID: 1004
Date: 11/24/2010
Time: 1:52:52 AM
User: N/A
Computer: SERVER
Description:
Reporting queued error: faulting application ZoomIndexer.exe, version 6.0.0.0, faulting module rpcrt4.dll, version 5.2.3790.4759, fault address 0x000326da.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 46 61 69 6c ion Fail
0010: 75 72 65 20 20 5a 6f 6f ure Zoo
0018: 6d 49 6e 64 65 78 65 72 mIndexer
0020: 2e 65 78 65 20 36 2e 30 .exe 6.0
0028: 2e 30 2e 30 20 69 6e 20 .0.0 in
0030: 72 70 63 72 74 34 2e 64 rpcrt4.d
0038: 6c 6c 20 35 2e 32 2e 33 ll 5.2.3
0040: 37 39 30 2e 34 37 35 39 790.4759
0048: 20 61 74 20 6f 66 66 73 at offs
0050: 65 74 20 30 30 30 33 32 et 00032
0058: 36 64 61 6da

Let me know if we should discuss this further by email -- perhaps the forum isn't the place for this type of support?

[Ray]

Yes, please e-mail us with more information including your ZCFG configuration file.

Try to create a debug log file to see if it crashes on the same file each time ("Configure"->"Index log" and enable "Save index log to file" and "Debug mode"). Make sure to save your configuration for this to apply to any scheduled task (and "append date and time" if you want multiple files to be created instead of overwriting). Include this log file with your e-mail.

I presume you have also made sure you are using the latest build as mentioned before.

[chinkle]

Will do... thanks.

I presume you have also made sure you are using the latest build as mentioned before.

Yes.