Announcement

**Ray** · Jul-14-2009, 01:32 AM

Can you contact us via e-mail with more information.

I presume your simultaneous incremental indexes are writing to different output folders.

What are the sizes involved (the number of pages indexed, the number of pages being added, the number of unique words, the total file size of the existing ZDAT files you are incrementally adding to).

How much memory is available on the machine and how much is being utilized? (this is reported on the "Status" tab of the Indexer). If these are very big indexes, it is possible that your system memory is being exhausted, especially with multiple indexers running at the same time.

Ideally, if possible, it would be most helpful to have the actual index files you are incrementally adding to, your ZCFG configuration file, and an accurate description of the incremental operation you are doing to trigger this crash (e.g. was it an "incremental add start points", or "incremental update", and what URLs you were adding, etc.). In other words, enough details for us to try and reproduce the behaviour here.

Also, have you tried doing the same operation but not running multiple instances of the indexer at the same time, and did the crash occur?

**maryjili** · Jul-15-2009, 01:32 PM

I just tried running the crashed incremental indexing again with no other indexing operations, and they crashed again. I also tried with different urls and got same results.

I was running from command line using "incremental adding pages" option for all the four incremental operations. I am not running out of memory and the crash happened after each successfully finishing 6 rounds of incremental indexing. The two operations that were running without problem were using the same ZCFG file with the crashed ones. The unique words were around 2,000,000 when the two indexing crashed and the pagetext files were 2GB.

**David** · Jul-15-2009, 07:37 PM

Ideally, if possible, it would be most helpful to have the actual index files you are incrementally adding to & your ZCFG configuration file. Can you zip them up & make them available via download.

Even though you are using release v6.0.1016 of the software now, did you create the initial set of index files with an earlier release of the software?

**maryjili** · Jul-16-2009, 12:35 AM

I cannot provide the files you requested. I have tested the indexed files and they are working fine. So I guess the zoomindexer64.exe file might be broken due to the crash. I just reinstalled the program and am now waiting to see what will happen. I found that the loading time for each new round of incremental indexing is very long, usually around 2 to 3 hours for 100,000 urls. Maybe four simultaneous loading of urls caused the application to crash? The initial set of index files were also created by v6.0 1016.

**Ray** · Jul-16-2009, 01:54 AM

A crash will not corrupt or break the executable.

If you turn on debug mode logging ("Configure"->"Index Log"->"Debug mode" and make sure to SAVE the configuration change to the file you're running from the command line), then you should be able to provide us with log files to indicate when the crash occurred during the indexing process.

If you can't give us the files (I presume due to data confidentiality and not because you don't know how -- but since you don't say, I'm guessing again), at least provide as much information as possible. There were many other things I mentioned before that were not answered: what were the number of files indexed (per index), the total size of all the ZDAT files (per index), the amount of memory you have installed on your machine. Please look at my previous post.

Also tell us the version of Windows you are using. The hardware you are using.

And I also suggested you try to see if it happens when you are not running multiple indexers simultaneously.

If you can't give us the files, we can't reproduce it here. So you'd need to give us as much information as possible for us to look into it.

**Ray** · Jul-16-2009, 02:04 AM

It's also worth pointing out that running several instances simultaneously, when dealing with such large data sets on the same computer, you are very likely to be slowing down each instance as it is bounded by the limitations of the hard drive speed, the ram, and the CPU available. I would suspect the several hours of loading should well improve if you're not running multiple instances at the same time.

**maryjili** · Jul-16-2009, 07:14 AM

The OS is windows web server 2008, and RAM is 8GB, Cpu is Intel Core 2 Duo E6600. The total number of files in each index was around 500,000 and total index file size was 2.94GB when the program crashed. Each index used about 1.4GB of memory. I tried running single index, and the crashed ones continued to crash and the working ones remained working. And reinstalling the program did not make a difference. I also tried version 1017, it just quickly opened and closed the program without doing anything and continued to the next round.

**Ray** · Jul-16-2009, 07:25 AM

You can't use build 1017 to incrementally add to index files created with build 1016. An error message would have been reported in the log file, which inform you about this.

To try to reproduce this with the latest build, you would need to do a full re-index using the new build, and then incrementally add to the resultant files.

It is possible that the new build might have fixed this with the bug fix for the pipe characters, so it is worth checking.

If you can reproduce this with the latest build, enable the debug logging as mentioned above and send this to us.

Announcement

application crashed when incremental indexing started

application crashed when incremental indexing started

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment