Bug - Stopping on one domain and not proceeding to next

[nibb]

Why is it that when a url isnt working. I have put a list or domains or url, each one is one website, so each one would be a entry point. When a particular domain isnt working for example because of webhost problems or because it doesnt work anymore, Zoom stops and indexes all the data. I think it should jump to the next entry point not stop there.

[Ray]

Zoom should already be reporting any errors on any start points that fails, and move on to the next one, as you suggest.

However, it can depend on what exactly the error is. For example, if a server is failing to respond, there is a timeout of 60 seconds which Zoom will wait for, in case the server is just slow. Only after having waited 60 seconds for a response, will Zoom report an error and move on.

Most other errors (eg. invalid domain name, 404/page not found errors), will be detected immediately and it'll move on to the next start point without the minute long delay.

Can you tell us exactly what you mean by "Zoom stops and indexes all the data"? What is the error message when this occurs?

[nibb]

Well if you have a list or domains or urls. Every domain or url is another website.

www.a1.com
www.b2.com
www.c3.com
www.d4.com

So if Zoom gets to c3.com and it doesnt work, it stops and index all data so far, instead of going just to d4.com

[David]

As Ray pointed out, Zoom does not behave like this. See Ray's post above for details. If you think it is doing this, then you should make a careful examination of the log for errors. Then E-mail us the log and your configuration file if you can't see what is going on.

[nibb]

Ups I have downloaded V4 by accident. I suppose this was corrected on V5 five right? I will download the new one.

No, it doesnt show anything on the error, it happens on a certain url, normally some php session link its following, it takes some time, then finally it says, Could not download file. Check your internet connection.

Internet works, i test the url then in Explorer and it works. Its not a certain url, its random. Thats whats bugging me. I have not tested this on V5, but that would definitly be a big issue, since it then stops and index all data instead of jumping to the next point. I see V5 has Incremental but still it should not do this, has anybody tested this on the new version?

[nibb]

I just bought V5 and the same problem, when it gets to a url that isnt working it stops and indexs all data instead of jumping to the next url. I need to index 1000 urls, its impossible this way, since after some hours just 1 single website must be down or give a timeout and the whole index fails.

[David]

No, it doesnt show anything on the error..

You say there is no error, but in your other post you say there is an error message about loosing your internet connection.

Loosing your internet connection will stop indexing becuase it is pointless to continue indexing without a network connection.

Maybe your connection is intermittent or maybe there is an issue at your ISP, maybe even the remote server has failed or maybe there is an issue with 3rd party software (like a firewall).

Are you using a wireless Internet connection? They are notorious for loosing connection for short periods.

[nibb]

Yes I have a Wireless connection. And yes it does lose a ping from time to time, 1 packet from 100 maybe. But i never saw that would affect internet or any program, i think its more the hosts where the websites are hosted they are very bad hosts, sometime they work sometimes they dont, the pages where it gives that error are because they timed out not because i dont have internet.

[David]

I have examined the log file and configuration file that you sent.

I have also used your configuration file to index the same sites in Paraguay you are having problems with. I indexed 1000's of pages for dozens of sites without getting a single error (from Australia).

You claimed that the sites you were indexing in are 'bad hosts' and unreliable. With you log file showing a failure after as little as 80 pages. But we didn't see any evidence of this.

I know it is easy to blame our software, or blame the remote sites, but the error in your log file shows that you lost your network connection. And we believe this error message is correct. Plus you have admitted your wireless internet connection is not 100% reliable.

So the solution would seem simple. Try connecting your PC to a wired Internet connection though a router or switch with CAT-5 cable and stop using your wireless connection, which is obviously not stable enough for what you are trying to do.

[nibb]

I replied by email. Ping to wrensoft.com 250 packets 0 lost.

So if i just lose 1 packet the whole indexing fails? I do think that should be fixed, i have 1000 sites to index, only 1 packet fail in 8 hours of indexing and all fails? I have also tested that it fails when internet is 100% without interruptions. I cannot assure that websites hosted on a server are 100 ping available at all times. Thats impossible. zoom must have the option to skip the site or re index later not to fail on that certain point.

[David]

It is not the loss of 1 packet or even 10 packets that is the problem. I never said this. It is the loss of your entire internet connection that is the problem. Windows detects the connection has 100% disappeared. I don't know if this is because of poor wireless coverage or bugs in your device driver or something else, but this is what is happening.

[nibb]

Ok, so as i see on the emails when zoom hits a non working url it gives the error:
Could not download file. Invalid URL or domain name. And it jumps to the next domain.

That makes me feel 10 times better now since at least i know its a local problem not a zoom problem. Now its only to find out why in my computer instead of tha t message it gives me Internet not found. So Zoom does skip not working urls and go to the next site. Great.

Does someone tested it with IE 7. Maybe thats causing my problem, anyway i will try to install and test it on another computer and see if i finally get the correct error.

[David]

Just an update with what appears to be a resolution to the issue. Nibb, resolved the problem by clearing the ARP cache (Address Resolution Protocol) on his local machine. The assumption is that the cache was corrupt and causing disconnects.