Base URLs disappearing

[RLF]

** YOU CAN DISREGARD THIS. I discovered that I was saving results for 2 different configs into the same folder and it didn't seem to like it. Once I saved search results into 2 separate folders the problem disappeared.**

I am noticing a problem with start urls for spidering where the base url will disappear after adding additional URLs.

For example - we will run a crawl with say 5 URLs. If we then add additional URL's and do an incremental update it will work fine.

If we then exit the program (configuration was saved) or simply run a complete new index it will not index the recently added urls. When we click on the edit for those URLs the Spider URL is there but the base URL is blank. This then prevents these start urls from being indexed.

This often also happens if we import or add URLs via the "more" button.

Appears to be a bug - can you please confirm/deny/fix?

[Ray]

Yes, when you run another configuration which writes to the same output directory as your previous configuration, you will end up overwriting the previous set of index files. In which case, doing incremental index on the previous config will fail (as the files have been overwritten by a different config!).

It is generally a good idea to have each configuration output to a different folder, or make it a habit to move your files after indexing. This is especially important for incremental indexing where you will need the previous set of index files to proceed.