admin panel

[nir]

Hi,
I found your site today and your search software look very good.

I want to ask why you choose that the admin panel will not work in the site (admin in the site), so all the function that need to make will not need the work with FTP and the need to work with computer to make the index, and instead using the corn jobs to make this action.

I ask this because I usually I work with system that are 100% online.

[David]

A few reasons
1) We support PHP, ASP, CGI, Javascript (and soon native .NET). So we would need to write the same admin interface 5 times if we did it via scripting. Which would mean almost 5 times the development work.

2) Most web hosts have limits on what resources a script can use. You don't see many scripts using 2GB or RAM and running for 4 hours while doing a large indexing job. Most hosting companies would kill any script that is resource intensive before the script was finished. We aim to support indexing 100,000's of documents. So the indexing process can take a while to scan this many files.

3) Scripting languages are VERY inefficient in terms of CPU usage and memory usage. If you index 1,000,000 files with out current software (written in C++), you might need 3GB of RAM. The same code in PHP or ASP would use 6 to 10GB of RAM.

4) Scripting languages are missing some very important functions. For example having multiple threads in ASP or PHP is pretty much impossible. As is doing cache control over HTTP.

5) We can have a much richer user interface with a desktop application.

There are also a bunch of other reasons relating to plug-ins and binary file conversions.

In short if we had written the indexer in a scripting language it would only work for very small sites and only on some hosts.

But you don't need to use FTP if you install the indexer software on your server (assuming you have a Windows based server).

[nir]

Thanks for your answer
I work with cgi and perl so I think I will need to use FTP,

Just two small things- Now I have something like 50,000 pages.
1. How long it takes to index and upload the entire site, And what is the recommendation time to make a new index for all the site (I mean do I need to reindex one time a day a week?).
2. The site is update dally with between 50 to 400 new pages plus some pages that update. How long it takes to update the search with just the new content.

[Ray]

I work with cgi and perl so I think I will need to use FTP,

I need to point out that this isn't true, in case other inexperienced users read this and believe it.

Perhaps you are implying that you have a Linux server because you use CGI and Perl - but that's also not true, because you could just as easily use CGI and Perl on a Windows server.

The only reasons one would not be able to run the Indexer on their web server are:
(a) If the web server is not running Windows
(b) You do not have the necessary permissions from the hosting company to run executables or scheduled applications on the server.

Just two small things- Now I have something like 50,000 pages.
1. How long it takes to index and upload the entire site,

Index time depends on not only the number of files, but also: the size of the files, the type of files (HTML pages, PDF files, etc), and in the case of Spider Mode, your Internet connection (or your network connection if you're indexing within a local network).

This rather old thread should give you a vague idea of the indexing time:
http://www.wrensoft.com/forum/showthread.php?t=514

It is using V4, which is about 3 years ago, and V5 is faster in most, if not all areas.

And what is the recommendation time to make a new index for all the site (I mean do I need to reindex one time a day a week?).

You should re-index as often as you need to. The search function will only search content that has been indexed, so if you add/update content, and don't re-index, then it won't be included in the search until you do.

One of the benefits of Zoom is that you have full control over this. With something like Google Site Search, you would have no control or ability to force a re-index. It may index on a weekly or monthly basis, depending on Google's fancy. It might even be several months after you've made some big changes to the site, and Google may not notice it.

Now, the nature of some websites don't require frequent re-indexing. Like the wrensoft.com website, where the majority of the content is static, until we make an update (in which case we re-index). The forum changes daily, but the daily posts are not so important until days later when we've weeded out the spam, etc.

A different scenario is our company Intranet, where we index e-mails that come in as text files. We need this to be completely up-to-date, to the very latest page, and we have PHP scripts which call Zoom, using Incremental Indexing to add pages dynamically.

(Incremental Indexing allows you to add pages to an existing index without having to re-index the entire website. See the Users Guide for details.)

You might also want to look at this FAQ:
Q. How do I schedule Zoom to automatically index on a regular basis?