Zoom Search fatal error memory limit

[DAR]

I have just indexed 23,500 php pages using Zoom v7.1 Professional. The total number of words came to just over 10 million and the unique words about 157k. When I installed a test search page on my site I kept getting fatal error messages when I tried searching, eg "Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 6 bytes) in ****/library/search/search.php on line 2897". Wrensoft support suggested that the problem might lie with the server's memory limit and when I contacted my hosting company I was told "The memory limit on the server is set to 256M, so that should be high enough for the app to work with." However, the host restarted the apache and mysql processes and the search engine did work then to some degree, although most times when I entered multiple word searches rather than single ones the error returned. Has anyone an idea of where the problem may lie? I have read as much as I can find on the issue.

[Ray]

The error message indicates that the script is requiring more than 256M of memory. I have an index here with 50,000 pages but only 71K unique words, and it required 180M of RAM for an intensive search.

Since your index has 157K unique words, it is possible that your data set could actually require 256M+ memory -- depending on the search query, as well as the size of the data (e.g. you might have particularly long key words).

However, first you should confirm that you don't have a corrupt index. This can occur if you accidentally mix files from different index sessions. If that was to happen, it can certainly use more memory than normal.

See this FAQ for a list of possible causes of a corrupt index:

Make sure you are also uploading the files to the same folder that you are checking from the browser. We quite often see users that have uploaded files to one folder, but are looking at the search files in another folder on the same server (i.e. they have accidentally uploaded the index files to 2 different locations).

Often you can quickly confirm if the files are from the same index session by looking at the date and time of the files created. And if any of the index files stand out as being from a different period of time. Especially if you are manually uploading the files with a different FTP program.

If you have confirmed all of the above and the problem persists, then it is quite possible the index actually requires more memory.

You can either ask your web host to allow you to use more memory (is this restricted to your hosting account level?), reduce your index size (by way of Limits in the Indexer), or move to the CGI (which typically requires more server privileges, so you should check with your web host first).

[DAR]

Thank you very much, Ray. Now that's what I call an answer! The index appears to be fine as the files are all in the one unique folder and were created on the same date, so it looks as though it is most likely a memory issue. I don't know what the position of that is with the hosting company, but can ask. I'm not familiar with cgi - I would have to read up on it - but if it requires changing the the content and extension of all the pages on the site then it could prove to be a bit of a nightmare. But now, thanks to you, the problem is identified then I can consider how best to address it.

[Ray]

Changing to CGI is only for the search engine (i.e, the search page). It would not require changing any of the existing pages on your site.

The only difference is that instead of "search.php", it would be "search.cgi" and it (along with your new index files created for CGI) would typically be hosted in a different folder, one that is specifically designed to run CGI -- usually "/cgi-bin/"

The only challenging part is that CGI is technically more complex to setup on the server side. Some web hosting companies (cheap ones) will not support it because its more demanding for them. Many web hosting admins (again, often the cheaper ones) would not even understand what it really is and confuse CGI with say, Perl. So you would have to have a good hosting company to help you if you aren't familiar with it.

More information here,
https://www.wrensoft.com/zoom/suppor...forms.html#cgi

My advice is to ask if you can increase the amount of RAM for your PHP use first. Maybe you need 360MB or maybe 512MB. If that doesnt help, ask about CGI support -- make sure they understand its a "CGI compiled binary" and not a "CGI script". Frankly, if they won't increase your RAM allowance, they probably won't allow you to use binary CGIs.

And if both of those aren't an option, then you should either consider a different hosting company (or different hosting account level), or splitting up your index into smaller sets -- make multiple search pages for different sections of your site.

[DAR]

Only just picked this up now, Ray. Thank you again - that information is most useful indeed. Currently I am using a number of smaller Zoom search engines for different parts of the site but may revisit a full site search later. I am in the process of changing host and securing my site and have a lot of tidying up to do, new content to add, etc,, but what you have explained is a bit of a revelation - in that I don't need to change the complete site to cgi.