Server performance for larger sites

[Ade]

Hi,

Is there any suggestions you can give for improving the search performance of large sites? When I index our 12000 page site I get a zoom_wordmap.zdat size of 20.3mb and a zoom_pagetext.zdat size of 15mb (that's after I've tweaked the configuration options and my web pages to ignore the navigation and other non-essential elements).

This works fine on my local pc but is rather slow on the web server. I just wonder whether it would be feasible to have an option in Zoom to break these files down into smaller sizes but more of 'em? Just thinking it could ease the load on the server - not sure how feasible/effective this would be though.

Just one additional request - could the "Next >>" link at the bottom of the page results listing use &raquo instead of ">>", or at least provide an option for us to amend it? It's just that it's inconsistent with what I've used elsewhere on my site. Same goes for "<< Previous", except use &laquo.

Great product BTW!

many thanks,

Ade.

[David]

12,000 pages is actually only a medium site. You should get good performance all the way up to 200,000 pages with the right configuration and hardware.

Splitting up files would do nothing to help performance. It would only make things worse in fact.

You can see our benchmark figures here,
http://www.wrensoft.com/zoom/benchmarks.html

As you can see we measured search times of 1.3 seconds with 180,000 pages when using the CGI option on a decicated server.

What search times are you seeing? What script option have you selected in Zoom? What hardware is in your server? How many other web sites share the same server?

-----
David

[Ade]

Wow, only a medium sized site!!

Search times are about 11 seconds! I'm using the php option. The site is a dynamic website with pretty much the whole page data stored in a database (which I understand shouldn't be an issue since zoom doesn't access the page when doing the search, only when indexing the site - or am I wrong here? ).

I've also made good use of the tags (and the respective RESTART) in all the pages which made a big difference on the final search file sizes.


There's only about 2 or 3 other websites which are very small by comparison sharing the same server. I don't know the spec of the server but it's starting to look like a server issue as it went down tonight and even after a reboot didn't come back up until we had the local support people have a look at it.

Saying that though, when I access a specific page of the website I get a good 1-2 second response time.

With a site of around 1200 pages and 60,000 unique words I guess you'd expect decent response times?

[David]

In your first post you said you had 12,000 pages. In you 2nd post you implied you have 1,200 pages??

With PHP, search times of 2 seconds can be expected on sites with about 50,000 pages. (assuming reasonable hardware)

However the PHP option is about 5 times slower than the CGI option. So you should give some consideration about moving to the faster CGI option.

You are correct, Zoom will not access your database during a search. (but will indirectly do so during indexing).

Yes you should check the hardware spec & if there is any other load on the machine. Many hosting companies put up to 100 web sites per server (making them very slow) so you might also want to double check this.

-----
David

[Ray]

Re: Server performance for larger sites

This works fine on my local pc but is rather slow on the web server.

This might be a good indication that there is a problem on the server. How significant is the speed difference between running the search on your local PC and the server? 11 seconds is very slow for a 12,000 page search.

Just one additional request - could the "Next >>" link at the bottom of the page results listing use &raquo instead of ">>", or at least provide an option for us to amend it? It's just that it's inconsistent with what I've used elsewhere on my site. Same goes for "<< Previous", except use &laquo.

At the moment, you'll have to modify the "search.php" source code if you really wish to change this. This is unfortunate, as we try to minimize the need to touch the script. It is also impossible to change for the CGI version (since it is compiled binary), which you might now be considering using.

For a future version, we might move the ">>" (&gt;&gt text to the ZLANG language file, so that users will be able to modify/remove this as they please, along with any of the other text on the search page. As it is at the moment, you can modify the words "Next" and "Previous" via the ZLANG files.

[Ade]

thanks for the comments guys - will continue the investigations/deliberations.

BTW, it's 12,000 pages not 1,200 - just a typo on my part!

[Ade]

I'm using the cgi version and am now getting excellent performance - approx 1 second searches! What a difference!!