Uploading new indices

[AndrewD]

I'm planning on automating the process of generating then uploading the search indices, but am a little concerned about the ftp 'push' process, as this seems a bit unreliable through our firewalls. I'm thinking instead about having a 'pull' mechanism on our web server, running a perl mirroring script.

To make sure that the process is fully reliable, I suppose one should disable searches while the indices are being replaced. Is there any semaphore built into the search.cgi to prevent it carrying out a search - e.g. should I kill one of the zdat files until everything is in place?

[David]

The FTP upload process in Zoom uploads the files with temprorary file names, then does a rename on the files after the upload. The reduces the downtime dramatically to just a couple of second and (largely) prevents the search script using half uploaded files.

If FTP is unrealiable on your site, then maybe any other mechanism will also suffer problems, when using the same network connection?

There is no semaphore. So you could delete the existing ZDAT file(s). But this is risky in my opinion. What if you kill the working file and then find you can't get new fiiles due to network issues? The search function would be left broken.

I would instead write a Perl script called search_offline.cgi. Then switch the search_offline.cgi and the search.cgi script before the file sync. Then switch bank once done, or if there was an transfer error

The search_offline.cgi would just print a simple message saying search was offline for 5 min.

[AndrewD]

Thanks - you've given me some good hints. I need to get to the bottom of the ftp problems - the network is perfectly reliable for http, etc - I suspect it's just a matter of sorting out the firewall rules, but am not clear whether it's the in-built window firewall or our h/w firewall that's the culprit.