noindex tag..different for online/offline

[ohyeahitsgood]

wondering why offline spidering ignores tha meta noindex flag while online spidering honors it?

actually i like that it ignores and so i can have one version of site search with pages i dont want search engines to both with. Anyway to manually specify to ignore that meta for online spidering?

[Ray]

The meta robots noindex tag implies it is for "robots" aka spiders.

Conceptually, there is no "offline spidering". There is an "offline mode" which scans all the files in given folder(s) and obeys the skip options you specify, and "spider mode" which retrieves pages via HTTP and crawls/follows links based on the page content (and meta tags and robots instructions).

To ignore the meta robots tag in Spider Mode, you should uncheck the option under "Configure"->"Spider Options"->"Enable 'robots.txt' support"