Spider not finding pages

[IanL]

Hello,

Stumbled on this product a few weeks ago. Amazing and congrats.

I’m a novice, have a small personal website (70+ pages) and have been trialling the free version for a few days using the php process. My ISP has the 5.2.11 version.

I have limited coding expertise so this product seems to fit my requirements.

I would like to use the Professional version on a website project I’m working on. But after many attempts I can only get the spider to locate 22 HTML pages on my personal site. What am I doing wrong? I’ve added copious META tags to pages, left them out, added a site map (not linked to the index page) with just one URL on it for testing purposes, but the spider did not find the site map page or the HTML link on it.

Do I understand correctly that the spider starts at the index page and only finds URL’s directly linked to that page. Or have I missed a process whereby I can spider all pages that do not have a direct link to the Index page.

The project I’m working on might eventually have 7,000+ HTML pages.

Can someone please enlighten me.

[Ray]

To start with, read the following FAQ:
Q. I am indexing with spider mode but it is not finding all the pages on my web site

The spider follows links. But that doesn't mean they have to be directly linked to the start page. The pages need to be interlinked, so that, for example, if you have links from Start page -> Page A -> Page B -> Page C, then all 4 pages will be indexed (by default, you can change this behaviour however). If you have a Page D that isn't linked from any of the above pages (that is, you cannot follow text links from the start page to get to this page), then it will not be found.

The Log tab will usually indicate why links are not followed and if configuration changes are necessary. See also this FAQ:
Q. Why are some of my pages being skipped by the indexer?

If you still have trouble, give us the URL to the start page you are using and we can take a look.

[IanL]

Thanks Ray for the prompt reply. I'll have a read and keep plugging away.