Indexing Dynamic Pages with PHP Script

[pathfinder]

Hello All, I am writing ASP.NET apps for use on Windows shared hosting servers. Please don't shake your head, I believe in the little guy. So I failed, predictably, to get the CGI version working wrapped in an .aspx page because of trust level inherent and unchangeable in these servers.

So I am trying out the PHP version which works ok except that it does not index my dynamic pages without me adding each page manually to the Advanced Spider URL Options. I have had these sites crawled successfully to make site maps but Wrensoft's Zoom will not index these.

I have searched through the users guide and looked at every entry of the word "dynamic" and read through the guide but can not find any answers. I also searched this forum and come up empty-handed.

Is it something stupid I am missing; I have overlooked something obvious before.

Any help appreciated as I hope to finishing testing and get each of my clients to kick out the cash for the Standard Edition.

[David]

Dynamic pages are a problem in offline mode (as there is no web server), but are treated like any other page in Spider mode. So no special action is required to index them.

There should be no need to manually add the list of pages, except if you don't have links to the pages for the spider to find.

See also this FAQ
Q. Why are some of my pages being skipped by the indexer?

Looking in the log might also give you a hint about why the pages are being skipped.

If you can't get the bottom of this can you post a link to your start URL and the URL of a page that is being skipped.

[pathfinder]

Thanks, still digging

Thanks wrensoft, I am still trying to configure to solve this problem. If not I may look to parsing a sitemap.xml and import into the Advanced spider URL options/Import.

The start URL is http://maximguns.com/

And a page being missed is http://maximguns.com/HK-G391CETME-cleaning-kit-p189/

[pathfinder]

Think I got it.

I think I got it when I chose to [index and follow all] when I opened the edit window for spider start point.

[David]

There are two related settings.

Index page and follow internal links (default)
Index page and follow internal and external links (this one you say you selected)

But in this case the missing page is not external to you site, so it the setting change should make no difference.

But I tried indexing your site from here.
On attempt 1 your server returned a page "Forbidden" error code.
On attempt 1 your server redirected the request to this page, and then crashed.

Could not download file: http://maximguns.com/Oops.aspx?aspxerrorpath=/default.aspx (Internal server error)

In fact this happened in a browser as well, outside of Zoom.

But if you are no longer having the problem, I guess it is all good.