Originally posted by perry
View Post
A spider generally works by making HTTP requests to the server, just like someone sitting at a computer with a browser. In other words, the spider indexes and scans a page as a visitor to your website would see it. This is why a PHP page can be indexed and scanned, despite whatever internal PHP calls it might be making. The spider never sees the internal files, it only sees the resultant page that is presented to the user. This is also what a spider can only locate files on a web site by following the links on the page. It can not find files on your web server which are not linked to via your web pages.
I hope that helps somewhat.
Leave a comment: