No files found to spider (wordpress) - have checked for meta/robots

[bjornarfjelldal]

Hi!

I have moved / rebuilt my site from localhost to a hosted server (https://sintefshop.no), but are unable to spider it for some reason. I have searched all known (to me) possibilities, from https certificates to meta=robots, unable to find a solution as to why.

Currently I am using a "under construction" plugin, but I have tried to spider it without it being active (open to all) and with it on. Regardless of this, it should at least index something, right?

Also - how do you use the login feature for wordpress for it to spider even further inside the page (if I want to use that feature)?

I have tried to set it up to wp-login.php and given it a user/password without luck. However, the first barrier is to actually get it to spider.

Help..?

[David]

Did you check the log file in Zoom? Usually the cause should be fairly obvious.

See also this FAQ pages
Q. Why are some of my pages being skipped by the indexer?
Q. Why are links in my Javascript menus being skipped?
Q. I am indexing with spider mode but it is not finding all the pages on my web site

And this one for passwords
Q. How do I index protected parts of my website requiring user authentication?

If you are using the paid for version of the software, send us the config file and the log and we'll take a look.

[bjornarfjelldal]

Hi! I checked all of those FAQ's and I did not find anything (but of course, it might be me). I have the paid version - and the files are attached here.

I sent you a PM with login info also for you to test

[Ray]

Your web pages are returning HTTP 406 (Not Acceptable) when requested by Zoom Indexer. It is returning OK for browsers.

This appears to be common behaviour with ModSecurity, a firewall module for Apache installed by your web host on your web server. There would be a number of rules setup for this firewall which make it determine what clients it allows requests from, and which to reject.

There is something in the firewall rules that is rejecting Zoom. Perhaps it is checking the User-Agent and only allowing clients that identify themselves as known browsers like Chrome, IE, etc.

[bjornarfjelldal]

Thank you, I had that as a plausible reason I couldn't confirm - I will have a talk with our provider