I just installed Sitemap Creator and run a test. I have a newly developed site using PhalconPHP framework. The issue that I need assistance with is how to prevent bots from creating links to pages that do not exist? Noting this site has never been crawled by anyone but me.

i.e. The log file reports a link to /buy-homes/1896.00 Sq Ft

At no time anywhere in the script is there a reference to such a page. The only time this is mentioned is within a div or a text reference for a google map information window pop-up as text.

Can someone help me understand how crawler bots create links from that which does not exist? How can I prevent this from happening? I would rather not have 1000+ 404 pages listed with Google.

  • "Bots" can request any non-existent page they like - and they regularly do this checking for vulnerabilities (which naturally results in many 404s on some sites) - there is no way to prevent this behaviour. However, if your "Sitemap Creator" is able to find these "non-existent" links then that would seem to imply that there is something wrong with the way your site is generating links when the Sitemap Creator crawls your site (as John suggests). Or, there is "something wrong" with the Sitemap Creator. – MrWhite Nov 1 '15 at 13:43
  • What confuses me about this is that there are valid links on the page(s) in question. None of them suggest anything do with "1896.00 Sq Ft". However, reviewing the log file of the crawler app it has a link for /buy-homes/1896.00 Sq Ft. I can only presume it as you suggested it is testing for vulnerabilities. Thank you and John for your thoughts. – Steven K7FAQ Nov 2 '15 at 1:43
  • "it is testing for vulnerabilities" - the "Sitemap Generator" really should not be testing for vulnerabilities. It is a "Sitemap Generator" after all! Nasty bots on the internet looking to hack your site are the ones looking for vulnerabilities (unless you have a tool specifically to test this yourself). But a URL like /buy-homes/1896.00 Sq Ft really does not look like a vulnerability check (it simply relates to your content it seems). Have you tried other sitemap generators / crawlers? Do they find similar non-existent URLs? – MrWhite Nov 2 '15 at 1:59

Fix your website. Bots don't create links. They follow them. If they can find links to pages that don't exist then your website has links to pages that don't exist*. Somewhere, or in multiple places, you have code that is creating a hyperlink that is not valid. You should check your logs to see what the referring page is for those invalid URL requests as that will point you in the right direction.

*Links from third party sites can link to non-existent pages on your site, too, bit that is out of the scope of this question.

| improve this answer | |
  • Ok. Forgive my confusion. If your page consists of <a href="this.html>This</a> or <a href="that.php">That</a> how do I find in my log files from a crawler a link to /some-thing-else? – Steven K7FAQ Nov 1 '15 at 12:42

Googlebot can create non-existing URLs and try to crawl them too, so you should strongly avoid that non-existing URLs send on requesting any server answer beside of 404.

Define your 404 document and force 404 server answer for any request of non-existing URLs. I.e. with following htaccess rules:

ErrorDocument 404 /404.php
AcceptPathInfo Off

<IfModule mod_rewrite.c>
RewriteEngine on 
RewriteBase / 
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f 
RewriteRule ^(.*)$ /404.php [NC,L]
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.