2

I just installed Sitemap Creator and run a test. I have a newly developed site using PhalconPHP framework. The issue that I need assistance with is how to prevent bots from creating links to pages that do not exist? Noting this site has never been crawled by anyone but me.

i.e. The log file reports a link to /buy-homes/1896.00 Sq Ft

At no time anywhere in the script is there a reference to such a page. The only time this is mentioned is within a div or a text reference for a google map information window pop-up as text.

Can someone help me understand how crawler bots create links from that which does not exist? How can I prevent this from happening? I would rather not have 1000+ 404 pages listed with Google.

3
  • "Bots" can request any non-existent page they like - and they regularly do this checking for vulnerabilities (which naturally results in many 404s on some sites) - there is no way to prevent this behaviour. However, if your "Sitemap Creator" is able to find these "non-existent" links then that would seem to imply that there is something wrong with the way your site is generating links when the Sitemap Creator crawls your site (as John suggests). Or, there is "something wrong" with the Sitemap Creator.
    – MrWhite
    Nov 1, 2015 at 13:43
  • What confuses me about this is that there are valid links on the page(s) in question. None of them suggest anything do with "1896.00 Sq Ft". However, reviewing the log file of the crawler app it has a link for /buy-homes/1896.00 Sq Ft. I can only presume it as you suggested it is testing for vulnerabilities. Thank you and John for your thoughts. Nov 2, 2015 at 1:43
  • "it is testing for vulnerabilities" - the "Sitemap Generator" really should not be testing for vulnerabilities. It is a "Sitemap Generator" after all! Nasty bots on the internet looking to hack your site are the ones looking for vulnerabilities (unless you have a tool specifically to test this yourself). But a URL like /buy-homes/1896.00 Sq Ft really does not look like a vulnerability check (it simply relates to your content it seems). Have you tried other sitemap generators / crawlers? Do they find similar non-existent URLs?
    – MrWhite
    Nov 2, 2015 at 1:59

3 Answers 3

1

Fix your website. Bots don't create links. They follow them. If they can find links to pages that don't exist then your website has links to pages that don't exist*. Somewhere, or in multiple places, you have code that is creating a hyperlink that is not valid. You should check your logs to see what the referring page is for those invalid URL requests as that will point you in the right direction.

*Links from third party sites can link to non-existent pages on your site, too, bit that is out of the scope of this question.

1
  • Ok. Forgive my confusion. If your page consists of <a href="this.html>This</a> or <a href="that.php">That</a> how do I find in my log files from a crawler a link to /some-thing-else? Nov 1, 2015 at 12:42
0

Googlebot can create non-existing URLs and try to crawl them too, so you should strongly avoid that non-existing URLs send on requesting any server answer beside of 404.

Define your 404 document and force 404 server answer for any request of non-existing URLs. I.e. with following htaccess rules:

ErrorDocument 404 /404.php
AcceptPathInfo Off

<IfModule mod_rewrite.c>
RewriteEngine on 
RewriteBase / 
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f 
RewriteRule ^(.*)$ /404.php [NC,L]
</IfModule>
0

This usually comes from compromised sites. As MrWhite pointed out this is many times done to detect vulnerabilities. However, it is also a common tactic of "negative SEO". So I generally classify the people carrying this out as either:

  • Hackers, seeking to infect/compromise systems and networks
  • Black-hat SEOs, seeking to hurt their competitors' search engine rankings

I'm not a security expert, so I won't speak on the "hows" in that regard, but for the most part, the difference is of the vector. On the black-hat SEO side, it is typically carried out in a simple, unsophisticated manner:

  1. Create a site and upload scraped content that violates Google's guidelines
  2. Register it via Google search console, request indexing
  3. 301 redirect URLs to the target host while preserving the request URI

Step 3 is done from either the server or client via JavaScript:

Apache

RewriteEngine On
RewriteCond %{HTTP_HOST} ^myhost\.com$ [NC]
RewriteRule ^(.*)$ https://otherhost.com/$1 [R=301,L]

Nginx

server {
    listen 80;
    server_name myhost.com;
    return 301 https://otherhost.com$request_uri;
}

JavaScript

if (window.location.host === "myhost.com") {
    window.location.href = "https://otherhost.com" + window.location.pathname + window.location.search;
}

Unfortunately, this can and does affect websites; mostly small, "up and coming", and independent ones.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.