3

Ok, this may turn out to be a silly question but I have observed Google displaying a URL in the index with the cache of that result being completely different when it shouldn't even be there in the first place.

Description:

I built a random function for the website docur.co

The function initiates with a request to:

http://docur.co/random

The robots are blocked from this URL:

http://docur.co/robots.txt

However Google has followed this URL and produced the following search result:

https://goo.gl/BCyOml

This is the cache:

enter image description here

My question is: Can anyone tell me what exactly is going on here? As aforementioned, I may have done something wrong...

Update:

Maybe adding the re="nofollow" directly to the anchor on top of the robots directive will ensure that Google will not follow the URL?

Improve this question
7
  • Is it possible that google crawled and indexed this link before you setup your robots.txt? Have you tried removing that 1 url using the removal tool google.com/webmasters/tools/removals ?
    – Analog
    Apr 8, 2016 at 22:53
  • The site is very sensitive about Google results so the robots file was updated before i pushed my code live. 100% sure.
    – John
    Apr 8, 2016 at 23:16
  • If you use the removal tool on that URL does Google re index it just with different content (whatever random happens to load)?
    – Analog
    Apr 8, 2016 at 23:21
  • I here you point. However Google should not even be following the URL according to themselves: postimg.org/image/np474rrgx So what?
    – John
    Apr 8, 2016 at 23:30
  • *Hear I find it really astonishing that Google will check out nofollowed URL's and process them.
    – John
    Apr 8, 2016 at 23:58

1 Answer 1

Reset to default
4

You have an error in your robots.txt file.

On line 11 you have Allow: /, a robots.txt file doesn't say what files and directories you can allow, only what you can disallow. The only supported commands for the robots.txt file are "User-agent" and "Disallow".

As the Disallow: /random command is after the invalid command it is possible the Google Searchbot detected an invalid command and because it couldn't process it stopped processing the entire robots.txt file as if it didn't exist at all.

You can validate your robots.txt file using a tool such as the one located at http://tool.motoricerca.info/robots-checker.phtml

As for why the cacheed version is different to the live version the cached version it what Google see's at the time the spider went through which in the case of your cached link was 6 April 2016 at 16:05:27 GMT.

A new version of your robots.txt file which you could use is...

#The date is August 29th, 1997.
#Robots have taken over the world and documentaries cease to be created by humans.
#what will happen next?

 #Want to join the Docur team?
#E-mail jonbonsilver\\//at\\//gmail\\//dot\\//com

#Full access for the internet archive. 

User-agent: ia_archiver
Disallow: /random

#Every robot that honours the robots.txt standard:

User-agent: *

#Request file from Docur once every second:

Crawl-delay: 1

#Disallowed urls:

#Lets not send bots on a random documentary mission:

Disallow: /random

Disallow: /new-documentaries
#Above is a temp line due to indexing problems. 
Disallow: /?page
Disallow: /live-search
Disallow: /vote
Disallow: /favourite
Disallow: /watch-later
Disallow: /save-list
Disallow: /comment
Disallow: /commentlike
Disallow: /commentdislike
Disallow: /add-review
Disallow: /submit-review
Disallow: /add-to/*
Disallow: /post-list
Disallow: /edit-list
Disallow: /documentary-search
Disallow: /new-list-item
Disallow: /settings
Disallow: /notificationread
Disallow: /documentary/*/l
Disallow: */newest
Disallow: */oldest
Disallow: */highest
Disallow: */lowest
Improve this answer
6
  • Nice catch! Oddly, we get a fair number of robots.txt errors here. I have seen some references to sites that are technically correct, but misleading or confusing. Some start with disallowing everything then allowing specific things which is easy to screw up. I have also seen some bad advice too.
    – closetnoc
    Apr 9, 2016 at 3:40
  • PlanetScale @ ia_archiver is alexa bot not Googlebot, and he already using User-agent: * which indicate all polite bots like Googlebot, MSNbot, baidubot etc. and for that useragent he is already using Disallow. @ John do one thing, Go to this URL, select your properties(Hope you already added your site in Google Search Console), then put random and click on fetch and render tools, and tell us whether it is displayed blocked or not.
    – Goyllo
    Apr 9, 2016 at 7:26
  • @Goyllo - Please take a careful look at what I said. I didn't say that ia_archiver is the Googlebot, as I am well aware that it isn't, all I said was that having Allow: / doesn't meet the standard, breaks validation, and could be what is causing the issue with the Googlebot, if the Googlebot is programmed to follow the standard.
    – Chris Rutherfurd
    Apr 9, 2016 at 7:38
  • @Goyllo postimg.org/image/np474rrgx i haven't changed the file according to PlanetScales' recommendations yet...
    – John
    Apr 9, 2016 at 9:52
  • I knew it, you have already blocked that random URL, now let me know, you want to hide that URL from search result?, because that URL already redirect to another one. If that content already exit in your site then do 301 redirect to proper documentary content otherwise just use 410 Gone error.
    – Goyllo
    Apr 10, 2016 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.