2

We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting our search results.

I'm trying to figure out how the pages got there and how to remove them. The pages have always had <meta name="robots" content="noindex, nofollow"> in the head.

I am kinda new at this but was under the impression this should prevent Google from showing my site in results. I am pretty sure the results appeared in google after I accidently copy pasted some codes from the staging site to the live site that contained links to pages on the staging site. If anyone can point me in the right direction to figure out what happened and prevent it from happening again would be much appreciated.

robots.txt looks like this:

User-agent: *
Disallow: /calendar-2/action~posterboard/
Disallow: /calendar-2/action~agenda/
Disallow: /calendar-2/action~oneday/
Disallow: /calendar-2/action~month/
Disallow: /calendar-2/action~week/
Disallow: /calendar-2/action~stream/
#Begin Attracta SEO Tools Sitemap. Do not remove
sitemap: http://cdn.attracta.com/sitemap/4035112.xml.gz
#End Attracta SEO Tools Sitemap. Do not remove

I also tried adding an X-Robots-Tag header and submited the site to be re-crawled. Did that a few days ago and I still see no changes. Here are the HTTP headers according to "Fetch as Google":

HTTP/1.1 200 OK
X-Robots-Tag: noindex,nofollow
Vary: Accept-Encoding
Transfer-Encoding: chunked
Date: Sun, 24 May 2015 16:26:49 GMT
Server: LiteSpeed
Connection: close
X-Pingback: http://trailheadpaddleshack.ca/staging1/xmlrpc.php
Content-Type: text/html; charset=UTF-8
Link: <http://trailheadpaddleshack.ca/staging1/?p=170>; rel=shortlink

I am now faced with a bunch of results I need to remove from Google asap as they contain out of date information and are affecting our search results. Webmaster Tools has something for removal of a single URL but I am looking to remove the entire /staging1/ subfolder. Any tips?

Improve this question
6
  • Do these indexed pages have a description in the SERPs? If the robots meta tag has been on these pages all the time AND the pages are not blocked by robots.txt then these pages should not get indexed, so something would seem to be amiss? Does the HTML validate?
    – MrWhite
    Commented May 24, 2015 at 18:55
  • Following what @w3d said, when a link is captured and a page is spidered, it will appear in the SERPs however when the robots.txt is read the page is not actually indexed, however, the fact that the page exists with a URL/URI remains within the index. Google thinks this is a good idea, however, it is not. It is actually immoral by not honoring robots.txt. In time, this URL/URI will be removed as long as any link does not exist, otherwise the process repeats itself. Remove all links. This is one area that Google has absolutely wrong. But it takes a while before Google sees the light sometimes.
    – closetnoc
    Commented May 24, 2015 at 22:03
  • Also consider the Remove URL option in Google Webmaster Tools- not sure it fully applies here, but give it a look anyway.
    – closetnoc
    Commented May 24, 2015 at 22:28
  • I'm just curious as to what happens with indexing when you remove the space inside "noindex, nofollow". I bet you found another one of google's bugs. Lately, they are starting to offer a poorer experience to users, especially with their maps and even groups sometimes.
    – Mike -- No longer here
    Commented May 24, 2015 at 22:58
  • 1
    After the fact fix, lock down access to your staging server by IP address blocks, the whole world doesn't need access and it's easier to prevent stealth indexing that way as well.
    – Fiasco Labs
    Commented May 24, 2015 at 23:03

2 Answers 2

Reset to default
2

As I mentioned in my comment above, if the robots meta tag has been on these pages all the time AND the pages are not blocked by robots.txt (which prevents crawling, but not indexing) then these pages should not get indexed.

Digging a little deeper, these pages do appear to be fully "indexed", with a complete description in the SERPs. So they have not been blocked by robots.txt and this is consistent with the robots.txt file in the question, which does not block /staging1. And the live pages do indeed have a noindex robots meta tag.

However, checking the Google cache of these pages in the SERPs reveals the problem: There is no robots meta tag! So, you would seem to have experienced a temporary "glitch" about a month ago (the Google cache shows dates of 15 and 21 April) that resulted in the robots meta not being output in the page as it should have been. Consequently Google indexed the pages!

I also tried adding an X-Robots-Tag header and submited the site to be re-crawled. Did that a few days ago and I still see no changes.

That's the right idea, but it seems you'll need to wait more than a few days. As I mentioned above, the cached pages in the SERPs are over a month old - so that suggests that these pages have not been recrawled yet, or Google has simply not updated it's index.

Improve this answer
3
  • Good call on checking the cached pages to see what they looked like when they got indexed. I'm pretty sure I know what happened a month ago for this to happen... What would remove these results the fastest? Wait until google re-indexes everything and sees the robots meta? Password protect the directory using .htaccess (this would return a 401 do these get removed from the index?). Using IP blocks? There is a remove url tool in webmaster tools but i don't see a remove directory option, is there such a thing?
    – Cleo Poulin
    Commented May 25, 2015 at 16:35
  • I've not used the "Removal Tool" myself, but the Help Docs do suggest that you should be able to remove a directory. And for "permanent removal" these pages should return a 404 or 410 (according to Google). Google does however state, "removal is not instant". However, is the actual problem that these pages appear in the SERPs, or that the content is accessible? You could make the content instantly inaccessible by password protecting or blocking by IP, etc.
    – MrWhite
    Commented May 25, 2015 at 19:48
  • You could even block access to users that click the results in the SERPs by checking the HTTP Referer. If the Referer is google.com then 404.
    – MrWhite
    Commented May 25, 2015 at 19:50
1

I've seen this before and I think it's caused by a vicious circle!

If you are blocking pages from being crawled by Google in robots.txt then Google cannot access the page to see the NOINDEX tag, so the pages will not be removed from the index if they had already got indexed

Blocking pages in robots.txt will stop Google crawling them, but it won't stop them getting indexed. If Google finds them linked elsewhere, they can still get indexed.

But where did Google find the links, well that's another topic entirely!

But if you are using NOINDEX tag and blocking them in robots.txt, the pages can still appear in the SERPS, as closetnoc mentioned usually with message saying

'A description for this result is not available because of this site's robots.txt – learn more.'

The sure fire way to guarantee Google doesn't incliude your URLs in the SERPs is password protecting the directory they are in

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.