We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting our search results.
I'm trying to figure out how the pages got there and how to remove them. The pages have always had
<meta name="robots" content="noindex, nofollow"> in the head.
I am kinda new at this but was under the impression this should prevent Google from showing my site in results. I am pretty sure the results appeared in google after I accidently copy pasted some codes from the staging site to the live site that contained links to pages on the staging site. If anyone can point me in the right direction to figure out what happened and prevent it from happening again would be much appreciated.
robots.txt looks like this:
User-agent: * Disallow: /calendar-2/action~posterboard/ Disallow: /calendar-2/action~agenda/ Disallow: /calendar-2/action~oneday/ Disallow: /calendar-2/action~month/ Disallow: /calendar-2/action~week/ Disallow: /calendar-2/action~stream/ #Begin Attracta SEO Tools Sitemap. Do not remove sitemap: http://cdn.attracta.com/sitemap/4035112.xml.gz #End Attracta SEO Tools Sitemap. Do not remove
I also tried adding an X-Robots-Tag header and submited the site to be re-crawled. Did that a few days ago and I still see no changes. Here are the HTTP headers according to "Fetch as Google":
HTTP/1.1 200 OK X-Robots-Tag: noindex,nofollow Vary: Accept-Encoding Transfer-Encoding: chunked Date: Sun, 24 May 2015 16:26:49 GMT Server: LiteSpeed Connection: close X-Pingback: http://trailheadpaddleshack.ca/staging1/xmlrpc.php Content-Type: text/html; charset=UTF-8 Link: <http://trailheadpaddleshack.ca/staging1/?p=170>; rel=shortlink
I am now faced with a bunch of results I need to remove from Google asap as they contain out of date information and are affecting our search results. Webmaster Tools has something for removal of a single URL but I am looking to remove the entire /staging1/ subfolder. Any tips?