We have a bizarre scenario I'd like opinions on. A site at example.com is being built on pantheon, using their example.pantheon.io staging / dev URL*.

Pantheon's development platform injects a robots.txt file to avoid indexing the dev site in Google:

User-agent: * 
Disallow: /

Experienced SEOs know this is not enough to keep something out of Google. At some point during development, a writer accidentally linked to a page on the dev site from the production site causing Google to index the example.pantheon.io domain.

Result: example.pantheon.io is now stuck in the index and is displacing the production site to Google #23 even for its own branded query. SEO guy is SEO sad.

We are verified in GSC** on both dev and production.

Normal advice would be:

Add 'noindex' directives to page, fetch and wait Add password to page (403,) fetch and wait. Temporarily redirect page to production (301,) fetch and wait.

Of course none of these work because unless gooblebot can see these 403/301/404/etc. responses, the page will remain in the index. With Pantheon's "injected" robots.txt, we're SOL.

Do you have any ideas of how we might force this out of the index?

*It's worth pointing out to non-Pantheon folks that there is no way to change "example" in the staging URL to something else. We have no control over the robots.txt file, and cannot remove it.

**If your idea is URL removal tool: URL removal offers us a short-term hide on the example.pantheon.io site, however this would only mask our efforts temporarily, and I have recommended against this for now. The removal tool will not work on 401.

  • 2
    Pages disallowed with robots.txt don't usually outrank a branded domain. The only thing I can think of is that your domain is fairly new (within the last few months). Is that true? – Stephen Ostermiller Aug 23 '19 at 20:51
  • Yes, it was launched in May and immediately acquired a substantial number of high authority, diverse links in the first 2 weeks and quite a few more since then. The domain had a bit of a history (adult forum) many years ago (like 2004) and it is not classified as adult on any of the major classifiers. No manual actions are visible in GSC. If it were someone asking me, I'd say it has a pretty clean bill of health. – user3516228 Aug 26 '19 at 19:07
  • 1
    Possible to redirect "everything" on the dev site to a known 404 instead of password protected or 301 response via .htaccess or similar directives? Essentially make the whole dev site "gone" then invite a rescan or two. – Greg Nickoloff Aug 29 '19 at 15:51
  • Pantheon does not use htaccess/apache . The concept of your idea is solid but we're stuck in a corner technologically on this right now. We are also investigating a browser level redirect but we can't be sure that Google will interpret this in a similar manner to a proper server-side 301. I'll ask the developer again about the equivalent on ngix, but I don't think Pantheon makes that configuration setup available on staging sites. – user3516228 Aug 31 '19 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.