4

We have a bizarre scenario I'd like opinions on. A site at example.com is being built on pantheon, using their example.pantheon.io staging / dev URL*.

Pantheon's development platform injects a robots.txt file to avoid indexing the dev site in Google: 

User-agent: * 
Disallow: /

Experienced SEOs know this is not enough to keep something out of Google. At some point during development, a writer accidentally linked to a page on the dev site from the production site causing Google to index the example.pantheon.io domain.

Result: example.pantheon.io is now stuck in the index and is displacing the production site to Google #23 even for its own branded query. SEO guy is SEO sad.

We are verified in GSC** on both dev and production.

Normal advice would be:

Add 'noindex' directives to page, fetch and wait Add password to page (403,) fetch and wait. Temporarily redirect page to production (301,) fetch and wait.

Of course none of these work because unless gooblebot can see these 403/301/404/etc. responses, the page will remain in the index. With Pantheon's "injected" robots.txt, we're SOL.

Do you have any ideas of how we might force this out of the index?

*It's worth pointing out to non-Pantheon folks that there is no way to change "example" in the staging URL to something else. We have no control over the robots.txt file, and cannot remove it.

**If your idea is URL removal tool: URL removal offers us a short-term hide on the example.pantheon.io site, however this would only mask our efforts temporarily, and I have recommended against this for now. The removal tool will not work on 401.

Improve this question
4
  • 3
    Pages disallowed with robots.txt don't usually outrank a branded domain. The only thing I can think of is that your domain is fairly new (within the last few months). Is that true?
    – Stephen Ostermiller ♦
    Commented Aug 23, 2019 at 20:51
  • Yes, it was launched in May and immediately acquired a substantial number of high authority, diverse links in the first 2 weeks and quite a few more since then. The domain had a bit of a history (adult forum) many years ago (like 2004) and it is not classified as adult on any of the major classifiers. No manual actions are visible in GSC. If it were someone asking me, I'd say it has a pretty clean bill of health.
    – rsc
    Commented Aug 26, 2019 at 19:07
  • 1
    Possible to redirect "everything" on the dev site to a known 404 instead of password protected or 301 response via .htaccess or similar directives? Essentially make the whole dev site "gone" then invite a rescan or two.
    – gnicko
    Commented Aug 29, 2019 at 15:51
  • Pantheon does not use htaccess/apache . The concept of your idea is solid but we're stuck in a corner technologically on this right now. We are also investigating a browser level redirect but we can't be sure that Google will interpret this in a similar manner to a proper server-side 301. I'll ask the developer again about the equivalent on ngix, but I don't think Pantheon makes that configuration setup available on staging sites.
    – rsc
    Commented Aug 31, 2019 at 10:40

1 Answer 1

Reset to default
2

I have 2 options for you

1st one : Did you try to temporary setup the dns subdomain name to an other webserver, apply the configuration you need (meta no index in the header or x robots tag) Setup a sitemap.xml in the search console with all the url you'd like to remove

Wait for removal in google

Then go back with the dns to you old setup with the password check this time

2nd one Are you able to add custom redirections in your solution ? So you might be able to specify an other robots.txt (without changing the name)

RewriteRule /robots.txt /custom-robots.txt [L]

If you have access to the .htaccess file (I'm sorry i don't know pantheon) you may be able to add no index directive with the x robots tag directive in the http header.

Hope it helps

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.