We have a bizarre scenario I'd like opinions on. A site at
example.com is being built on pantheon, using their
example.pantheon.io staging / dev URL*.
Pantheon's development platform injects a robots.txt file to avoid indexing the dev site in Google:
User-agent: * Disallow: /
Experienced SEOs know this is not enough to keep something out of Google. At some point during development, a writer accidentally linked to a page on the dev site from the production site causing Google to index the
example.pantheon.io is now stuck in the index and is displacing the production site to Google #23 even for its own branded query. SEO guy is SEO sad.
We are verified in GSC** on both dev and production.
Normal advice would be:
Add 'noindex' directives to page, fetch and wait Add password to page (403,) fetch and wait. Temporarily redirect page to production (301,) fetch and wait.
Of course none of these work because unless gooblebot can see these 403/301/404/etc. responses, the page will remain in the index. With Pantheon's "injected" robots.txt, we're SOL.
Do you have any ideas of how we might force this out of the index?
*It's worth pointing out to non-Pantheon folks that there is no way to change "example" in the staging URL to something else. We have no control over the robots.txt file, and cannot remove it.
**If your idea is URL removal tool: URL removal offers us a short-term hide on the
example.pantheon.io site, however this would only mask our efforts temporarily, and I have recommended against this for now. The removal tool will not work on 401.