2

I want to prevent my site from being indexed and listed by Google. So I follow the instructions in https://yoast.com/prevent-site-being-indexed/ and add:

Header set X-Robots-Tag "noindex, nofollow"

to my .htaccess. Now how to verify that my site is not indexed? Is there a way to do so?

Thanks

4
  • How long ago did you add the header? – Stephen Ostermiller Apr 20 '20 at 14:14
  • @StephenOstermiller, About 10 hours ago, thank you. – alancc Apr 20 '20 at 22:33
  • You need to wait at least 2 weeks before that takes effect. – Stephen Ostermiller Apr 20 '20 at 23:23
  • @StephenOstermiller, Thank you. Then how to know it has taken effects? – alancc Apr 21 '20 at 22:37
2

As A Biron mentioned, after you've verified that the noindex directive is properly written, use Search Console's coverage tool to look at the number of "valid" indexed pages.

If there are still results, either wait for Google to re-crawl your site on its own, or use Search Console's temporary removals tool to pwn your site's entire search presence at once.

Google Search Console Removal Tool

You'll need to verify your site with Google Search Console to be able to use this tool.

I'm not about to try it on my own site, but I bet if you put your homepage url into that box and click "remove all with prefix", Google will deindex your entire site within a day or so.

Double-check your work by going back to the Search Console coverage tool to make sure there are zero "valid" pages, and use the site:example.com query with your domain in Google Search to make sure there's nothing there either.

2
  • Just want to mention that Google recently clarified that the "site:" command isn't an accurate way to check indexing (or ranking). It will show you "random and incomplete" results. This may still work if the whole site will be noindexed but I wanted to let OP know it may not be accurate. searchenginejournal.com/google-site-search/355248 – A Biron Apr 22 '20 at 20:06
  • 1
    @ABiron "Random", definitely. "Incomplete" is big news to me, though, but plain as day that's exactly what the google engineer said in your link. Thank you for the correction. I'll edit my answer! – Maximillian Laumeister Apr 22 '20 at 20:12
3

The most accurate way is to add your site to Google Search Console and check the Coverage report. If pages are indexed they will be in the green Valid Pages section. If they're not they'll be in the grey Excluded section, in a sub-report called "Excluded by 'noindex' tag".

These reports take time to update if you've just set up Search Console, so if you want to check right after adding your site there, another way is to inspect some pages with the URL Inspector tool, like homepage, category pages, etc. If it worked it will say something along the lines of "URL cannot be indexed, blocked by robots.txt".

In Bing Webmaster tools this info will be in the Index Explorer under the Reports & Data section. Other search engines should have equivalent tools to check indexation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.