1

We have added following meta tag on some of our pages:

<meta  content="noindex" name="robots"/>

Does google crawl the content and links on this page and just don't show it on the google search results.
If yes then how can i stop bot not to crawl the content and the links on this page ?

  • 2
    It is not possible for a meta tag to stop the crawler because the meta tag is part of the page itself. The crawler can't see the tag until it has crawled the page. The noindex here says that the crawler shouldn't include it in the search results. – Stephen Ostermiller Sep 22 '17 at 14:14
  • What is your actual goal for this page? Do you want it crawled? Do you want the URL itself in the search index? Do you want the content in the search index? Do you want crawlers to follow the links on the page? Do you want search engines to make a copy of the page available to the public in their cache? – Stephen Ostermiller Sep 22 '17 at 14:33
2

By default Google or any other modern search engine will crawl, index and follow the links in a page when the above meta tag is absent.

In your example Google will find the noindex instruction and this is what is expected to happen:

  • Google bot will crawl the page (if robots txt allows it)
  • Your page will not get indexed and will be removed from SERP, eventually.
  • The search engine will follow all the links in the page.
  • linked pages will get indexed.

Add nofollow

Add this instruction to the meta tag, this will tell Google not to follow the links. But, if these pages are linked from other pages without this instruction the pages will be indexed.

Do not allow Google to crawl the page

Make sure to Disallow o block access via robots txt. However, do this after Google has crawled the page and is aware of the noindex, nofollow instruccions.

2

Once a page is out there, Google is just going to crawl it, suck it up, and retain it in its vast and very long term memory.

If the page you want removed is in Google's SERPs, then the noindex alone won't resolve this, it'll simply advise new search engines not to rank it. You need to add nofollow,noarchive to that page's robots command (it can all be in one command) to flush it out of Google etc search cache, and stop the crawler following the links on that page.

Make sure you're allowing search engines to crawl that page via your robots.txt file, otherwise they'll never see this request.

Once your page has gone from the SERPs (do a few site:[yourdomain] queries in Google search box to check), then you can use the robots.txt to advise search not to go there. I'd opt to also keep the noindex,nofollow,noarchive page-level command in the page too, to keep it out of SERPs just in case the robots.txt file gets an edit.

It can take a while to fix this, but it'll get there.

  • 3
    "If the page you want removed is in Google's SERPs, then the noindex alone won't resolve this" - this is misleading. noindex is enough to remove the page from Google SERPs - it's just that it will take time as Google will need to recrawl the page. nofollow,noarchive does not help with removing a page from the SERPs (a page can't be "archived" if it is not indexed) as you seem to imply. nofollow is only required to prevent links being followed - as you suggest. – MrWhite Sep 22 '17 at 14:34
  • Yes, you're right, nofollow won't stop something being indexed in the first place - but in combination with some noindex tags, it can stop the indexing of subsequent pages connected with the crawler's start page. I just wanted to cover off the original questioner's comment about following links with the nofollow bit. I've found noarchive, in collaboration with noindex to be very effective at flushing pages out of Google's cache. – Andrew Martin Sep 22 '17 at 14:47
1

The tag tells the search engines not to include the page in their listings. As far as i'm aware you can't stop a search engine crawling your page as it will need to craw the page in order to see if you remove the tag. (If you change your mind and want the page to be listed)

  • 2
    ...can't stop a search engine crawling your page with a meta tag. You could suggest robots.txt to prevent crawling. – Stephen Ostermiller Sep 22 '17 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.