Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I can understand how to disallow bots to crawl some pages/folder in normal application. For example for google-bot it is nicely described here.

But what should I do if I have a single page application (the one that uses only ajax to upload new content and has routing and page generation on the client). How to make it crawlable is described here and here, but what if I do not a bot to follow some links (that are on my starting page)? By this I mean the following:

When SPA is loaded for the first time it loads some basic HTML. This html can have specific links like:

  • home (#!home/)
  • about (#!about/)
  • news (#!news/)

but I do now want a bot to crawl #!about link.

share|improve this question
    
Please clarify your question. "but what if I do not a bot to follow some links?" does not make much sense, I'm afraid. –  edvinas.me May 1 at 7:56
    
@edvinas.me I modified my question. May be this makes more sense? –  Salvador Dali May 1 at 8:19

1 Answer 1

up vote 3 down vote accepted

I have found a way to do exactly what I want. It is nicely documented by google:

When your site adopts the AJAX crawling scheme, the Google crawler will crawl every hash fragment URL it encounters. If you have hash fragment URLs that should not be crawled, we suggest that you add a regular expression directive to your robots.txt file. For example, you can use a convention in your hash fragments that should not be crawled and then exclude all URLs that match it in your robots.txt file. Suppose all your non-indexable states are of the form ' #DONOTCRAWLmyfragment. Then you could prevent Googlebot from crawling these pages by adding the following to your robots.txt:

Disallow: /*_escaped_fragment_=DONOTCRAWL

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.