I can understand how to disallow bots to crawl some pages/folder in normal application. For example for google-bot it is nicely described here.
But what should I do if I have a single page application (the one that uses only ajax to upload new content and has routing and page generation on the client). How to make it crawlable is described here and here, but what if I do not a bot to follow some links (that are on my starting page)? By this I mean the following:
When SPA is loaded for the first time it loads some basic HTML. This html can have specific links like:
- home (#!home/)
- about (#!about/)
- news (#!news/)
but I do now want a bot to crawl #!about link.