I have links on my site which lead to internal pages that depend on first bringing data from an external API. This only takes time the first time they are pushed. Links to pages which exists in the DB, load a lot faster. I'm specify for search engines to only crawl the internal pages. This is what I thought of:
1. Creating a site map with the internal links
2. Adding this to every page on my site <META NAME="ROBOTS" CONTENT="NOFOLLOW" />

Will this succeed? Does the site map over-ride the nofollow? Would you suggest a different direction?


1 Answer 1


Adding that meta tag will prevent search engines from crawling your entire site, so you should avoid that! Adding pages to the sitemap should allow them to be crawled and indexed (since you are not telling search engines to ignore the actual pages). But if you have no links to them on your own site (as search engines see it) then they will not rank well, or at all.

One solution would be to use robots.txt and block the URLs that use the API, assuming they follow a standard format or it is easy to generate the list of things to block.

However, the better solution would be to spider the site yourself and make sure all pages you link to are already generated and in your database. This way the pages will be fast for users (your main concern of course) and search engines as well.

  • the robots.txt is possible (but not easy) and will also make that file very only (10k*C). Regarding the second solution, whenever a "slow" link is pressed, I'm notifying the user this is something that might take some time. Removing the links will remove the option to request that data for the user, so that isn't an option. Any other thoughts you can share?
    – Noam
    Commented Aug 3, 2011 at 11:58
  • @Noam, I don't mean removing links, I mean generating the pages those links point to, before a user or search engine gets to them. So you just make a list of pages that take a long time to generate, then generate a cache of those at regular intervals, and simply serve up that page whenever a user requests it. Commented Aug 3, 2011 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.