5

I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are generated and he waits. Considering I cannot pre-create them all, I am trying to figure out the best SEO approach to handle these pages.
Options:

  1. Create really simple pages for the web spiders and only real users will fetch the data and generate the page. A little bit 'afraid' google will see this as low quality content, which might also feel duplicated.
  2. Put them under a directory in my site (e.g. /non-generated/) and put a disallow in robots.txt. Problem here is I don't want users to have to deal with a different URL when wanting to share this page or make sense of it. Thought about maybe redirecting real users from this URL back to the regular hierarchy and that way 'fooling' google not to get to them. Again not sure he will like me for that.
  3. Letting him crawl these pages. Main problem is I can't control to rate of the API calls and also my site seems slower than it should from a spider's perspective (if he only crawled the generated pages, he'd think it's much faster).

Which approach would you suggest?

Improve this question
7
  • Is the page quick on subsequent requests (ie. after the first request)? Presumably you do want these pages indexed?
    – MrWhite
    Sep 8, 2012 at 11:49
  • @w3d yep it is quick on subsequent requests. I want these pages indexed, but am afraid of the site speed and lack of control over how much, so thinking of doing the spidering myself before spiders.
    – Noam
    Sep 8, 2012 at 11:51
  • 1
    Yes, I would guess that Googlebot would timeout if it was taking 1 minute to get a response. When you generate the links to these pages do you know if that page has been generated already? If so then you could perhaps include rel="nofollow" on the link if it has not yet been generated, and remove it when it has? Google will not index the page the first time, but should once the page has been generated and nofollow is removed - IMO. The other concern is site performance as this is calculated from real visitors, but the nofollow attribute might resolve this as well?
    – MrWhite
    Sep 8, 2012 at 12:05
  • You could generate a page with only title, some generic content dynamically generated based on the page, and the actual content will take a minute through an ajax call. Would this be possible in your case?
    – milo5b
    Sep 8, 2012 at 13:44
  • @milo5b wouldn't spiders consider such pages as low quality pages?
    – Noam
    Sep 8, 2012 at 15:41

1 Answer 1

Reset to default
1

If I were you, I'd ensure the link is "nofollow" unti the page has been generated once. Once it's generated, store it in a cache or create a static page from it. Then make the link "follow" so the content can be indexed.

Optionally, if it's an option, consider generating all those pages in some Cron Job overnight so they won't take that long to load.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.