Consider the case of Quora, which is really SEO-friendly. The problem is that there are no links to questions from Quora homepage. Essentially nothing (even via an indirect URL). So I could never find any link to Quora questions from its homepage; I could only do so via Google (or other search engines), or have a Quora account.

So my question is how to tell Google the list of pages to crawl without listing them all in the homepage (or via an indirect links)?


The problem is that there is no links to questions from Quora homepage.

There is a link to the Directory (at the bottom of the page) which then links to People, Topics, Questions and Blogs which then drills down further. Once the Topics have been indexed then this will also help to discover new questions (together with a possible XML Sitemap, as Ivo Vdv mentions). There is also an RSS feed on the homepage which appears to list a large number of questions.

  • Wow such a mysterious link. Anyway, thank you very much. – hiro Nov 19 '13 at 10:31
  • I don't think that link was always there. Mainly they got direct links to questions from elsewhere on the web. – DisgruntledGoat Nov 19 '13 at 14:19

You're looking for a sitemap. Basically, it's an XML file with the list of (preferably) all of your pages. You can read more about it on https://support.google.com/webmasters/answer/156184?hl=en.

  • Do you think that adding links which are not discoverable from the homepage to a sitemap is a black-SEO technique? – hiro Nov 19 '13 at 10:33
  • 3
    No. It might be bad for user experience, but it's not black hat SEO. – Ivo van der Veeken Nov 19 '13 at 11:29
  • 3
    It's always good to have a clean internal linking structure to make it possible to crawl a site by default. With the Sitemap file you can make it easier for search engines to recognize new & updated pages within the normal site. A Sitemap file shouldn't replace internal links. – John Mueller Nov 19 '13 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.