3

Let's say I'm building an open wiki system. Each user can start a new page, give it a name like "how-to-travel-faster-than-light" which will turn into the permalink: "mysite.com/how-to-travel-faster-than-light".

There is no site map listing every link to every user page.

My question is: How would google ever find and index the "how-to-travel-faster-than-light" page if it can't find it, even recursively from my landing page (recursively).

How did wikipedia or stack exchange solve this problem? obviously every page of theirs is indexed and they have no single list of all page links in a central location which can be found by a crawler. Have they submitted every page to google for indexing?

thanks.

Improve this question
1
  • How would anyone find that page if it’s not linked from anywhere?
    – DisgruntledGoat
    Commented Jan 17, 2018 at 21:04

1 Answer 1

Reset to default
2

How would google ever find and index the "how-to-travel-faster-than-light" page if it can't find it, ...

If it can't find it; it can't index it. It's that simple. However, Google can potentially discover URLs in many ways, from scanning emails in Gmail to URLs typed into its Chrome web browser (although these methods are naturally unreliable if you are trying to get a URL indexed).

How did wikipedia or stack exchange solve this problem?

Well, Wikipedia does have numerous Contents and hierarchy Indices, as well as a complete alphabetical index - so it would certainly seem spiderable. It is also very well cross-linked and inbound links are second to none. It might even have a sitemap (although I can't find it), as it's still well within the 2.5 billion sitemap URL limit (2017 figures) as set by Google.

The Stack Exchange Questions page - that lists new questions first (and active questions rise to the top) is naturally a spiderable index of all pages/questions on the site. There is also an RSS question feed. And, until recently, there was an XML sitemap. (The sitemap URL stated in the Webmasters robots.txt seems to result in a 404 currently?)

There is no site map listing every link to every user page.

Why not? This is a standard way of informing search engines about hard to reach pages.

When pages are created you can ping Google and other search engines to notify them of an update. They will then request your updated (auto-generated) sitemap.

Just having Google Analytics installed on your site will at least allow Google to discover URLs the very first time you (or the author) visits the page.

Improve this answer
2
  • 1
    understood. So as long as I have an index, even if users can't really see it or use it, it's worthwhile to keep it and link to it from the main page hierarchy so that google can find it. with GA, it'll add URLs to be indexed every time a user loads up a deep page for the first time, and there are ways to ping google with new URLs you've just created. Excellent. Thanks for your response.
    – JasonGenX
    Commented Jan 15, 2018 at 23:31
  • Yes, having some way for the search engines/Google to discover the new URL is the first step. This should allow the search engine to index the URL/page. However, how well the page ranks depends on other factors and may come much later.
    – DocRoot
    Commented Jan 15, 2018 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.