Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I argued, "Googlebot can crawl the webpage even if we don't submit a sitemap.xml in Google Webmaster Tools or don't have any links in the page. That is, Googlebot can crawl the page without any external force, it would automatically visit the page and crawl it."

What's the exact answer? Please inform me of any other factors involved.


Sitemap Confusion

Many people believe that a sitemap is required for the site to be discovered this is not true, a sitemap's purpose to help Google ensure that the crawling of data is done correctly. Google has many discovery methods that expends further than just a sitemap.

DNS Collective

When a new domain is registered these records are obtainable though DNS servers, Google even owns its own DNS servers via and which is a public DNS service, new domains will surely be added to these automatically and while I can't find official response on this I'm dead certain that they are sure to have something in place.

Unstoppable Text References and Backlinks

Often newly registered domains will appear on lists all over the world, even some sites publish whois data whenever anyone does a whois on the domain.. It saves it to a page for example and then crawls for more data.

Deep Pages

These can't be detected unless there is some reference to them, this includes text mentions, link mentions.

| improve this answer | |
  • thank you for your answer but my question is not about domain and its about other webpages.. – Sathiya Kumar Jun 13 '13 at 14:15
  • 1
    Added to answer, without a backlink, on page link, or sitemap Google will not be able to discover the page. – Simon Hayter Jun 13 '13 at 14:20
  • 1
    @bybe: to your comment, I answer it's not 100% sure, it's theoretical as I mentioned in a comment of my answer. In practice, Googlebot can try to follow a text representing a link found in a webpage (not a link with <a> tag, just text). – Zistoloen Jun 13 '13 at 14:51
  • Google does read links in raw format, this has been proven several times with people using the referrer trick using logs for text backlinks.. You know sites that stupidly have their logs viewable online when people use referrers on them to gain back links in text form - its vary common. – Simon Hayter Jun 13 '13 at 15:12

Sitemap.xml helps indexing a website by specifying to Googlebot to crawl your website but if a website doesn't have a sitemap.xml, of course Googlebot can crawl it.

Googlebot follow links on internet and if it finds a dofollow link to one of your webpage, it follows it and crawl your webpage (and maybe your website).

| improve this answer | |
  • So you are saying that without any link or site map Google bot won't crawl that page.. right? – Sathiya Kumar Jun 13 '13 at 14:16
  • 1
    @John Sanjay: If you don't submit sitemap.xml to Google Webmaster Tools and your webpage don't have incoming links, Google theoretically can't find your webpage. – Zistoloen Jun 13 '13 at 14:24
  • 1
    Theoretically yes, but it does happen. I'm not sure how, but I've seen Google discover pages that I know have no links to and are not in any sitemaps. Theory's for other means of discovery are Google analytics's codes, data from chrome browser, amongst other things. – Max Jun 13 '13 at 16:20
  • 1
    @moobot: I would be interested by a link that proves your saying, any to share? – Zistoloen Jun 13 '13 at 18:38
  • 1
    @Zistoloen: No none at the moment. I might throw up a test URL and see what happens. If I do, I'll keep you informed. – Max Jun 14 '13 at 2:09

Google doesn't guarantee to crawl your website. So it is a dice roll with waiting on Googlebot to crawl, let alone index your website. You should setup a sitemap.xml file in Google Webmaster Tools -- your website will be indexed much faster as you are making it known to Google that your website exists.

Without links from other webpages to your website, Googlebot may never find your page and crawl it.

Stick with a sitemap.xml file and tell Google about your site.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.