Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I argued, "Googlebot can crawl the webpage even if we don't submit a sitemap.xml in Google Webmaster Tools or don't have any links in the page. That is, Googlebot can crawl the page without any external force, it would automatically visit the page and crawl it."

What's the exact answer? Please inform me of any other factors involved.

share|improve this question

closed as not constructive by John Conde Jun 14 '13 at 11:40

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Sathiya, you may want to read this introduction: ligatures.net/content/expertise/web-crawling.html –  JVerstry Aug 11 at 22:01

3 Answers 3

up vote 4 down vote accepted

Sitemap Confusion

Many people believe that a sitemap is required for the site to be discovered this is not true, a sitemap's purpose to help Google ensure that the crawling of data is done correctly. Google has many discovery methods that expends further than just a sitemap.

DNS Collective

When a new domain is registered these records are obtainable though DNS servers, Google even owns its own DNS servers via 8.8.8.8 and 4.4.4.4 which is a public DNS service, new domains will surely be added to these automatically and while I can't find official response on this I'm dead certain that they are sure to have something in place.

Unstoppable Text References and Backlinks

Often newly registered domains will appear on lists all over the world, even some sites publish whois data whenever anyone does a whois on the domain.. It saves it to a page for example and then crawls for more data.

Deep Pages

These can't be detected unless there is some reference to them, this includes text mentions, link mentions.

share|improve this answer
    
thank you for your answer but my question is not about domain and its about other webpages.. –  Sathiya Kumar Jun 13 '13 at 14:15
1  
Added to answer, without a backlink, on page link, or sitemap Google will not be able to discover the page. –  bybe Jun 13 '13 at 14:20
    
Thank you bybe.. –  Sathiya Kumar Jun 13 '13 at 14:25
1  
@bybe: to your comment, I answer it's not 100% sure, it's theoretical as I mentioned in a comment of my answer. In practice, Googlebot can try to follow a text representing a link found in a webpage (not a link with <a> tag, just text). –  Zistoloen Jun 13 '13 at 14:51
    
Google does read links in raw format, this has been proven several times with people using the referrer trick using logs for text backlinks.. You know sites that stupidly have their logs viewable online when people use referrers on them to gain back links in text form - its vary common. –  bybe Jun 13 '13 at 15:12

Sitemap.xml helps indexing a website by specifying to Googlebot to crawl your website but if a website doesn't have a sitemap.xml, of course Googlebot can crawl it.

Googlebot follow links on internet and if it finds a dofollow link to one of your webpage, it follows it and crawl your webpage (and maybe your website).

share|improve this answer
    
So you are saying that without any link or site map Google bot won't crawl that page.. right? –  Sathiya Kumar Jun 13 '13 at 14:16
1  
@John Sanjay: If you don't submit sitemap.xml to Google Webmaster Tools and your webpage don't have incoming links, Google theoretically can't find your webpage. –  Zistoloen Jun 13 '13 at 14:24
1  
Theoretically yes, but it does happen. I'm not sure how, but I've seen Google discover pages that I know have no links to and are not in any sitemaps. Theory's for other means of discovery are Google analytics's codes, data from chrome browser, amongst other things. –  Max Jun 13 '13 at 16:20
1  
@moobot: I would be interested by a link that proves your saying, any to share? –  Zistoloen Jun 13 '13 at 18:38
1  
@Zistoloen: No none at the moment. I might throw up a test URL and see what happens. If I do, I'll keep you informed. –  Max Jun 14 '13 at 2:09

Google doesn't guarantee to crawl your website. So it is a dice roll with waiting on Googlebot to crawl, let alone index your website. You should setup a sitemap.xml file in Google Webmaster Tools -- your website will be indexed much faster as you are making it known to Google that your website exists.

Without links from other webpages to your website, Googlebot may never find your page and crawl it.

Stick with a sitemap.xml file and tell Google about your site.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.