I have an e-commerce website www.beyondtime.in. I have been constantly monitoring Googlebot crawling on my website and my webmaster account. Lately, I have found two issues that I have not been able to understand.

1.) The Google Bots have been only crawling www.beyondtime.in/telecom.php when the URL is not even valid. What needs to be done to let Google crawl other pages of the website as well?

2.) The second question is about the Google Webmaster account, where I've submitted my sitemap with 227 URLs. Out of that, only 156 have been indexed. None of the images of my website have been indexed by Google.

  • You say that googlebot has only crawled that one page, yet you have many indexed pages - so googlebot must have crawled many pages? How did you measure this?
    – MrWhite
    Sep 27, 2012 at 14:15
  • w3d.. we have the log system of our site.. who-ever visits our page, it will store the IP and the Name of the system..the name is always google-bot
    – Anusha
    Sep 28, 2012 at 3:53

1 Answer 1


Your images are probably not indexed because you block the bot in your robots.txt file to crawl directory /images.

At first glance I would also add a User-agent: * line above the Allow: / line.

Also add the line

Sitemap: http://www.beyondtime.in 

to your robots.txt file (to let more search engines detect it).

Double check your webmastertools (under section Status > Blocked Urls) say nothing is blocked.

You have a low (a very low) number of backlinks so far (as verified by Opensiteexplorer. That and in combination with low traffic can lead to slow indexing of all your pages. A quick check with this SEO spider tool results in good looking status codes.

One BIG Problem I see is that almost every page has a rel=canonical directive linking to the homepage. That reads like "Almost every page is a duplicate of the homepage, so ignore it (the page with the canonical-reference).

(If this is all fixed I would double check if my sitemap.xml lists all pages and if the pages allow indexing and following).

  • But why The Google Bots have been only crawling beyondtime.in/telecom.php this URL of my website, when the URL is not even valid.
    – Anusha
    Sep 27, 2012 at 12:22
  • The link could have existed at some point in time. Another reason is: In order for your site to get noticed by a search engine someone placed a link to that url, or that url was submitted to a search engine, things like that.
    – initall
    Sep 27, 2012 at 12:40
  • Just to add, XML sitemaps are only a helpful suggestion for Google. There is no guarantee that Google will index all the pages in your sitemap.
    – MrWhite
    Sep 27, 2012 at 14:19
  • @w3d Correct. But invented especially to allow to make the search engines do a better job and potentially crawl ALL pages (sitemaps.org). It's a helper, a fallback thing for Anusha if some of his pages are not interlinked.
    – initall
    Sep 27, 2012 at 14:31
  • @initall: My comment was really intended for Anusha, since they were specifically asking why only 156 of the 227 submitted URLs were indexed. The fact is that this is common and could be perfectly normal even when all other website factors are correct. Don't expect Google to index all of the submitted pages, in the same way that Google will not necessarily index all of your pages, period. That was the only point I was making.
    – MrWhite
    Sep 28, 2012 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.