Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This question already has an answer here:

I own a website http://citrusbug.com. We did not have proper content before so to prevent search engine from crawling our site we added a robots.txt with code to prevent crawling.

Now we had updated our sites content and want to get it crawled, but Google is not crawling it yet. We removed the code from robots.txt to prevent Googlebot to crawl, but still it is not getting crawled.

We almost did this change before 20-21 days but still Google is not crawling our site and if search I search on Google it displays:

A description for this result is not available because of this site's robots.txt – learn more.

Kindly guide me, what I need to do for this?

share|improve this question

marked as duplicate by Simon Hayter Aug 1 '13 at 10:08

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Also many other questions and answers explaining why the site not be completely indexed, but Moobot answered well about more time to crawl. It's normal for sites to take ages to crawl, but do check obvious things like noindex, robots.txt (Again, tons of pages on here with this information). – Simon Hayter Aug 1 '13 at 10:10

I'd highly recommend signing up to Google's Webmaster Tools: https://www.google.com/webmasters/.

You can submit your site to Google there, check for crawl errors and get stats and details on how your site is performing in Google Search.

share|improve this answer

Your site is being by crawled Google, you can see pages are getting indexed here

You may need to just give it a bit more time to crawl and re-index the enitre site.

I see you have a sitemap.xml file, have you submitted it in Google Web Master Tools? If not, do this, it could speed up the process.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.