0

I own a website http://citrusbug.com. We did not have proper content before so to prevent search engine from crawling our site we added a robots.txt with code to prevent crawling.

Now we had updated our sites content and want to get it crawled, but Google is not crawling it yet. We removed the code from robots.txt to prevent Googlebot to crawl, but still it is not getting crawled.

We almost did this change before 20-21 days but still Google is not crawling our site and if search I search on Google it displays:

A description for this result is not available because of this site's robots.txt – learn more.

Kindly guide me, what I need to do for this?

Improve this question
1
  • Also many other questions and answers explaining why the site not be completely indexed, but Moobot answered well about more time to crawl. It's normal for sites to take ages to crawl, but do check obvious things like noindex, robots.txt (Again, tons of pages on here with this information).
    – Simon Hayter
    Commented Aug 1, 2013 at 10:10

2 Answers 2

Reset to default
1

I'd highly recommend signing up to Google's Webmaster Tools: https://www.google.com/webmasters/.

You can submit your site to Google there, check for crawl errors and get stats and details on how your site is performing in Google Search.

Improve this answer
0

Your site is being by crawled Google, you can see pages are getting indexed here

You may need to just give it a bit more time to crawl and re-index the enitre site.

I see you have a sitemap.xml file, have you submitted it in Google Web Master Tools? If not, do this, it could speed up the process.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.