Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I finished my website and launched it, I think I had a problem with my robots.txt - so I changed it to look like this:

# Allows all bots

Sitemap: http://www.mysite.co.za/sitemap.xml

User-agent: *
Disallow: /dashboard/

When I google my domain.co.za - I get this back: Home

A description for this result is not available because of this site's robots.txt – learn more.
You've visited this page 3 times. Last visit: 2012/08/15

Now since I fixed this and added a 301 redirect to redirect mysite.co.za to www.mysite.co.za I would love it if google bot would come do a visit.

Is there anything I can do to get this fixed?

share|improve this question
in google webmaster tools, check when google bot last crawled your robots.txt – YardenST Aug 17 '12 at 16:48
up vote 4 down vote accepted

Dave is right on the money (virtual +1 as my score is still too low)

Please Do the Following

  • Go to Webmaster Tools
  • Do a "Fetch as Googlebot"
  • once fetched, a "Submit to Index" option will appear.
  • When a question pops-up, select "This and all linked URLs"

This will speed things up, but don't expect any instant results.

share|improve this answer

You can now use the 'Fetch as Googlebot' tool in Google Webmaster Tools as a way to persuade Google to re-crawl (or crawl for the first time) a page - http://googlewebmastercentral.blogspot.co.uk/2011/08/submit-urls-to-google-with-fetch-as.html

share|improve this answer
# 03/08/2012
# Allows all bots

You should comment out (prefix with #) the first line also.

Google should refetch your robots.txt file quite quickly (within a few days, depending on the cache headers of your site), but it could take a while longer for your site to be reindexed. Unfortunately you're probably just going to have to be patient.

share|improve this answer

I would like to add that it is not necessary to add a robots.txt file on your website to get it crawled. I found many people having misconceptions about the robots.txt file. Search engine bots will crawl your website even though they don't have the robots.txt file.

If you don't want to show some content to search engine, however, robots.txt is required.

In your case the following steps will result in your website being crawled quickly:

  • Submit your website to relevant directories
  • Bookmark your website in leading bookmarking websites like (digg, stubleupon, reddit)
  • If you are having good marketing budget, write press releases about launch of your website and publish on reputed PR website like, prweb.com etc.
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.