We accidentally deployed a robots.txt from our development site that disallowed all crawling. This has caused traffic to dip dramatically, and google results to report:

A description for this result is not available because of this site's robots.txt – learn more.

We've since corrected the robots.txt about a 1.5 weeks ago, and you can see our robots.txt here.

However, search results still report the same robots.txt message. The same appears to be true for Bing.

We've taken the following action:

  • Submitted site to be recrawled through google webmaster tools
  • Submitted a site map to google

(basically doing everything possible to say "Hey we're here! and we're crawlable!")

Indeed a lot of crawl activity seems to be happening lately, but still no description is crawled.

I noticed this question where the problem was specific to a 303 redirect back to a disallowed path.

We are 301 redirecting to /blog, but crawling is allowed here. This redirect is due to a site redesign, wordpress paths for posts such as /2012/02/12/yadda yadda have been moved to /blog/2012/02/12. We 301 redirect to wordpress for /blog to keep our google juice. However, the sitemap we submitted might have /blog URLs. I'm not sure how much this matters. We clearly want to preserve google juice for URLs linked to us from before our redesign with the /2012/02/... URLs.

So perhaps this has prevented some content from getting recrawled? How can we get all of our content, with links pointed to our site from pre-and-post redesign reporting descriptions? How can we resolve this problem and get our search traffic back to where it used to be?

  • 1
    What do you mean by 'no description is crawled'? Are your refering to meta description? Commented Aug 17, 2014 at 14:03
  • 1
    @JVerstry - I think the OP means that no description appears in the SERPs (ie. is indexed).
    – MrWhite
    Commented Aug 17, 2014 at 14:09
  • 1
    Possibly bit of an aside, but... Much of your site is available on both www and non-www domains (except for your blog that redirects to www). Since your sitemap refers to the www subdomain, I would assume you should redirect one to the other. You mention that your are "301 redirecting" blog paths to /blog/ - however, this does not seem to be the case? Both URLs are valid, it looks like you are internally rewriting (not redirecting) one to the other? The /blog/ URLs do not appear in your sitemap? You are also referencing a non-existent xml-stylesheet in your sitemap.
    – MrWhite
    Commented Aug 17, 2014 at 14:27

2 Answers 2


As you already noticed that bots are crawling your pages again, it is only a matter of time when they will crawl more of your pages and show indexed snippets in their search results.

While it will likely not result in any problems, your current robots.txt is invalid according to the original specification, because your record doesn’t contain a Disallow line (emphasis mine):

The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below.

So your robots.txt should be:

User-agent: *

Apart from that …

As also mentioned by w3d in the comments, it would be a good practice to use the same host name for all your pages: either with the www subdomain or without it.

Your sitemap tries to load an XSL stylesheet which doesn’t exist.

Your sitemap is in the wrong location. Because it is at http://opensourceconnections.com/blog/sitemap.xml, it may only contain URLs that start with http://opensourceconnections.com/blog/. But the paths of the URLs you list don’t start with /blog/.

Your blog redirect doesn’t seem to work. These two URLs show the same page:

  • 1
    Thanks, we've since added a redirect to /blog, and made www redirect to non-www in all cases. We've also fixed the sitemap. The sitemap was actually completely incorrect and about 4 months old. Apparently generated by an uninstalled WordPress plugin. I guess we can only wait as google catches up
    – Doug T.
    Commented Aug 20, 2014 at 1:41

I see on your robots.txt

User-agent: *

To allow all, you should declare a user agent with a specific directive. Use :

User-agent: *
Allow: /

This will make sure that Google knows that everything under the root folder is allowed to be in the index. Give it a few weeks, Google's crawler deals with dozens of thousands of sites and it does not update in a jiffy.

As for the redirection - I'm not I got your question. It seems that you did right by appending the /blog to the old URLs using 301. The juice should flow to the new URLs. The sitemap should reflect the new URL structure, so there seems to be no problem here either.

  • To be honest I don't think it will make any difference if you have Allow: / (or Disallow:) here or not.
    – MrWhite
    Commented Aug 17, 2014 at 14:43
  • And why is that?
    – dm-guy
    Commented Aug 17, 2014 at 17:58
  • By itself, with no "records" following it, User-agent: * is simply an empty "group" - it means nothing. Default behaviour is to allow, so this is likely to be simply ignored by the search engine crawler. In fact, if you try this in the robots.txt Tester tool in GWT, it returns a result of "allowed".
    – MrWhite
    Commented Aug 17, 2014 at 18:15
  • Well, official documentation encourages you to be specific with your directives, and this way you are able to control better what you want to reach. "default behavior" is not always the same across the different search engines (there is more out there than just Google, Bing and Yahoo), but thanks for your feedback.
    – dm-guy
    Commented Aug 17, 2014 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.