3

Two weeks ago I changed the address of my site. I forgot to renew my domain, and for a few days my site was unreachable. This resulted in google indexing my site improperly. There is a robot.txt error on search results of my website. A description for this result is not available because of this site's robots.txt.

This is my robots.txt:

User-agent: *
Allow: /

I've already forced google to index my site via webmaster tools.

EDIT: It only doesn't work with some posts

  • "a robot.txt error" - what was the error exactly? – DocRoot Mar 16 '16 at 20:49
  • @DocRoot "A description for this result is not available because of this site's robots.txt" – mszoszk Mar 16 '16 at 20:53
  • The robots.txt file above won't block Google. So, this implies that at some point in the recent past you had a different/blocking robots.txt file? – DocRoot Mar 16 '16 at 21:05
5

The standard format for a robots.txt file, to allow everything, would be

User-Agent: *
Disallow: 

You only need to use Allow to override a Disallow rule. I would suggest making the change; Google can take up to a day to then update the robots.txt cached version.

|improve this answer|||||
  • I did a robots.txt as a result of the error. Before I hadn't had a robots.txt. – mszoszk Mar 16 '16 at 20:11
  • When you say it was improperly indexed, what was the problem? If it indexed the holding page that was there when your site was down it may just need that page resubmitting? Similarly if there are 404 errors but the pages are now active on the site again you should be able to mark as resolved. If you change the robots.txt, does it resolve the error? – Jaryd Buggins Mar 16 '16 at 20:24
  • I've just read you resubmitted your site, it may take a few days to update the cached version, when did you resubmit? – Jaryd Buggins Mar 16 '16 at 20:26
  • It was February 28 when my site went back online and I added it to webmaster tools (including submitting the change of address after I re-bought my old domain 2 days later). I forced indexing 1.5 weeks ago. – mszoszk Mar 16 '16 at 20:49
  • 2
    @mszoszk Search engines are notoriously slow. If you can Fetch and Render your site okay in Google Search Console, then all should be okay. You can also test the robots.txt file there too. If these two things work, then you are golden. Also, Jaryd Buggins is giving good advice on how to structure your robots.txt file. You should make this change. – closetnoc Mar 16 '16 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.