3

Two weeks ago I changed the address of my site. I forgot to renew my domain, and for a few days my site was unreachable. This resulted in google indexing my site improperly. There is a robot.txt error on search results of my website. A description for this result is not available because of this site's robots.txt.

This is my robots.txt: 

User-agent: *
Allow: /

I've already forced google to index my site via webmaster tools.

EDIT: It only doesn't work with some posts

Improve this question
3
  • "a robot.txt error" - what was the error exactly?
    – DocRoot
    Mar 16, 2016 at 20:49
  • @DocRoot "A description for this result is not available because of this site's robots.txt"
    – mszoszk
    Mar 16, 2016 at 20:53
  • The robots.txt file above won't block Google. So, this implies that at some point in the recent past you had a different/blocking robots.txt file?
    – DocRoot
    Mar 16, 2016 at 21:05

1 Answer 1

Reset to default
5

The standard format for a robots.txt file, to allow everything, would be

User-Agent: *
Disallow:

You only need to use Allow to override a Disallow rule. I would suggest making the change; Google can take up to a day to then update the robots.txt cached version.

Improve this answer
7
  • I did a robots.txt as a result of the error. Before I hadn't had a robots.txt.
    – mszoszk
    Mar 16, 2016 at 20:11
  • When you say it was improperly indexed, what was the problem? If it indexed the holding page that was there when your site was down it may just need that page resubmitting? Similarly if there are 404 errors but the pages are now active on the site again you should be able to mark as resolved. If you change the robots.txt, does it resolve the error?
    – Jaryd Buggins
    Mar 16, 2016 at 20:24
  • I've just read you resubmitted your site, it may take a few days to update the cached version, when did you resubmit?
    – Jaryd Buggins
    Mar 16, 2016 at 20:26
  • It was February 28 when my site went back online and I added it to webmaster tools (including submitting the change of address after I re-bought my old domain 2 days later). I forced indexing 1.5 weeks ago.
    – mszoszk
    Mar 16, 2016 at 20:49
  • 2
    @mszoszk Search engines are notoriously slow. If you can Fetch and Render your site okay in Google Search Console, then all should be okay. You can also test the robots.txt file there too. If these two things work, then you are golden. Also, Jaryd Buggins is giving good advice on how to structure your robots.txt file. You should make this change.
    – closetnoc
    Mar 16, 2016 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.