Until now I've had my robots.txt file configured as follows:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
As this has been some time now since I last touched it, I've been searching around to see what is the best practice nowdays in 2018 and if I should update the file.
I've found this article by Yoast where it basically says not to block anything at all and set the robots.txt file as follows for the best results:
# This space intentionally left blank User-agent: * Disallow: /suggest/?*
Hmm, now I'm confused … while this article by Yoast comes from a reputable source and is a good read giving a few tips, it totally fails to explain what it actually is that they're telling the bots to do.
First of all, what does
/suggest/?* actually mean? Is the 'suggest' meant to be just a placeholder? Should the line actually say
Disallow: /?* instead (without 'suggest')?
And secondly, is this really the ideal robots.txt for the best SEO results? Or should I just stick to what I currently have? Or is there any other suggestion?