1

Until now I've had my robots.txt file configured as follows:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

As this has been some time now since I last touched it, I've been searching around to see what is the best practice nowdays in 2018 and if I should update the file.

I've found this article by Yoast where it basically says not to block anything at all and set the robots.txt file as follows for the best results:

# This space intentionally left blank
User-agent: *
Disallow: /suggest/?*

Hmm, now I'm confused … while this article by Yoast comes from a reputable source and is a good read giving a few tips, it totally fails to explain what it actually is that they're telling the bots to do.

First of all, what does /suggest/?* actually mean? Is the 'suggest' meant to be just a placeholder? Should the line actually say Disallow: /?* instead (without 'suggest')?

And secondly, is this really the ideal robots.txt for the best SEO results? Or should I just stick to what I currently have? Or is there any other suggestion?

closed as too broad by Stephen Ostermiller Jun 12 '18 at 10:47

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • I also found this line as a suggestion Disallow: /?s=* – Joe Bloggs Jun 11 '18 at 2:38
  • WordPress is very modular. Many plugins require their own robots.txt directives. There is no single answer to this question. The robots.txt for your WordPress needs to be customized to what you actually have installed. – Stephen Ostermiller Jun 12 '18 at 10:48
1

The article you linked explains that /suggest/ is from Yoast's own suggest tool. That is something specific to their site only (not sites using their plugin) so it's irrelevant for you.

So you don't need to block anything. You can either use a blank robots.txt file, or a simple version like this which disallows nothing:

User-agent: *
Disallow:

Not the answer you're looking for? Browse other questions tagged or ask your own question.