Until now I've had my robots.txt file configured as follows:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

As this has been some time now since I last touched it, I've been searching around to see what is the best practice nowdays in 2018 and if I should update the file.

I've found this article by Yoast where it basically says not to block anything at all and set the robots.txt file as follows for the best results:

# This space intentionally left blank
User-agent: *
Disallow: /suggest/?*

Hmm, now I'm confused … while this article by Yoast comes from a reputable source and is a good read giving a few tips, it totally fails to explain what it actually is that they're telling the bots to do.

First of all, what does /suggest/?* actually mean? Is the 'suggest' meant to be just a placeholder? Should the line actually say Disallow: /?* instead (without 'suggest')?

And secondly, is this really the ideal robots.txt for the best SEO results? Or should I just stick to what I currently have? Or is there any other suggestion?

  • I also found this line as a suggestion Disallow: /?s=*
    – Joe Bloggs
    Jun 11, 2018 at 2:38
  • WordPress is very modular. Many plugins require their own robots.txt directives. There is no single answer to this question. The robots.txt for your WordPress needs to be customized to what you actually have installed. Jun 12, 2018 at 10:48

1 Answer 1


The article you linked explains that /suggest/ is from Yoast's own suggest tool. That is something specific to their site only (not sites using their plugin) so it's irrelevant for you.

So you don't need to block anything. You can either use a blank robots.txt file, or a simple version like this which disallows nothing:

User-agent: *

Not the answer you're looking for? Browse other questions tagged or ask your own question.