1

How to add robots.txt to my wordpress.com site?

1

WordPress.com comes with a robots.txt file already installed, and does not allow you to edit it directly. However, if you edit your blog's privacy settings, your robots.txt will update itself accordingly.

A setting of public gives this robots.txt file:

# If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
# Please see https://developer.wordpress.com/docs/firehose/ for more details.

Sitemap: https://example.wordpress.com/sitemap.xml
Sitemap: https://example.wordpress.com/news-sitemap.xml

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /wp-signup.php
Disallow: /press-this.php
Disallow: /remote-login.php
Disallow: /activate/
Disallow: /cgi-bin/
Disallow: /mshots/v1/
Disallow: /next/
Disallow: /public.api/

# This file was generated on Thu, 13 Dec 2018 19:12:49 +0000

And a setting of hidden or private gives this robots.txt file:

User-agent: *
Disallow: /

# This file was generated on Thu, 13 Dec 2018 19:13:08 +0000
  • I did stumble across this blog article that appears to show how you can modify robots.txt (on wordpress.com) by "hook[ing] into the do_robotstxt action, or filter the output by hooking into robots_txt"? – MrWhite Dec 13 '18 at 19:44
  • 1
    @MrWhite That's for "Wordpress VIP" - an enterprise product. I think it's safe to assume that the asker isn't using the enterprise version or they would have mentioned it. But it's good to know that it works differently. – Maximillian Laumeister Dec 13 '18 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.