How to add robots.txt to my wordpress.com site?
1
1
WordPress.com comes with a robots.txt file already installed, and does not allow you to edit it directly. However, if you edit your blog's privacy settings, your robots.txt will update itself accordingly.
A setting of public gives this robots.txt file:
# If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
# Please see https://developer.wordpress.com/docs/firehose/ for more details.
Sitemap: https://example.wordpress.com/sitemap.xml
Sitemap: https://example.wordpress.com/news-sitemap.xml
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /wp-signup.php
Disallow: /press-this.php
Disallow: /remote-login.php
Disallow: /activate/
Disallow: /cgi-bin/
Disallow: /mshots/v1/
Disallow: /next/
Disallow: /public.api/
# This file was generated on Thu, 13 Dec 2018 19:12:49 +0000
And a setting of hidden or private gives this robots.txt file:
User-agent: *
Disallow: /
# This file was generated on Thu, 13 Dec 2018 19:13:08 +0000
-
I did stumble across this blog article that appears to show how you can modify
robots.txt(onwordpress.com) by "hook[ing] into thedo_robotstxtaction, or filter the output by hooking intorobots_txt"? – MrWhite Dec 13 '18 at 19:44 -
1@MrWhite That's for "Wordpress VIP" - an enterprise product. I think it's safe to assume that the asker isn't using the enterprise version or they would have mentioned it. But it's good to know that it works differently. – Maximillian Laumeister Dec 13 '18 at 19:47
