How to add
robots.txt to my wordpress.com site?
WordPress.com comes with a
robots.txt file already installed, and does not allow you to edit it directly. However, if you edit your blog's privacy settings, your
robots.txt will update itself accordingly.
A setting of
public gives this
# If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead. # Please see https://developer.wordpress.com/docs/firehose/ for more details. Sitemap: https://example.wordpress.com/sitemap.xml Sitemap: https://example.wordpress.com/news-sitemap.xml User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-login.php Disallow: /wp-signup.php Disallow: /press-this.php Disallow: /remote-login.php Disallow: /activate/ Disallow: /cgi-bin/ Disallow: /mshots/v1/ Disallow: /next/ Disallow: /public.api/ # This file was generated on Thu, 13 Dec 2018 19:12:49 +0000
And a setting of
private gives this
User-agent: * Disallow: / # This file was generated on Thu, 13 Dec 2018 19:13:08 +0000