0

I have a headless wordpress site just using wordpress API.

My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress.

I set up robots.txt as follows:

User-agent: *
Disallow: /wordpress/
Disallow: /wordpress/wp-admin/
Disallow: /wordpress/wp-login.php

However, looks like google still indexed pages under /wordpress, and I have a coverage issue warning in google search console Indexed, though blocked by robots.txt.

  1. Most of the pages it indexed actually don't even exist, they 404. So I could allow google to try to crawl them and consequently de-index. But wow did Google even find these pages? I don't have them linked anywhere on my site or in my submitted sitemap.xml. E.g. something like https://example.com/wordpress/product/awesome-product is indexed with No information is available for this page., while the real page is also indexed and exists at https://example.com/product/awesome-product

  2. Other pages do actually exist - for example API endpoints that are consumed by my site. Should I care if these are indexed or not?

2

Using Disallow in robots.txt doesn't prevent search engines from indexing pages, only from crawling them - this is a common SEO myth.

To answer your questions:

  1. Googlebot finds pages not only through links and the sitemap, but also by reading the page's source code for URLs in JavaScript code, etc. It's quite clever.

  2. I wouldn't worry too much about API endpoints being indexed.

The pages that you don't want to be indexed should serve up a meta noindex HTML tag, or a X-Robots-Tag: noindex HTTP header. Make sure to remove those pages from your robots.txt so that Google can crawl them and see the directive that they shouldn't be indexed.

More details here:

https://developers.google.com/search/reference/robots_meta_tag

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.