Bit of a technical question here that I'm working on with my dev. I have a load of Ajax request URL's appearing in Search Console as soft 404's. These requests are for buttons on the page i.e. 'compare products'. We are considering adding an x-robots noindex meta tag to the header for the request if the request is from a search engine. Does anybody know if this noindex tag will only be considered for the request URL, or is there a risk of this being seen as a noindex tag for the html page that the request exists on, and remove the entire page from the index? Thanks in advance.
2 Answers
Search engines will remove and not index any URLS that return a no-index response either by the header, within the HTML or within robots.txt So to answer your question... any URLS that have these responses will not be indexed.
If you are concerned then you should test the URLS you want indexed and the ones you don't want indexed using Google's Fetch Tool, or command CURL, or online curl - look at the header responses.
answered Jul 2, 2019 at 12:24
Google won't index URLs that it thinks are soft 404. You don't need to add a noindex to them. Google already treats them that way.
If you have a noindex on them, it won't remove them from the Google Search Console soft 404 report. It doesn't matter though. Having items in that report won't hurt your SEO of the pages you want to get indexed. Google has included those URLs because it found little or no content on them. It is letting you know that you could have a potential problem. It sounds like it isn't actually a problem and you should just ignore those URLs in that report.
If anything, you could list those URLs in robots.txt. Google doesn't need to crawl AJAX associated with user interaction. Google only need to crawl AJAX used while the page loads. It is perfectly fine to disallow AJAX URLs that are used when a user presses a button. Something like:
User-Agent: *
Disallow: /compare-products-ajax.script
answered Aug 1, 2019 at 16:35
