I have a few hundred soft 404 errors reported in Google Search Console. Almost all of them are for CSV files containing data. For example here is the HTTP response for one of them:

HTTP/1.1 200 
Content-Disposition: attachment; filename="fewer-bank-failures.csv"
Content-Length: 116
Content-Type: text/csv; name="fewer-bank-failures.csv";charset=UTF-8
Date: Thu, 01 Feb 2018 11:32:56 GMT
Server: Apache
Connection: keep-alive

"",Bank Failures
2000,2
2001,4
2002,11
2003,3
2004,4
2005,0
2006,0
2007,3
2008,25
2009,140
2010,157
2011,92
2012,51

Why is Google reporting that this is a soft 404? I've usually seen soft 404 because:

  • You have a "200 OK" status but say "not found" in the page
  • You redirect to the home page
  • The page is blank

I can't figure out why Google would think that this CSV file would indicate a not found error.

I do understand other reasons that Google might not want to index this content:

  • It is a download attachment rather then a page
  • CSV wouldn't be the best landing page experience
  • The content is duplicate -- we have the an HTML page with the same data including a graph.

I would expect Google to choose not to index the page for one of those reasons, but I am completely surprised that they call it a "soft 404".

What can I do to tell Google that the page is real? Would using a Link: <https://example.com/fewer-bank-failures.html>; rel="canonical" HTTP header help?

  • Google does not use 404 for content it does not want to index. Check where the CSV is linked from but other than that sometimes the Googlebot reports false negatives, simply mark the issue as corrected (if it works for you) and then see if it reoccurs, chances are it will not. – Simon Hayter Feb 1 at 12:03
  • The HTML version of the page links to the CSV so users can download the data. I don't think it is a temporary thing. I have 354 soft 404 errors in search console, all of which are these CSV files. Google is reporting crawl dates from 7/16/16 through 1/12/18, so it looks like this has been an issue for a while, but I'm just seeing it with the new index report in search console. – Stephen Ostermiller Feb 1 at 12:10
  • Based on What's the smallest page size for Googlebot to not go “Soft 404”?, it may have to do with limited amount of content in these CSV files. I'd still like to know how to tell Google that it is real data or to direct them to the HTML version. – Stephen Ostermiller Feb 1 at 14:28
  • 1
    It seems like the two of you have a handle on this. Just throwing in 2¢. I think I would use nofollow and possibly restrict with robots.txt. Why? Because sometimes you have to make compromises. This seems like a fight I would rather not take on if the data is in HTML files. Jist a thought. Cheers!! – closetnoc Feb 1 at 16:17

Well, it does fall within the realms of Google's definition of a soft 404 (highlighting my own):

A soft 404 means that a URL on your site returns a page telling the user that the page does not exist and also a 200-level (success) code to the browser. (In some cases, instead of a "not found" page, it might be a page with little or no usable content -- for example, a sparsely populated or empty page.)

So, from that "definition" you can't really say it's not a soft 404.

What can I do to tell Google that the page is real?

But is the .csv file is a "real page"? It is a file that is available for download from the "real page". You could try setting a rel="canonical" Link: HTTP response header (as you suggest), but it may just be ignored, as it's not duplicate nor indexed (which is what the rel="canonical" tag is usually used to resolve). However, there would be no harm in doing so.

Since Google already considers these files "soft-404" they aren't going to be indexed anyway. You could simply block crawling with robots.txt (as @closetnoc suggested in comments) to save some time and suppress the warning.

Alternatively, you could send an X-Robots-Tag: noindex HTTP response header to simply block indexing (and still allow the resource to be crawled).

At the end of the day, the "soft-404" warning is just a FYI from Google. If Google considers there to be "no usable content" then it's not going to be indexed, whether Google tells you about it or not. So there is nothing you really can do, apart from suppressing the warning in GSC.

It does seem like it falls under the Google's linked description of a soft 404 by @Docroot. I think in a situation like this, even though we know that these CSV files aren't worth indexing, we might not want to block Google from being able to crawl them because we don't want to hide content from Google so that Google doesn't think there's anything deceptive going on with the site.

I'm not sure of a remedy to this issue, but here are some possible solutions:

  1. Maybe have the user download the csv files through Javascript rather than a followed link. Google will be less likely to crawl the Javascript URLs and will be more likely to see these URLs as server requests than actual landing pages. While it might be wise to allow Google to crawl these CSV files, it also isn't useful to pass link juice to them. Javascript could remedy this.

  2. Perhaps there is an alternate header status that you can use, though I'm not sure if that is possible.

  3. I don't think it's possible to set a NOINDEX NOFOLLOW status in a csv file, but it's something to think about. If there's a way to tell Google not to index these pages that would be better than them interpreting all of these links as soft 404s.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.