I have both setup in my routes:

Route::get('/robots.txt', function() {
    // robots.txt contents here

Route::get('/sitemap.xml', function() {
    // sitemap.xml contents here

I can access them perfectly through the browser but I'm getting a message from Google Search Console that they are not detected. Do they need to be physical files in the root folder so they can be detected?

  • An HTTP client has no way remotely (except vague heuristics on timing at best) to know if a given resource (URL) is a file on disk, a generated content by a script, a proxy to something else, etc. – Patrick Mevzek Feb 22 '18 at 18:59
  • You are filtering on the User-Agent probably. wget on your URL results in a 404. – Patrick Mevzek Feb 22 '18 at 18:59
  • Seems even worse than User-Agent filtering, wget --user-agent="" with a true browser fingerprint still creates a 404. – Patrick Mevzek Feb 22 '18 at 19:04
  • It actually appears that the sitemap is both present, but also returning a 404 at the same time. I tested with curl -s 'https://www.houstonfertilityspecialists.com/sitemap.xml' --dump - which shows the sitemap content gets returned but that the HTTP status is 404. – Stephen Ostermiller Feb 22 '18 at 19:16
  • 1
    I've figured it out, I had the directory set to redirect to 404 in my nginx config. My bad! Thanks for all your help guys! – ag16920 Feb 22 '18 at 22:11

It is possible to have a sitemap.xml and robots.txt file in your root directory and return these files as 404 not found to bots.

I recommend that you search google for a header status page checker. Run your sitemap and robots URL through the checker and see what header status is being reported. If the header status is 404 then there is an issue. You need the status to be 200 (ok).

There can be many reasons why a page returns a 404 status despite existing. A lot of the time it has to do with conflicting code. The most likely reason why a page would return 404 despite already existing is that something in your .htaccess file is changing its status.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.