1

We have an app we are migrating. The old website had deeply nested robots.txt files at some point and then the old developers started handling those request with redirects to the root page. So:

a request to http:://example.com/foo/bar/robots.txt is now a 302 to http://example.com

I assume that because the old devs are doing a 302 instead of a 301, it continues to index that old path. Will a 301 to http://example.com/robots.txt tell google to abandon the old requests? What other things can I do to encourage these old paths to disappear from search engine indexing?

1

Remove the robots.txt files and have the urls simply return 410. It tells google the resource is gone and is never coming back.

No need to get complicated with robots tags.

1

If you want google to stop indexing a site, page or directory and you don't want to redirect to a new site, you have a few options.

  1. Delete all content
  2. Add <meta name="robots" content="noindex,nofollow"> flags to your content.
  3. Use webmaster tools to remove URLs from Google's index
  4. Add a server-level password to your directories you no longer want indexed, via .htaccess file on Apache.
0

robots.txt does not stop Google from indexing content. It only tells them not to crawl it. Content blocked by robots.txt can still be listed in Google's search results.

To actually prevent Google from indexing content you need to use the x-robots-tag HTTP header. You can do this in code or in your .htaccess file. But any pages you want to block from indexing should output the following HTTP header:

X-Robots-Tag: noindex
  • since I'm not using Apache, I'll be injecting the headers and rendering the file myself. should the old path location render the robots.txt file as a 200 with the X-Robots-Tag header? – Jed Schneider Nov 3 '14 at 14:46
  • Are you asking if you should use x-robots-tag with your robots.txt file? – John Conde Nov 3 '14 at 14:47
  • User-Agent: * (newline here)Disallow: /* – Martijn Nov 3 '14 at 14:54
  • @JohnConde correct, the issue is that that nested robots file does not exist. it should use the top level robots file at root, correct? but google keeps requesting this deeply nested robots.txt file because of decisions made on the old site. I no longer want google to look for this file there, but I need to respond in some way to that http request. I'm unsure of the implications between doing a 302, a 301, a 404, or some other code in regards to maintaining good seo with google. does that explain it better? – Jed Schneider Nov 3 '14 at 15:00
  • @Martijn I actually want google to index the content at /foo/bar but I dont want it to look for a robots.txt file in that directory/resource path. – Jed Schneider Nov 3 '14 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.