I've been having some issues removing some content from Google's search results. I have some word documents and pdf files in public directories that I have since password protected. When you try to access any of the documents you are prompted for a user name and password. The documents have also been completely removed from the server.

I did have a disallow on the directories in my robots.txt but this didn't seem to do the trick either.

I've also been adding the urls to the temporary removal tool but they just expire and appear back in the results.

Any suggestions on how to completely remove the content that used to be in these directories from Googles search results? The urls currently return a 401. Would a 410 be more effective?

  • 1
    Are they still blocked in robots.txt? Googlebot will never see that they are removed if it can't crawl them. Commented Aug 21, 2017 at 14:52
  • No, I removed the block from robots.txt. Would that be the case for a 401 as well? Since Google can't access the document it will keep it in it's index?
    – brandozz
    Commented Aug 21, 2017 at 15:43
  • I've removed the password protection and I'm letting everything 404 for the time being. I'm trying to add a 410 response code to those directories but it's not working.
    – brandozz
    Commented Aug 21, 2017 at 16:26
  • How long have you waited? It may take Google a month or more to re-crawl everything. Once it crawls each document, it should remove it from the index within 24 hours after getting a 404 error. Commented Aug 21, 2017 at 16:27
  • Well, I've had the directories password protected for well over a month. I just now removed that password protection and let the urls 404.
    – brandozz
    Commented Aug 21, 2017 at 16:28

1 Answer 1


Google have two tools to remove content from search.


This will remove the content for temporary(90 days), and it may index the content again as you said in your question.

And this is new tool, I don't know when they launch it, but it work's really great for me.


This tool have only two rules, if the content return 4xx error, simply remove it from search results ASAP, no matter you own the site or not.

enter image description here

It's damn good, I successfully remove search results from github site as well, which return 404 error.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.