2

I am developing an AngualrJS application and since it is not easy to optimize it for search engines, I put it public and let Google index the page by registering a "Property" in their "Search Console" to see the results and to improve the page. The page has been successfully indexed and Google does display search results.

Now I wanted to remove the page from public again to finish the project and therefore I wanted that Google to remove it from the search results. However, this seems not to be easy.

First, I adjusted the robots.txt to deny any indexing. Even after Google checked the new robots.txt, the search results were still visible.

Then I tried to setup temporarily URL removals in their search console. They were accepted, but the URLs were not removed.

Then I setup the site such that Googlebot does no more get any access to the server - it gets 404 for every URL requested. This has been the case for almost 4 weeks but Google still displays search results.

Two weeks ago, I deleted the "Property" from the search console of my google account and deleted the webroot validation HTML file of Google (googlexxxxxxxxxxxxxxx.html - well, Googlebot will get 404 anyway) but they still display search results...

How can I get Google to completely and quickly remove my site from all search results?

Improve this question
5
  • "display search results" - are you doing a site: search or just using a normal search phrase?
    – MrWhite
    Apr 7, 2016 at 8:35
  • I am doing a site: search. Is this the problem?
    – hans
    Apr 7, 2016 at 9:12
  • Well, it's just that a site: search is likely to return results that would not be returned in an ordinary web search. (A site: search is "special" in this respect.) So, the average user might not be able to find your pages. However, you still probably don't want those pages returned in a site: search either - but if they are blocked anyway then that may be enough. A 410 is likely to make Google drop the pages quicker (but don't block with robots.txt). Btw, whether pages return in the SERPs has nothing to do with the property being present in the search console.
    – MrWhite
    Apr 7, 2016 at 9:46
  • Search console has a remove URL feature which is what I assume he used to "delete property in search console"
    – Stephen Ostermiller ♦
    Apr 7, 2016 at 12:10
  • I tried to use the remove URL feature, however, the search results were still visible with site: searchs. Then I deleted the whole 'Property' (I mean the whole domain, for which we must upload a validation file - it is called Property in german; it is probably different in english)
    – hans
    Apr 7, 2016 at 12:16

2 Answers 2

Reset to default
2

Google indexes many sites that don't have a search console account. Deleting the property from search console does not effect whether Googlebot will crawl your site and include it in the search index.

If you disallow crawling via robots.txt Googlebot will never be able to see that those pages are 404. Google needs to be able to crawl to see that you have removed the content.

After you have done the temporary removal, allow crawling again to let Googlebot see the 404 errors. Google should crawl all of your pages within a couple weeks. Pages get removed after 24 hours when Googlebot finds a 404 error. If Google can't crawl and see the 404 errors, it may include URLs in the search index indefinitely.

Instead of serving 404 errors just to Googlebot to attempt to get the pages removed, you should tell Google explicitly that you do not want the pages indexed. The way to do that is to include robots meta noindex tags in head section of every page:

<meta name="robots" content="noindex">

Once you put in those tags, you can safely let Googlebot crawl your entire site and non of it will appear in the search engine.

To speed up the no-index process for all the URLs on your site, Google has a Noindex: directive in robots.txt. Change your robots.txt file to:

User-Agent: *
Noindex: /

Even if you do that, you will need to add the noindex meta tags because Google says the Noindex robots.txt is experimental and may go away at any time.

Improve this answer
7
  • Is it also possible that google - if googlebot gets a 404 during crawling - check with other useragent than "googlebot", if it also gets 404, and if not, it will keep the results?
    – hans
    Apr 7, 2016 at 11:39
  • Are you serving 404 errors just to Googlebot?
    – Stephen Ostermiller ♦
    Apr 7, 2016 at 11:40
  • Yes - I wanted to remove the page from google results, not from the public itself (testing...)
    – hans
    Apr 7, 2016 at 11:41
  • I've edited my answer to include a better solution for you.
    – Stephen Ostermiller ♦
    Apr 7, 2016 at 11:46
  • OK, thank you. The problem is that it will take weeks until all pages are removed because it will only be removed after reindexing each url. In fact, my page was indexed in lets say 2 days, in total 7000 entries. However, some days later, only about 50 pages per day were recrawled. I hoped for a way to remove all together...
    – hans
    Apr 7, 2016 at 11:54
-3

You should remove URL in Search console few times and maybe Google will see your request))

Improve this answer
1
  • Do you have any experience or references that would indicate the it takes multiple tries? Without something to back up your statement, this answer looks like pure conjecture.
    – Stephen Ostermiller ♦
    Apr 7, 2016 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.