1

I have made these unwanted pages noindex (using "X-Robots-Tag: noindex" HTTP header). Then I removed these pages using "Remove URLs" tool in Search console.

These pages have disappeared from Google for a while, but now they have come back (and I really don't want them there).

What am I missing?

P.S. Currently these pages show up as "Expired" in "Remove URLs" in Search console.

EDIT: These pages are not blocked in robots.txt

EDIT2: Here is how these pages look like in "URL Inspection" in Google Search console (however, contrary what is search console says, these pages ARE appearing in Google search results): enter image description here

Improve this question
10
  • 1
    Are they blocked in robots.txt? If so, Google can't see the noindex directive because it can't crawl them. What is an example URL path, and what are the contents of your robots.txt file?
    – Stephen Ostermiller ♦
    Nov 8, 2018 at 14:08
  • @StephenOstermiller I double-checked, these pages are not blocked in robots.txt. A couple example URLs: example.com/buy, example.com//resourses1/libs/loaders.css Contents of our robots.txt: User-agent: * Disallow: /demo/ Disallow: /support/ Disallow: /wp/ Disallow: /downloads/ Disallow: /uploads/ Sitemap: example.com/sitemap.xml
    – Sergey
    Nov 8, 2018 at 14:12
  • 1
    It's gotta be something else then, I'm not sure what. Hopefully somebody here will have some better ideas.
    – Stephen Ostermiller ♦
    Nov 8, 2018 at 14:13
  • Do you have a sitemap.xml? Those pages should not appear in the sitemap.xml either if you have one. Google gets "confused" if a page is in the sitemap and has a noindex flag.
    – Alexis Wilke
    Nov 8, 2018 at 20:56
  • What do you mean by "disappeared from Google for a while, but now they have come back", Are you saying that they are showing up in Google search results. Is it for site: based searches or real world searches?
    – Tony McCreath
    Nov 11, 2018 at 2:32

1 Answer 1

Reset to default
3

This is actually very common; pages removed using the Google Search Console tool, are not not meant to be permanent. Google even lists this under the Remove URLs tool section:

Remove URLs

To remove content permanently, you must remove or update the source page. Temporarily remove URLs that you own from search results.

The difficult part of what you are trying to do is de-index a page already indexed by Google. Once Google indexes a page, it doesn't like to let it go, even if you 404 or 410 the page, it will periodically over time come back and hit it just to see if the page has returned.

As long as you have the correct noindex tags set, you should be fine. Over time Google and other Search Engines will de-index the page -- but the key here is time. You can combined the noindex tags with the GSC Remove URLs tool to have a more immediate effect.

What I would do is: go under the robots.txt Tester section in the Google Search Console, and test one of the URLs you are trying to de-index from Google (such as your example.com/buy). Assuming there are no conflicts, move on, if there are conflicts, address them with your robots.txt file, as Google should be able to crawl the pages you are trying to de-index.

After confirming your robots.txt is fine, I would then move on to making sure that I have either one or both the meta (preferred) or header tag configured properly on the pages you want de-indexed:

Using the robots meta tag (preferred)

<meta name="robots" content="noindex, nofollow">

Using the X-Robots-Tag HTTP header

X-Robots-Tag: noindex, nofollow

More info on these can be found here.

If all this is done, you should eventually see the pages permanently removed from the Search Engine's Indexes.

Improve this answer
5
  • I think your answer is confusing. The only problem is most certainly just time. That is, the "Remove URLs" command was applied immediately but the index was not updated per se and if the pages do not get re-checked Google won't see the "noindex" and thus still view those pages as index-able. Until Google re-check the pages, nothing much will happen.
    – Alexis Wilke
    Nov 8, 2018 at 20:55
  • @AlexisWilke Sorry, not quite sure what part is confusing. Remove URLs is always applied immediately, but just because that is applied to the visual search results of the pages seen in say www.google.com, doesn't mean that Google doesn't still crawl those pages and won't see the "noindex" tag. Remove URLs is more of a filter on the search results returned nothing more.
    – devnull
    Nov 8, 2018 at 21:50
  • Note that the "nofollow" in the X-Robots-Tag (or meta) is probably not useful on that page. You should have rel="nofollow" on the other pages that link to this page (if any.) Anyway, my point in my first comment is that you should first answer the main question which is "why am I still seeing a page marked with 'noindex'?" then go on explaining various other possible issues and solutions.
    – Alexis Wilke
    Nov 8, 2018 at 22:02
  • Thanks @devnull, but I already did all that. And as I explained in the question, it didn't work. I still see these pages even though I used both "Remove URLs" and noindex.
    – Sergey
    Nov 11, 2018 at 14:22
  • @Sergey if all that is done, you just need to be patient and wait, the pages will eventually drop off. Unfortunately Google moves at its own pace and depending on the page, it could take some time before you see the changes reflected. I’m sorry I can’t be more specific than that.
    – devnull
    Nov 11, 2018 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.