6

Almost 3 months ago I've put no-index tag to about 3,000 pages on my website and 301 redirect (which is permanent redirect) for almost 15,000 pages. (The all site is about 50,000 pages)

And yet as for today, all of these pages are still appearing in google index.

I've also updated the sitemap.

What can cause it? any advice?

example of no-index page: http://www.carz.co.il/review/8186

example of 301 redirect page:http://www.carz.co.il/gen/491/overview/2010/

All other pages are exactly the same (same meta tags, HTTP response etc..)

according to Google webmaster tools there are about 4000 pages crawled per day.

Improve this question
4
  • This question is a bit fuzzy. Assuming you have done everything correctly, it can take quite a while for Google to refetch all of your pages. The speed is highly dependent upon the sites freshness, popularity, etc. I would expect that some of your noindex and 301's have been found. If And yet as for today, all of these pages are still appearing in google index. is true, then there is something you have done incorrectly. We would need details in order to help you. Can you give us more details?? Example code? Robots.txt, .htaccess? Cheers!!
    – closetnoc
    Commented May 8, 2016 at 18:56
  • @closetnoc thanks, ive added more info to help understand the situation
    – YardenST
    Commented May 8, 2016 at 19:02
  • Huh! That is interesting. You seem to be using noindex correctly and your 301 appears to be working correctly too. With 4000 pages fetched each day, you should see something change, I would think anyway. Have you tried to use Googles Search Console Fetch as Google option to make sure Google can see these pages? I assume they can, but sometimes there are surprises. BTW- Nice looking site!
    – closetnoc
    Commented May 8, 2016 at 19:11
  • How are you checking the index status of these pages? A site: search can certainly return the source page of a redirect, which would not ordinarily show in a normal search. Not sure about the noindex pages though? Have these, at any time, been blocked with robots.txt?
    – MrWhite
    Commented May 8, 2016 at 22:08

1 Answer 1

Reset to default
4

I'm going to have to say google is too slow.

Your best bet is to go into Google search console, access your domain within it, then select the gear icon and then site settings, then for crawl rate, select "Limit Google's maximum crawl rate" and select the biggest crawl value by moving the slider all the way to the right. Then google will make up to 2+ requests per second. If you're lucky it might even allow you to choose 10 requests per second.

When you allow Google to make maximum requests, then it will scan your pages faster and find the no-index tags much faster than before and the waiting time for results will likely be lower.

In either case, I suggest waiting about a day for results. Here's the math that explains it.

  3000 ... You applied a no-index tag to 3000 pages.
+15000 ... You applied a 301 redirect to 15000 pages.
------
 18000 ... Total modified pages
 / 0.2 ... (divide by google's default scan rate of 0.2 pages per second)
------
 72000 ... 72,000 seconds waiting time which is 1,200 minutes or 20 hours.

Now if you crank up the setting to 10 requests per second and google does 10 requests per second, then here's your new math for the waiting time for results:

  3000 ... You applied a no-index tag to 3000 pages.
+15000 ... You applied a 301 redirect to 15000 pages.
------
 18000 ... Total modified pages
 /  10 ... (10 requests per second)
------
  1800 ... 1,800 seconds waiting time which is only 30 minutes.
Improve this answer
1
  • Thanks I changed it, let's wait 1-3 days to see if this affects something
    – YardenST
    Commented May 9, 2016 at 5:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.