2

For 5 months I accidentally had thousands of pages from my site marked with a noindex meta tag, due to a bug in my code. Some of these came up as "soft 404" errors in GWT, but only a small portion.

I fixed this a couple of weeks ago and resubmitted my sitemaps, but it's still showing that only ~30,000 of my ~160,000 sitemap URLs are indexed. This is essentially the same as it was 2 weeks ago when I spotted the problem.

It used to much more along the lines of ~150,000 of 160,000 indexed.

Is there anything I can do to help get these pages back into Google's indexes?

Improve this question
6
  • Resubmitting sitemaps does nothing to get Google to look at your site again. Bugs like that can be very costly from an SEO standpoint.
    – Stephen Ostermiller ♦
    Jun 6, 2017 at 9:37
  • Thanks @StephenOstermiller. Do you have any recommendations for me? A lot of my site was unaffected by the bug and is still indexed fine, it's just a significant portion that isn't...
    – Codemonkey
    Jun 6, 2017 at 10:31
  • It will take months for Google to fully re-crawl 160,000 pages. There is nothing to do but wait, unfortunately.
    – Stephen Ostermiller ♦
    Jun 6, 2017 at 10:35
  • That's ok. You used to be able to tell google how aggressively to crawl, but that option seems to have disappeared as far as I can tell. They do crawl on average 6000 pages a day of mine, so fingers crossed.!
    – Codemonkey
    Jun 6, 2017 at 11:18
  • They don't crawl round robin though. They re-crawl pages based on their PageRank. Your popular pages will be crawled daily while your backwater pages may be crawled every couple months.
    – Stephen Ostermiller ♦
    Jun 6, 2017 at 11:32

1 Answer 1

Reset to default
1

Are you actively updating your sitemaps when changes occurr or do you upload them every once in a while? I would recommend updating your sitemaps regularly IF your website is frequently updated. This will make Google come visit your website and update/crawl your website and sitemaps more often.

Do you have a Google Search Console account?

I would suggest that you Fetch as Google (in the Crawl menu). For instance one of the pages that were affected or one or two levels above in the site hierarcy. After a minutes you get the option to "Request indexing". Do that and pick the option "Crawl this URL and its direct links".

Now wait for Google to index your website and check status after a week. Be patient.. these things can take time. Good luck @Codemonkey!

Improve this answer
3
  • Actively updating them. They're generated via PHP and as such spit out accurate lastmod values for every page.
    – Codemonkey
    Jun 6, 2017 at 10:28
  • I did test it earlier via "Fetch as Google", and clicked on request index/crawl with direct links as you said. Fingers crossed. I've read some more and I'm starting to think it's more that google's algorithms have changed since I last paid any real attention to the webmaster console, and it's just that my pages are too similar (despite actually being pretty different - there's not much unique text, just unique data).
    – Codemonkey
    Jun 6, 2017 at 10:30
  • If you've got access to your server log you might be able to see when/how often Google crawls your sitemap. Could be interesting to see the behaviour.
    – Kristian Svensson
    Jun 6, 2017 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.