3

I have a website and we recently changed content management systems. With the new system came new URLs/hierarchy. Google Webmaster Tools doesn't have any issues with the site and I get no crawler errors. But AdSense seems to still be looking for URLs on my old site and I get tons of "Page not found" errors.

What can be done to get Adsense to crawl the new sitemap?

This is on the crawler errors page. It says "Blocked URLs" and then to the side of the URL it lists "Page not found."

It explains: "Our crawler was unable to access the pages listed in the ’Blocked URLs’ column to determine the content and display ads. Click the ’plus’ icon beside each domain to see an expanded list of affected URLs. When our crawler can’t access your content we won’t show ads, resulting in lower revenue and coverage. Follow the link in the ’Error’ column to learn how to correct these errors."

I just want it to forget these pages exist, because they don't anymore.

Improve this question
2
  • Are you talking about 404 errors caused by the AdSense bot "Mediapartners-Google" that you are seeing in your server logs?
    – Stephen Ostermiller ♦
    Dec 11, 2014 at 14:57
  • This is on the crawler errors page. It says "Blocked URLs" and then to the side of the URL it lists "Page not found." It explains: "Our crawler was unable to access the pages listed in the ’Blocked URLs’ column to determine the content and display ads. Click the ’plus’ icon beside each domain to see an expanded list of affected URLs. When our crawler can’t access your content we won’t show ads, resulting in lower revenue and coverage. Follow the link in the ’Error’ column to learn how to correct these errors." I just want it to forget these pages exist, because they don't anymore.
    – elfshadowreaper
    Dec 11, 2014 at 21:56

1 Answer 1

Reset to default
0

AdSense uses a crawler called Mediapartners-Google. It doesn't crawl your entire site using the sitemap. Rather, when a user requests a page with ads on it, it crawls that URL on the fly.

If you are no longer serving pages with ads on those URLs, the AdSense bot will stop crawling those URLs without any intervention from you.

My guess is that when you took the pages down, there were a few users that accessed cached versions of the pages. That triggered the crawl and the error. I would expect the errors to disappear within a couple weeks.

Improve this answer
3
  • Thanks! Do crawler errors affect revenue? Because when I click on "Revenue optimization" that's where it tells me about the crawler issues.
    – elfshadowreaper
    Dec 12, 2014 at 16:31
  • 1
    It won't show ads on the pages that it can't crawl. But if you don't have pages there anymore, it shouldn't be something to worry about.
    – Stephen Ostermiller ♦
    Dec 12, 2014 at 16:32
  • They never disappear, or takes years, i had them for many many many months.
    – Michael Rogers
    Dec 9, 2020 at 18:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.