3

I have a website and we recently changed content management systems. With the new system came new URLs/hierarchy. Google Webmaster Tools doesn't have any issues with the site and I get no crawler errors. But AdSense seems to still be looking for URLs on my old site and I get tons of "Page not found" errors.

What can be done to get Adsense to crawl the new sitemap?

This is on the crawler errors page. It says "Blocked URLs" and then to the side of the URL it lists "Page not found."

It explains: "Our crawler was unable to access the pages listed in the ’Blocked URLs’ column to determine the content and display ads. Click the ’plus’ icon beside each domain to see an expanded list of affected URLs. When our crawler can’t access your content we won’t show ads, resulting in lower revenue and coverage. Follow the link in the ’Error’ column to learn how to correct these errors."

I just want it to forget these pages exist, because they don't anymore.

  • Are you talking about 404 errors caused by the AdSense bot "Mediapartners-Google" that you are seeing in your server logs? – Stephen Ostermiller Dec 11 '14 at 14:57
  • This is on the crawler errors page. It says "Blocked URLs" and then to the side of the URL it lists "Page not found." It explains: "Our crawler was unable to access the pages listed in the ’Blocked URLs’ column to determine the content and display ads. Click the ’plus’ icon beside each domain to see an expanded list of affected URLs. When our crawler can’t access your content we won’t show ads, resulting in lower revenue and coverage. Follow the link in the ’Error’ column to learn how to correct these errors." I just want it to forget these pages exist, because they don't anymore. – elfshadowreaper Dec 11 '14 at 21:56
0

AdSense uses a crawler called Mediapartners-Google. It doesn't crawl your entire site using the sitemap. Rather, when a user requests a page with ads on it, it crawls that URL on the fly.

If you are no longer serving pages with ads on those URLs, the AdSense bot will stop crawling those URLs without any intervention from you.

My guess is that when you took the pages down, there were a few users that accessed cached versions of the pages. That triggered the crawl and the error. I would expect the errors to disappear within a couple weeks.

  • Thanks! Do crawler errors affect revenue? Because when I click on "Revenue optimization" that's where it tells me about the crawler issues. – elfshadowreaper Dec 12 '14 at 16:31
  • 1
    It won't show ads on the pages that it can't crawl. But if you don't have pages there anymore, it shouldn't be something to worry about. – Stephen Ostermiller Dec 12 '14 at 16:32

protected by Community Aug 25 '18 at 3:46

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.