Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to submit my a merchant feed to Google Shopping Search (aka Froogle) but every now and then I get an email saying that the URLs in the feed are "Roboted Content".

The URLs in the feed are tracking ones which means that they redirect to the real product pages. Both directories aren't in my robots.txt. Yet Google keeps telling me that they are blocked for its crawlers.

How can I find out exactly what the offending URL is?

share|improve this question
up vote 1 down vote accepted

Usually when you post a feed to your merchant account in Google Base, it will give you lines where the errors are. Take a look at that.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.