I'm trying to submit my a merchant feed to Google Shopping Search (aka Froogle) but every now and then I get an email saying that the URLs in the feed are "Roboted Content".

The URLs in the feed are tracking ones which means that they redirect to the real product pages. Both directories aren't in my robots.txt. Yet Google keeps telling me that they are blocked for its crawlers.

How can I find out exactly what the offending URL is?


Usually when you post a feed to your merchant account in Google Base, it will give you lines where the errors are. Take a look at that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.