1

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special YouTubers. All of that works fine. And all of that is not accessible from the outside (you'd need a username and a password, which only I have).

On my start page, there are a lot of links to videos that are in my subscriptions, but aren't downloaded yet. When I click one of these links, I get that video included with the original YouTube embed feature. All of that works fine.

But now my problem: a few minutes ago I watched a video with that embed-thing and I just saw this in my Apache log:

66.249.89.90 - - [20/Dec/2014:21:40:52 +0100] "GET my_youtube_clone HTTP/1.1" 200 2780 "-" "Mediapartners-Google"

I already have all bots blocked via robots.txt, so obviously Google is using the YouTube referrers to crawl the pages and while it is doing that, it is ignoring the robots.txt.

Google didn't get anything useful from that, as I said, you'd need a password for that, but I am quite annoyed by Google ignoring the robots.txt and using the referrers of YouTube as crawl URL sources.

Is there any way to completely stop that?

  • 5
    So you're violating Google terms of service and then mad at them for violating yours? – John Conde Dec 20 '14 at 22:37
  • Just to be sure, could you include your robots.txt content? – unor Dec 21 '14 at 15:36
5

Mediapartners-Google is the user agent that Google uses to crawl pages with AdSense ads on them. The crawling is likely related to Ads that are shown by the video.

Remove the ads and Google will stop trying to crawl like this.

0

You can use tags in your headers (<head></head>) to prevent crawling from most search engines:

<meta name="robots" content="noindex">

They also specify that if you only want to block google specifically then you can use this:

<meta name="googlebot" content="noindex">

Google throws up the same: https://support.google.com/webmasters/answer/93710?hl=en

  • 1
    This disallows indexing, not crawling. – unor Dec 21 '14 at 15:32
  • In which case blocking certain user agents should work as well. Alternatively adding a robots.txt containing information regarding permitted or denied urls would work too. – Mike Dec 25 '14 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.