Tag Info

New answers tagged

2

Mediapartners-Google is the user agent that Google uses to crawl pages with AdSense ads on them. The crawling is likely related to Ads that are shown by the video. Remove the ads and Google will stop trying to crawl like this.


0

You can use tags in your headers (<head></head>) to prevent crawling from most search engines: <meta name="robots" content="noindex"> They also specify that if you only want to block google specifically then you can use this: <meta name="googlebot" content="noindex"> Google throws up the same: ...


5

It is redundant and unnecessary. /* Applies to all robots */ User-agent: * Disallow: /page/ Disallow: /ajax Disallow: *?back*


0

I'd say one of the most common practices hear would be adding noindex to the listing pages, apart from the first one, so as you said they will still get crawled but will not get added to the index. I think this is preferable to blocking them in robots.txt as Google is still free to crawl through the site and can still follow links on the pages. but nofollow ...


1

The way Google says to handle it makes me think they don't appreciate iframes that much. Google supports frames and iframes to the extent that it can. Frames can cause problems for search engines because they don't correspond to the conceptual model of the web. In this model, one page displays only one URL. Pages that use frames or iframes display ...


2

This error occur when your robots.txt file exists but is unreachable. Your site should return 200 HTTP status if the file exists or 404 if it doesn't, otherwise you would face such message from google. Before Googlebot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. If ...


3

Take a look at your Google Analytics data. First to confirm if its good traffic or bad: Check Your Audience -> Overview Report. Is your bounce rate high? Are your pages per visit very low? Is you time per session very low? If the answer is "Yes" to those questions then it has to be some sort of bot and etc. Then to find the source, pay attention to the ...


13

Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching. Say if you wanted to block all URLs that have a example any where in the URL, you can use a wild card entry * User-agent: * Disallow: /*example You can also use the dollar sign $ to specify that the URLs must end that way. So if you ...



Top 50 recent answers are included