Tag Info

Hot answers tagged

13

Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching. Say if you wanted to block all URLs that have a example any where in the URL, you can use a wild card entry * User-agent: * Disallow: /*example You can also use the dollar sign $ to specify that the URLs must end that way. So if you ...


5

It is redundant and unnecessary. /* Applies to all robots */ User-agent: * Disallow: /page/ Disallow: /ajax Disallow: *?back*


3

Take a look at your Google Analytics data. First to confirm if its good traffic or bad: Check Your Audience -> Overview Report. Is your bounce rate high? Are your pages per visit very low? Is you time per session very low? If the answer is "Yes" to those questions then it has to be some sort of bot and etc. Then to find the source, pay attention to the ...


2

Mediapartners-Google is the user agent that Google uses to crawl pages with AdSense ads on them. The crawling is likely related to Ads that are shown by the video. Remove the ads and Google will stop trying to crawl like this.


2

if there are no other characters after a pattern match like /somestring or /*/somestring exclude, if there are other characters, include. You can use $ to designate the end of the URL. Disallow: /somestring$ Disallow: /*/somestring$ If there are other characters in the URL after somestring, then they will not match and will therefore be allowed by ...


2

This error occur when your robots.txt file exists but is unreachable. Your site should return 200 HTTP status if the file exists or 404 if it doesn't, otherwise you would face such message from google. Before Googlebot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. If ...


1

The way Google says to handle it makes me think they don't appreciate iframes that much. Google supports frames and iframes to the extent that it can. Frames can cause problems for search engines because they don't correspond to the conceptual model of the web. In this model, one page displays only one URL. Pages that use frames or iframes display ...



Only top voted, non community-wiki answers of a minimum length are eligible