Tag Info

Hot answers tagged

13

Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching. Say if you wanted to block all URLs that have a example any where in the URL, you can use a wild card entry * User-agent: * Disallow: /*example You can also use the dollar sign $ to specify that the URLs must end that way. So if you ...


3

Take a look at your Google Analytics data. First to confirm if its good traffic or bad: Check Your Audience -> Overview Report. Is your bounce rate high? Are your pages per visit very low? Is you time per session very low? If the answer is "Yes" to those questions then it has to be some sort of bot and etc. Then to find the source, pay attention to the ...


3

It is redundant and unnecessary. /* Applies to all robots */ User-agent: * Disallow: /page/ Disallow: /ajax Disallow: *?back*


2

if there are no other characters after a pattern match like /somestring or /*/somestring exclude, if there are other characters, include. You can use $ to designate the end of the URL. Disallow: /somestring$ Disallow: /*/somestring$ If there are other characters in the URL after somestring, then they will not match and will therefore be allowed by ...


2

This error occur when your robots.txt file exists but is unreachable. Your site should return 200 HTTP status if the file exists or 404 if it doesn't, otherwise you would face such message from google. Before Googlebot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. If ...


1

The way Google says to handle it makes me think they don't appreciate iframes that much. Google supports frames and iframes to the extent that it can. Frames can cause problems for search engines because they don't correspond to the conceptual model of the web. In this model, one page displays only one URL. Pages that use frames or iframes display ...



Only top voted, non community-wiki answers of a minimum length are eligible