Tag Info

Hot answers tagged

20

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render it exactly as a browser would. If you block CSS or JS files, it could harm how well your website performs in the rankings. More info here: Updating our ...


4

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


3

The strategy for your robots.txt should always be answered by the question: which sections of my web must not be crawled by a robot and which sections may be crawled by a robot. Robots implement their own logic and have multiple purposes (not only Google has a crawler...) so if you're assuming that a robot get's somehow "distracted" by your CSS and JS ...


2

Robots.txt is a way of telling bots (robot agents) where they can go and cannot go. It is placed in the root of your web site as a standard to be found easily. It is really that simple. In your example: User-agent: Mediapartners-Google is not disallowed. The Disallow: with nothing following is an allow all (without restriction). User-agent: * is a ...


2

Honestly you are better off removing everything from your robots.txt. As far as I can see, all PHP files in Joomla contain the line defined('_JEXEC') or die; Which means if you load a PHP file directly in the browser all you get is a blank file, which search engines will ignore. (They shouldn't ever come across these anyway unless you linked them ...


2

A robots.txt can disallow crawling, not indexing. The next time Google tries to crawl your pages, it will probably check your robots.txt and notice that they are no longer allowed to crawl. This would stop Google from visiting your pages, but they don’t necessarily remove these pages from their index (nor does it mean that new pages won’t be indexed; they ...


2

Just use the URL removal tool found in Google webmaster tools to speed things up (after blocking with robots.txt)- this should take no longer then 24 hours (just enter / to remove an entire site) https://www.google.com/webmasters/tools/url-removal Your robots file should obviously contain one line telling all robots to not crawl your site: User-agent: * ...


1

if there are no other characters after a pattern match like /somestring or /*/somestring exclude, if there are other characters, include. You can use $ to designate the end of the URL. Disallow: /somestring$ Disallow: /*/somestring$ If there are other characters in the URL after somestring, then they will not match and will therefore be allowed by ...


1

It depends. All robots.txt does is tell a crawler what content you want—or don’t want—indexed. But that just communicates to the crawler. How the owner of the crawler handles the indexing—and cleanup—of the content depends on their internal processes. Meaning, they can take as long as they want to handle that aspect of the process. In practical terms I have ...


1

Robots.txt is a technological politeness. However, it is not a defined legal standard and legally, search engines and indexing engines do not have to follow it. Yes, big search engines like Google are designed to follow the standard; that's why you get "a description for this page is not available because of this site's robots.txt". However, many sites ...


1

In addition to closetnoc's answer... Should I remove /search from the Disallow: line? No. It is a good idea to block bots from crawling your search results (which I assume is what this is referring to). You don't normally want your search result pages from appearing in Google's search results pages! And Google doesn't want this either. Google wants to ...


1

Templates are used by the blogging or content management software (CMS) application and should not be accessible externally where as CSS is read by the browser and search engines and should be read. Having said that, I would not block either one but I also would not change any that are by the CMS. Search engines and bots do not care about your templates. ...


1

If you're using Apache web-server you could use an .htaccess configuration to white-list by User Agent and prevent genuine bots from reaching your 'tarpit': RewriteEngine On RewriteCond %{HTTP_USER_AGENT} SEMrush [NC] RewriteRule .* - [F,L]



Only top voted, non community-wiki answers of a minimum length are eligible