Tag Info

New answers tagged

2

if there are no other characters after a pattern match like /somestring or /*/somestring exclude, if there are other characters, include. You can use $ to designate the end of the URL. Disallow: /somestring$ Disallow: /*/somestring$ If there are other characters in the URL after somestring, then they will not match and will therefore be allowed by ...


-1

In robots.txt file write this code. User-agent: * Disallow: /assets/ico/favicon.png so it can't access anything from Assets folder anymore


0

Apart from the overall use/lack thereof, of robots.txt in a well managed Joomla site, with "good" third party extensions — the only places that should contain CSS, JS or images are: /images /media /templates and of course their sub-directories. So, you could just remove those from robots.txt.


4

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


1

If you're using Apache web-server you could use an .htaccess configuration to white-list by User Agent and prevent genuine bots from reaching your 'tarpit': RewriteEngine On RewriteCond %{HTTP_USER_AGENT} SEMrush [NC] RewriteRule .* - [F,L]


2

A robots.txt can disallow crawling, not indexing. The next time Google tries to crawl your pages, it will probably check your robots.txt and notice that they are no longer allowed to crawl. This would stop Google from visiting your pages, but they don’t necessarily remove these pages from their index (nor does it mean that new pages won’t be indexed; they ...


2

Just use the URL removal tool found in Google webmaster tools to speed things up (after blocking with robots.txt)- this should take no longer then 24 hours (just enter / to remove an entire site) https://www.google.com/webmasters/tools/url-removal Your robots file should obviously contain one line telling all robots to not crawl your site: User-agent: * ...


1

It depends. All robots.txt does is tell a crawler what content you want—or don’t want—indexed. But that just communicates to the crawler. How the owner of the crawler handles the indexing—and cleanup—of the content depends on their internal processes. Meaning, they can take as long as they want to handle that aspect of the process. In practical terms I have ...


0

There are literally thousands of sites similar to these. Most of them are scraper sites that use Alexa data and possibly other data to build their pages. But some go against your site specifically. Woorank.com and builtwith.com are two sites among the many that access your server directly. I did a cursory search and did not see that either site will respect ...


1

Robots.txt is a technological politeness. However, it is not a defined legal standard and legally, search engines and indexing engines do not have to follow it. Yes, big search engines like Google are designed to follow the standard; that's why you get "a description for this page is not available because of this site's robots.txt". However, many sites ...


0

If you see your pages without errors when fetching as Google in WMT, then you're probably fine. But, in future, you might upgrade some content on your website, which will demand some scripts/css from some of blocked folders. Therefore, I think you might be better with allowing search engines to crawl all these folders containing CSS/JavaScript.


0

Robots.txt is a file that other websites, ISP's, and search engines use to "ask you" whats ok to visit. It allows you to whitelist or blacklist all or specific bots from areas of your realm. It's like a treaty. It's a promise. Good things keep the promise, bad things do not. As far as search: I agree that in the past it was not good practice to allow robots ...


1

In addition to closetnoc's answer... Should I remove /search from the Disallow: line? No. It is a good idea to block bots from crawling your search results (which I assume is what this is referring to). You don't normally want your search result pages from appearing in Google's search results pages! And Google doesn't want this either. Google wants to ...


2

Robots.txt is a way of telling bots (robot agents) where they can go and cannot go. It is placed in the root of your web site as a standard to be found easily. It is really that simple. In your example: User-agent: Mediapartners-Google is not disallowed. The Disallow: with nothing following is an allow all (without restriction). User-agent: * is a ...


2

Honestly you are better off removing everything from your robots.txt. As far as I can see, all PHP files in Joomla contain the line defined('_JEXEC') or die; Which means if you load a PHP file directly in the browser all you get is a blank file, which search engines will ignore. (They shouldn't ever come across these anyway unless you linked them ...


3

The strategy for your robots.txt should always be answered by the question: which sections of my web must not be crawled by a robot and which sections may be crawled by a robot. Robots implement their own logic and have multiple purposes (not only Google has a crawler...) so if you're assuming that a robot get's somehow "distracted" by your CSS and JS ...


20

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render it exactly as a browser would. If you block CSS or JS files, it could harm how well your website performs in the rankings. More info here: Updating our ...


1

Templates are used by the blogging or content management software (CMS) application and should not be accessible externally where as CSS is read by the browser and search engines and should be read. Having said that, I would not block either one but I also would not change any that are by the CMS. Search engines and bots do not care about your templates. ...



Top 50 recent answers are included