35 votes
Accepted

Is there a difference between an empty robots.txt and no robots.txt at all?

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of ...
user avatar
  • 15k
31 votes

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404 A 404 is probably preferable to blocking with robots.txt if you want these URLs dropped ...
user avatar
  • 41.3k
28 votes
Accepted

Can we use regex in robots.txt file to block URLs?

Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching. Say if you wanted to block all URLs that have a example any where in the URL, ...
user avatar
  • 6,077
24 votes
Accepted

Only allow Google and Bing bots to crawl a site

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed ...
user avatar
  • 21.5k
23 votes

Robots.txt - CSS allow or disallow

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render ...
user avatar
  • 6,077
21 votes

Will banning bots make a site harder to find on search engines?

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your ...
user avatar
19 votes
Accepted

Are web crawlers able to find a secondary robots.txt in a sub-directory?

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of ...
user avatar
16 votes

Is there a difference between an empty robots.txt and no robots.txt at all?

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler ...
user avatar
  • 398
14 votes
Accepted

Can a sitemap index contain other sitemap indexes?

I'm not sure if this changed over the years since this was asked; while in theory you can (as the first answer states), in practice Google at least will give you an error (as seen in their Webmaster ...
user avatar
  • 256
14 votes
Accepted

How can I use robots.txt to disallow subdomain only?

You can serve a different robots.txt file based on the subdomain through which the site has been accessed. One way of doing this on Apache is by internally rewriting the URL using mod_rewrite in ....
user avatar
  • 41.3k
14 votes

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

I'm going to answer the 2nd question. I was wondering if there is a quick way other than to manually remove these URLs from the google webmaster tools? https://developers.google.com/webmasters/...
user avatar
13 votes
Accepted

Is there a way to disallow crawling of only HTTPS in robots.txt?

There is no way to do it in robots.txt itself as served over HTTP. You could serve a different robots file entirely for secure HTTPS connections. Here is one of doing so using rewrite rules in ...
user avatar
13 votes
Accepted

Will banning bots make a site harder to find on search engines?

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get ...
user avatar
  • 690
13 votes

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

In short: you can't. Answer depends on the definition of "bots", but if a bot can crawl the e-mail, the bot can hit the link. A bot doesn't necessarily respect robots.txt, or mentioned html ...
user avatar
  • 131
11 votes
Accepted

What bots are really worth letting onto a site?

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search ...
user avatar
  • 32.5k
11 votes
Accepted

How to properly (dis)allow the archive.org bot? Did things change, if so when?

Update: As @KevinFegan notes in the comments, their documentation changed. The below part describes how the Internet Archive handled it in the past (at least in 2014). Their FAQ How can I have my ...
user avatar
  • 21.5k
11 votes

When you move a site via a 301 redirect should you setup a robots.txt disallowing robots to crawl the old address?

Your suspicions are right, with the exact reasoning that you mentioned. If you disallow robots from accessing the old site, they won't be able to see the 301 redirect.
user avatar
10 votes
Accepted

Serve a different robots.txt file for every site hosted in the same directory

I wouldn't count on all spiders being able to follow a redirect to get to a robots.txt file. See: Does Google respect a redirect header for robots.txt to a different file name? Assuming you are ...
user avatar
10 votes
Accepted

How to resolve Google "Indexed, though blocked by robots.txt"

Google isn't crawling your page, but it is indexing the URL. It isn't indexing the content of the page, just the URL itself, possibly along with anchor text of links that point to it. Google says: ...
user avatar
10 votes
Accepted

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

Incorrectly used meta tags Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two. <meta name="...
user avatar
9 votes
Accepted

robots.txt blocking specific files also block unnecessary URLs

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight ...
user avatar
  • 6,077
9 votes

Robots.txt for multiple domains with same website

You need to conditionally serve a different robots.txt file based on which domain/host has been accessed. On Apache you can do this in .htaccess: RewriteEngine On RewriteCond %{HTTP_HOST} ^(example1\....
user avatar
  • 41.3k
9 votes
Accepted

How does "Noindex:" in robots.txt work?

Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work. Here is what Google's John Mueller says about Noindex: ...
user avatar
9 votes

Are indexed Wordpress admin pages a security threat?

When a new bug is found, in this case in wordpress, the first thing hackers will do is try to find vulnerable sites that use wordpress. A good way of doing it is trying to find wp-admin pages on ...
user avatar
9 votes
Accepted

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

Google includes uncrawlable pages in the index when they are linked from other sites. That means that a link to the website like<a href="domain.com/en">[CHINESE] - Hey nighthawk</a> can ...
user avatar
9 votes
Accepted

Allow a folder and disallow all sub folders in robots.txt

User-agent: * Allow: /news/$ Disallow: /news/ Explanation: Google's robots.txt spec (https://developers.google.com/search/reference/robots_txt), which is more up to date than the "official" spec, ...
user avatar
9 votes
Accepted

What if robots.txt disallows itself?

Robots.txt directives don't apply to robots.txt itself. Crawlers may fetch robots.txt even if it disallows itself. It is actually very common for robots.txt to disallow itself. Many websites ...
user avatar
8 votes

Should I set NOINDEX header for my JS, CSS and image files?

(I work with Google's crawling & indexing teams) Let's start with the easy one: assuming you don't want these files used for indexing at all, you can just use the robots.txt file to disallow ...
user avatar
  • 7,303
8 votes

Is it possible to exclude Data-URIs (data:image/xxx) through robots.txt?

It's not supported at all since it is a not a HTTP call. It is embedded in the document and is essentially part of the HTML.
user avatar
  • 85.8k
8 votes

Are indexed Wordpress admin pages a security threat?

Security issue? Doubt it. It just makes your site more visible to people who likes to attack wordpress sites. Other than the visibility everything is the same, security through obscurity is not ...
user avatar
  • 189

Only top scored, non community-wiki answers of a minimum length are eligible