32

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty robots....


31

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404 A 404 is probably preferable to blocking with robots.txt if you want these URLs dropped from the search engines (ie. Google). If you block crawling then the URL could still remain indexed. (Note that robots.txt primarily blocks crawling, not ...


27

Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching. Say if you wanted to block all URLs that have a example any where in the URL, you can use a wild card entry * User-agent: * Disallow: /*example You can also use the dollar sign $ to specify that the URLs must end that way. So if you ...


24

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...


23

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render it exactly as a browser would. If you block CSS or JS files, it could harm how well your website performs in the rankings. More info here: Updating our ...


21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


20

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your site you don’...


17

As indicated here, create a text file named robots.txt in the top-level directory of your web server. You can leave it empty, or add: User-agent: * Disallow: If you want robots to crawl everything. If not, then see the above link for more examples.


14

In the root of the directory of the subdomain website, add a file called robots.txt containing: User-agent: * Disallow: / This will tell web crawlers not to index the site at all. They do not have to obey, but the main ones will.


14

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


14

I'm going to answer the 2nd question. I was wondering if there is a quick way other than to manually remove these URLs from the google webmaster tools? https://developers.google.com/webmasters/hacked/docs/clean_site Google is explicitly states that removal through Google Search Console (the new name of webmaster tools) is the fastest. If the hacker ...


13

There is no way to do it in robots.txt itself as served over HTTP. You could serve a different robots file entirely for secure HTTPS connections. Here is one of doing so using rewrite rules in your .htaccess file: RewriteEngine On RewriteCond %{HTTPS} =on RewriteRule ^robots.txt$ robots-deny-all.txt [L] Where robots-deny-all.txt has the contents: ...


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


13

You can serve a different robots.txt file based on the subdomain through which the site has been accessed. One way of doing this on Apache is by internally rewriting the URL using mod_rewrite in .htaccess. Something like: RewriteEngine On RewriteCond %{HTTP_HOST} !^(www\.)?example\.com$ [NC] RewriteRule ^robots\.txt$ robots-disallow.txt [L] The above ...


12

I'm not sure if this changed over the years since this was asked; while in theory you can (as the first answer states), in practice Google at least will give you an error (as seen in their Webmaster Tools): Incorrect Sitemap index format: Nested Sitemap indexes The Google help page further states: A sitemap index file can't list other sitemap index ...


12

No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed. However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...


12

In the Disallow field you specify the beginning of URL paths of URLs that should be blocked. So if you have Disallow: /, it blocks everything, as every URL path starts with /. If you have Disallow: /a, it blocks all URLs whose paths begin with /a. That could be /a.html, /a/b/c/hello, or /about. In the same sense, if you have Disallow: /search, it blocks ...


11

You don't want the page to appear in the SERPs at all... Don't disallow in robots.txt. Add a noindex meta tag (or X-Robots-Tag HTTP header) to your pages instead. As j0k suggests, your pages could be found somehow. Stats reports, directory listings, etc... Disallowing in robots.txt prevents the page from being crawled, but could still be indexed and could ...


11

It takes some time for Googlebot to crawl your site. This amount of time can vary depending on the number of links to your site, among other factors. After your site is crawled, the URLs in the robot.txt file will be updated.


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


11

Your suspicions are right, with the exact reasoning that you mentioned. If you disallow robots from accessing the old site, they won't be able to see the 301 redirect.


10

You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed. User-agent: * Disallow: /templates_c But to answer your question, according to Google: At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] ...


10

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


10

Google isn't crawling your page, but it is indexing the URL. It isn't indexing the content of the page, just the URL itself, possibly along with anchor text of links that point to it. Google says: A robotted page can still be indexed if linked to from from other sites While Google won't crawl or index the content blocked by robots.txt, we might still ...


9

I wouldn't count on all spiders being able to follow a redirect to get to a robots.txt file. See: Does Google respect a redirect header for robots.txt to a different file name? Assuming you are hosted on an Apache server, you could use mod_rewrite from your .htaccess file to to serve the correct file for the correct domain: RewriteEngine On RewriteCond %{...


9

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located below: ...


9

Update: As @KevinFegan notes in the comments, their documentation changed. The below part describes how the Internet Archive handled it in the past (at least in 2014). Their FAQ How can I have my site's pages excluded from the Wayback Machine? refers to Removing Documents From the Wayback Machine, which documents that their bot is called ia_archiver. So ...


9

You need to conditionally serve a different robots.txt file based on which domain/host has been accessed. On Apache you can do this in .htaccess: RewriteEngine On RewriteCond %{HTTP_HOST} ^(example1\.com|example2\.com)$ RewriteRule ^robots\.txt$ /robots-disallow.txt [L] This is specific to your example, where it will serve a "disallow" robots.txt (robots-...


9

Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work. Here is what Google's John Mueller says about Noindex: in robots.txt: We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don'...


9

When a new bug is found, in this case in wordpress, the first thing hackers will do is try to find vulnerable sites that use wordpress. A good way of doing it is trying to find wp-admin pages on google. Maybe will even use automated tools to find-and-exploit sites based on that. Avoiding that would be the main security reason for doing it. If you have no ...


Only top voted, non community-wiki answers of a minimum length are eligible