30

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty robots....


30

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404 A 404 is probably preferable to blocking with robots.txt if you want these URLs dropped from the search engines (ie. Google). If you block crawling then the URL could still remain indexed. (Note that robots.txt primarily blocks crawling, not ...


27

Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching. Say if you wanted to block all URLs that have a example any where in the URL, you can use a wild card entry * User-agent: * Disallow: /*example You can also use the dollar sign $ to specify that the URLs must end that way. So if you ...


25

It seems that Google deliberately includes URLs disallowed in robots.txt in their index if there are links to those URLs from other pages they've crawled. To quote their Webmaster Tools help pages: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a ...


24

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...


23

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render it exactly as a browser would. If you block CSS or JS files, it could harm how well your website performs in the rankings. More info here: Updating our ...


21

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder. Here are all the ones I know of: favicon.ico Gives your pages an icon in tabs, bookmarks, etc. robots.txt Useful if ...


21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


18

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...


17

As indicated here, create a text file named robots.txt in the top-level directory of your web server. You can leave it empty, or add: User-agent: * Disallow: If you want robots to crawl everything. If not, then see the above link for more examples.


14

In the root of the directory of the subdomain website, add a file called robots.txt containing: User-agent: * Disallow: / This will tell web crawlers not to index the site at all. They do not have to obey, but the main ones will.


14

I'm going to answer the 2nd question. I was wondering if there is a quick way other than to manually remove these URLs from the google webmaster tools? https://developers.google.com/webmasters/hacked/docs/clean_site Google is explicitly states that removal through Google Search Console (the new name of webmaster tools) is the fastest. If the hacker ...


13

There is no way to do it in robots.txt itself as served over HTTP. You could serve a different robots file entirely for secure HTTPS connections. Here is one of doing so using rewrite rules in your .htaccess file: RewriteEngine On RewriteCond %{HTTPS} =on RewriteRule ^robots.txt$ robots-deny-all.txt [L] Where robots-deny-all.txt has the contents: ...


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


13

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


12

No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed. However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...


11

I'm not sure if this changed over the years since this was asked; while in theory you can (as the first answer states), in practice Google at least will give you an error (as seen in their Webmaster Tools): Incorrect Sitemap index format: Nested Sitemap indexes The Google help page further states: A sitemap index file can't list other sitemap index ...


11

You don't want the page to appear in the SERPs at all... Don't disallow in robots.txt. Add a noindex meta tag (or X-Robots-Tag HTTP header) to your pages instead. As j0k suggests, your pages could be found somehow. Stats reports, directory listings, etc... Disallowing in robots.txt prevents the page from being crawled, but could still be indexed and could ...


11

It takes some time for Googlebot to crawl your site. This amount of time can vary depending on the number of links to your site, among other factors. After your site is crawled, the URLs in the robot.txt file will be updated.


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


11

You can serve a different robots.txt file based on the subdomain through which the site has been accessed. One way of doing this on Apache is by internally rewriting the URL using mod_rewrite in .htaccess. Something like: RewriteEngine On RewriteCond %{HTTP_HOST} !^(www\.)?example\.com$ [NC] RewriteRule ^robots\.txt$ robots-disallow.txt [L] The above ...


11

Your suspicions are right, with the exact reasoning that you mentioned. If you disallow robots from accessing the old site, they won't be able to see the 301 redirect.


10

Maybe someone didn't want to pay for spider traffic? Regardless, you are reading it correctly: http://www.robotstxt.org/robotstxt.html Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://...


10

You can't make them re-download your robots.txt when you want them to. Google will re-crawl it and use the new data whenever they feel it is appropriate for your site. They tend to crawl it regularly so I wouldn't expect it to take long for your updated file to be found and your pages re-crawled and re-indexed. Keep in mind that it may take some time after ...


10

You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed. User-agent: * Disallow: /templates_c But to answer your question, according to Google: At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] ...


10

In the Disallow field you specify the beginning of URL paths of URLs that should be blocked. So if you have Disallow: /, it blocks everything, as every URL path starts with /. If you have Disallow: /a, it blocks all URLs whose paths begin with /a. That could be /a.html, /a/b/c/hello, or /about. In the same sense, if you have Disallow: /search, it blocks ...


10

Google isn't crawling your page, but it is indexing the URL. It isn't indexing the content of the page, just the URL itself, possibly along with anchor text of links that point to it. Google says: A robotted page can still be indexed if linked to from from other sites While Google won't crawl or index the content blocked by robots.txt, we might still ...


9

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


9

I wouldn't count on all spiders being able to follow a redirect to get to a robots.txt file. See: Does Google respect a redirect header for robots.txt to a different file name? Assuming you are hosted on an Apache server, you could use mod_rewrite from your .htaccess file to to serve the correct file for the correct domain: RewriteEngine On RewriteCond %{...


9

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located below: ...


Only top voted, non community-wiki answers of a minimum length are eligible