Hot answers tagged

30

Lack of a robots.txt file will not be harmful. From the robotstxt.org website: To allow all robots complete access User-agent: * Disallow: (or just create an empty "/robots.txt" file, or don't use one at all) However, even if you are not specifying anything in your robots.txt file, it is a good way of informing search engines of the ...


24

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty ...


21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


21

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render it exactly as a browser would. If you block CSS or JS files, it could harm how well your website performs in the rankings. More info here: Updating our ...


21

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...


19

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder. Here are all the ones I know of: favicon.ico Gives your pages an icon in tabs, bookmarks, etc. robots.txt Useful ...


19

It seems that Google deliberately includes URLs disallowed in robots.txt in their index if there are links to those URLs from other pages they've crawled. To quote their Webmaster Tools help pages: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a ...


13

There are 2 main ways to prevent search engines from indexing specific pages: A Robots.txt file for your domain. The Meta Robots tag on each page. Robots.txt should be your first stop for URL patterns that match several files. You can see the syntax here and more detailed here. The robots.txt file must be placed in the root folder of your domain, i.e. at ...


13

I would strongly recommend registering your site with Google Search Console (previously Google Webmaster Tools). There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where ...


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


12

Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching. Say if you wanted to block all URLs that have a example any where in the URL, you can use a wild card entry * User-agent: * Disallow: /*example You can also use the dollar sign $ to specify that the URLs must end that way. So if you ...


11

You don't want the page to appear in the SERPs at all... Don't disallow in robots.txt. Add a noindex meta tag (or X-Robots-Tag HTTP header) to your pages instead. As j0k suggests, your pages could be found somehow. Stats reports, directory listings, etc... Disallowing in robots.txt prevents the page from being crawled, but could still be indexed and could ...


11

It takes some time for Googlebot to crawl your site. This amount of time can vary depending on the number of links to your site, among other factors. After your site is crawled, the URLs in the robot.txt file will be updated.


11

As indicated here, create a text file named robots.txt in the top-level directory of your web server. You can leave it empty, or add: User-agent: * Disallow: If you want robots to crawl everything. If not, then see the above link for more examples.


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


11

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...


10

Hard to say for sure why you aren't indexed yet but: 1) Number of visitors has no bearing whatsoever on your indexing or rankings. Google doesn't know this information and, even if it they did, it really offers nothing in terms of relevance of any page for search. 2) 20 backlinks is hardly a lot. Even then, unless Google knows about those backlinks they ...


10

You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.


10

Maybe someone didn't want to pay for spider traffic? Regardless, you are reading it correctly: http://www.robotstxt.org/robotstxt.html Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say ...


10

You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed. User-agent: * Disallow: /templates_c But to answer your question, according to Google: At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the ...


10

No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed. However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...


9

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located ...


9

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


9

When a new bug is found, in this case in wordpress, the first thing hackers will do is try to find vulnerable sites that use wordpress. A good way of doing it is trying to find wp-admin pages on google. Maybe will even use automated tools to find-and-exploit sites based on that. Avoiding that would be the main security reason for doing it. If you have no ...


8

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.


8

You don't need the trailing slash to block those directories. This will work: Disallow: /first_url/* Disallow: /second_url/* This Google help page covers this quite well.


8

Yes, you can have a sitemap index containing references to other sitemap indexes [source], as long as they're all on the same domain. Each individual sitemap index can include up to 1,000 references to another <sitemap>. If you need to reference more than that, create multiple sitemap indexes and submit each one. Each individual sitemap can include ...


8

It's very simple to spoof a User-Agent header and pretend to be a Google Bot. But much harder to fake the I.P. from which the request is coming. Check that the I.Ps making these request are owned by Google.



Only top voted, non community-wiki answers of a minimum length are eligible