Tag Info

Hot answers tagged

21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


21

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...


13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


9

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located ...


7

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


6

I wouldn't count on all spiders being able to follow a redirect to get to a robots.txt file. See: Does Google respect a redirect header for robots.txt to a different file name? Assuming you are hosted on an Apache server, you could use mod_rewrite from your .htaccess file to to serve the correct file for the correct domain: RewriteEngine On RewriteCond ...


6

You have nothing to worry about. You can use display: none; to switch menus. Search engines are much better at understanding JS and CSS. As long as you are not intentionally trying to manipulate things to get a better ranking. Using display: none; to hide big blocks of text will get you penalized. So if you are only using to hide your desktop menu on ...


5

wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string. wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent ...


5

Most bots don't accept cookies (including Googlebot), however, some bots do. You send a Set-Cookie header in the response, but the bot does not send back a Cookie header in subsequent requests - so the cookie is effectively lost. Whether Google is monitoring whether the site is setting cookies, we don't know for sure, but I'd wager they probably are. ...


5

Google only cares that they see the same thing your users see. If you're only serving up different content to these bad crawlers, and your users are getting your normal content and Google is getting that same content, you're ok.


5

Google expects differences between mobile and desktop sites. Even major differences, including differences in link structure, are not a problem. Google crawls the web with different Googlebot user agents for mobile. As long as your server shows that version of Googlebot the same thing that your actual mobile users see, you don't have any penalty risk. ...


4

The current GoogleBot Smartphone agent, as tested with the 'Fetch as Google' Tool is essentially a fake iPhone using a headless Webkit Safari 6.0 Engine, running on a Linux x86_64 desktop machine. The default non-responsive viewport width is that of an iPhone at 980px. With a viewport <meta name="viewport" content="width=device-width, ...


4

While this should work, it has a few potential drawbacks: Every crawler has to do two HTTP requests: one to discover the redirect, and another one to actually fetch the file. Some crawlers might not handle the 301 response for robots.txt correctly; there's nothing in the original robots.txt specification that says anything about redirects, so presumably ...


4

To block all "blank" User-Agents or User-Agents consisting of a hyphen, you could use the following in your .htaccess RewriteCond %{HTTP_USER_AGENT} ^-?$ RewriteRule ^ - [F]


4

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


3

I think that you may be interested in this article: Design AJAX-powered sites for accessibility. It's a very basic article, but it should give you some pointers about what to do and what not to do. AJAX should be used to improve the experience on specific pages and elements of the site, but not to build a whole site. For the user perspective, it's a bad ...


3

Stephen Ostermiller’s answer explains how to configure it in Google Webmaster Tools (which answers your question). If you care about other search engines, too, or in addition, you could use the Canonical link type. You could specify the canonical link in a HTTP header, or you could specify it in the HTML: On ...


3

Google Webmaster Tools does have the ability to tell Googlebot to ignore parameters. You can tell Googlebot to ignore the parameter "MobileOptOut" anywhere it appears on your site. To do so: Navigate in Webmaster Tools to Crawl -> URL Parameters. Click the Configure URL parameters » link. Click the Add Parameter button. Put in the name: "MobileOptOut". ...


3

In order to ensure that Google has crawled and indexed your content, you can just do a search with the keyword you'd like to be indexed. E.g., you've a heading as follows: <h1>The h1 heading on my site</h1> Try to search with Google the 'The <h1> heading on my site' phrase. If the SERP will contain the keyword and a link to your ...


3

Here is Google's document about cloaking. It defines cloaking as "the practice of presenting different content or URLs to human users and search engines." That page also has a Matt Cutts video. In the video at 0:28, Matt defines cloaking as "showing different content to users than Googlebot." Google only cares what you show to real users and what you ...


3

AOL hasn't powered its own search engine for over a decade it now uses Google search. SOURCE In fact, in June of 1999 Netscape Search was updated and AOL/Netscape search began to be powered by Google bringing their search volume to approximately 3 million per day; huge for the time. More related information can be found on AOL's website: ...


3

This is Vincent from Bing Webmaster Tools and I noticed your post. First of all, I'm sorry to hear about the problem you are having with our crawler's crawl activity across your subdomains. I am sure we can do better. Couple of things: I noticed you mentioned crawl pattern setting in Webmaster Tools wasn't working for. The reason is that when using a ...


3

Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt


3

It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to. However it is possible your site will still be displayed in search results, ...


3

To prevent misuse the inner workings of Googlebot are kept secret so there is no real "expected behaviour". While searching the net I stumbled upon the following: ... search engines understand URL parameters but often ignore fragments. This implies URL fragments might get crawled. On another page I found a second hint. I always thought Googlebot ...


2

After a lot of research and experimentation with this, I finally bit the bullet and set up a Baidu Webmaster Tools account. Its quite straightforward to use when armed with Google Translate in another window. You may need to have firebug activated in order to be able to copy-and-paste Chinese text from buttons that you cannot capture from the normal browser ...


2

Pages that are not frequently updated or are not important in the eyes of a search engine, after time, will be crawled (visited) by search engines less and could eventually be dropped from Google's index altogether eventually. This doesn't mean there is an issue at all and is quite common for large sites with many pages.


2

Just changing from XHTML to HTML 5 won't be an issue or make a difference. If you change your markup you will affect your rankings as semantic markup is a ranking factor. So if your markup is different it stands to reason your rankings may be different. No one can say if it will be helpful or harmful as we have no idea what changes you will be making or if ...



Only top voted, non community-wiki answers of a minimum length are eligible