Tag Info

Hot answers tagged

21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


21

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...


20

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty ...


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


8

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


6

No. Google may sometimes execute Javascript to determine content, but it's a very bad idea to rely on Javascript for your site to be crawled as you want it to be.


5

Google only cares that they see the same thing your users see. If you're only serving up different content to these bad crawlers, and your users are getting your normal content and Google is getting that same content, you're ok.


5

You have nothing to worry about. You can use display: none; to switch menus. Search engines are much better at understanding JS and CSS. As long as you are not intentionally trying to manipulate things to get a better ranking. Using display: none; to hide big blocks of text will get you penalized. So if you are only using to hide your desktop menu on ...


5

Google expects differences between mobile and desktop sites. Even major differences, including differences in link structure, are not a problem. Google crawls the web with different Googlebot user agents for mobile. As long as your server shows that version of Googlebot the same thing that your actual mobile users see, you don't have any penalty risk. ...


4

The current GoogleBot Smartphone agent, as tested with the 'Fetch as Google' Tool is essentially a fake iPhone using a headless Webkit Safari 6.0 Engine, running on a Linux x86_64 desktop machine. The default non-responsive viewport width is that of an iPhone at 980px. With a viewport <meta name="viewport" content="width=device-width, ...


4

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


4

So, the solution seems to be that Amazon cloudfront also evaluates my robots.txt and somehow uses different syntax rules from google. The working version of my robots.txt is the following: User-agent: Googlebot-Image Disallow: / User-agent: * Disallow: /homepage Disallow: /uncategorized Disallow: /page Disallow: /category Disallow: /author Disallow: /feed ...


3

This is Vincent from Bing Webmaster Tools and I noticed your post. First of all, I'm sorry to hear about the problem you are having with our crawler's crawl activity across your subdomains. I am sure we can do better. Couple of things: I noticed you mentioned crawl pattern setting in Webmaster Tools wasn't working for. The reason is that when using a ...


3

AOL hasn't powered its own search engine for over a decade it now uses Google search. SOURCE In fact, in June of 1999 Netscape Search was updated and AOL/Netscape search began to be powered by Google bringing their search volume to approximately 3 million per day; huge for the time. More related information can be found on AOL's website: ...


3

Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt


3

Here is Google's document about cloaking. It defines cloaking as "the practice of presenting different content or URLs to human users and search engines." That page also has a Matt Cutts video. In the video at 0:28, Matt defines cloaking as "showing different content to users than Googlebot." Google only cares what you show to real users and what you ...


3

Google now officially processes JavaScript. In order to solve this problem, we decided to try to understand pages by executing JavaScript. It’s hard to do that at the scale of the current web, but we decided that it’s worth it. We have been gradually improving how we do this for some time. In the past few months, our indexing system has been rendering a ...


3

It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to. However it is possible your site will still be displayed in search results, ...


3

To prevent misuse the inner workings of Googlebot are kept secret so there is no real "expected behaviour". While searching the net I stumbled upon the following: ... search engines understand URL parameters but often ignore fragments. This implies URL fragments might get crawled. On another page I found a second hint. I always thought Googlebot ...


2

As mentioned in the comments, robots.txt only blocks crawling of files, it wont remove them from the SERPs. To remove PDF files from the SERPs add an X-Robots-Tag: noindex in the HTTP header used to serve the file. They’ll drop out over time if you use the X-Robot-Tag with the noindex directive, or as mentioned you can speed up the removal process in GWT. ...


2

Google's bots will still want to request /robots.txt from your sub domain and not /robots_static.txt which would have no meaning to them. RewriteCond %{HTTP_HOST} ^www\.static\..*$ [NC] RewriteRule ^/robots\.txt$ /robots_static.txt [L] When requests for /robots.txt are made from your www.static domain the /robots_static.txt file will be served up as if ...


2

There's no set time frame for when Google will stop. But they will eventually stop. They need to keep crawling them for a period of time because those pages may only be unavailable temporarily due to an error on your end. So they want to give them a chance to reappear in those cases. It is assumed if those pages were intentionally removed you would serve up ...


2

Google crawlers are quite flexible as it comes to changes it shouldn't affect your rankings. I'd just proceed as usual in terms of SEO. You may get a lot of crawl errors from non-existent but already indexed forum permalinks. Upload & Test your new sitemaps in Webmaster Tools. Make sure you check Webmaster Tools/Crawl Errors for 404/Not Found URLs ...


2

You should be doing a 301 redirect from the .php URLs to the rewritten URLs. If that is not possible for some reason you need to add a canonical URL to the .php URL pointing to the rewritten URL.


2

So there are already some answers given. For the second set, the one with the http://url in the request, these tries to find badly configured (not secured) proxy servers that could be wide open. These can then be used to hide the real origin of attacks/scans directed onto an other machine. The other machine will then see the attack as if it was coming from ...


2

Robot traffic does not affect page rank. Whilst robots can & may index your website, the sheer volume of robots hitting your website does not play any role whatsoever in affecting your page rank, the driving force behind Google's organic search traffic algorithm. Brian Dean shared a very, very comprehensive list of factors which influence this which can ...


2

PHP user agents are known as libraries. It should be no problem if you block them since legitimate crawlers use other string types, however, can't you check and block only the IP's? In this site, for example, you have ways to block only Chinese traffic.


2

Yes you can. In fact, I recommend it and others too. I would not work so hard on my regular expression to include version numbers- just ^PHP.*$ You will find that some agent names are consistently up to no good even though some will argue that they can be used for good and should not be blocked. That is a negative argument. I study these things and I ...


2

Bing may change titles in their SERPs (Just like Google). See Bing’s blog post How Does Bing Choose The Title For My Web Page?: Sometimes, despite a webmaster’s best efforts, Bing may choose to serve a title that is different to the title of a web site or document. Why "My social networks"? Maybe because it is your first heading on the homepage, which ...



Only top voted, non community-wiki answers of a minimum length are eligible