Tag Info

Hot answers tagged

21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


21

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...


20

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty ...


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


9

There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from some registrars. For example, Google found one domain name I registered using GoDaddy, indexed it, and began sending search results within 20 minutes of ...


8

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


7

Google expects differences between mobile and desktop sites. Even major differences, including differences in link structure, are not a problem. Google crawls the web with different Googlebot user agents for mobile. As long as your server shows that version of Googlebot the same thing that your actual mobile users see, you don't have any penalty risk. ...


6

No. Google may sometimes execute Javascript to determine content, but it's a very bad idea to rely on Javascript for your site to be crawled as you want it to be.


5

You have nothing to worry about. You can use display: none; to switch menus. Search engines are much better at understanding JS and CSS. As long as you are not intentionally trying to manipulate things to get a better ranking. Using display: none; to hide big blocks of text will get you penalized. So if you are only using to hide your desktop menu on ...


4

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


4

So, the solution seems to be that Amazon cloudfront also evaluates my robots.txt and somehow uses different syntax rules from google. The working version of my robots.txt is the following: User-agent: Googlebot-Image Disallow: / User-agent: * Disallow: /homepage Disallow: /uncategorized Disallow: /page Disallow: /category Disallow: /author Disallow: /feed ...


4

The pages will be found and most likely be crawled. You're not far from the truth if you assume bots don't use Javascript. A crawler works similar to this: Go to a webpage and get it's contents Get all the information from the page, like keywords and all other SEO checks Get all internal and external links from the source Per link, go to step 1. It ...


3

To prevent misuse the inner workings of Googlebot are kept secret so there is no real "expected behaviour". While searching the net I stumbled upon the following: ... search engines understand URL parameters but often ignore fragments. This implies URL fragments might get crawled. On another page I found a second hint. I always thought Googlebot ...


3

It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to. However it is possible your site will still be displayed in search results, ...


3

Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt


3

This is Vincent from Bing Webmaster Tools and I noticed your post. First of all, I'm sorry to hear about the problem you are having with our crawler's crawl activity across your subdomains. I am sure we can do better. Couple of things: I noticed you mentioned crawl pattern setting in Webmaster Tools wasn't working for. The reason is that when using a ...


3

AOL hasn't powered its own search engine for over a decade it now uses Google search. SOURCE In fact, in June of 1999 Netscape Search was updated and AOL/Netscape search began to be powered by Google bringing their search volume to approximately 3 million per day; huge for the time. More related information can be found on AOL's website: ...


3

The space is a delimiter (ie. a special character) in .htaccess so must be backslash escaped if you want to match a literal space in the regex. Eg. DV\ CRAWLER. (Otherwise you are likely to get a less than helpful 500 Internal Server error.) Or, you can use the shorthand character class \s which matches any white space character (space, tab or new line / ...


2

Google crawlers are quite flexible as it comes to changes it shouldn't affect your rankings. I'd just proceed as usual in terms of SEO. You may get a lot of crawl errors from non-existent but already indexed forum permalinks. Upload & Test your new sitemaps in Webmaster Tools. Make sure you check Webmaster Tools/Crawl Errors for 404/Not Found URLs ...


2

You should be doing a 301 redirect from the .php URLs to the rewritten URLs. If that is not possible for some reason you need to add a canonical URL to the .php URL pointing to the rewritten URL.


2

So there are already some answers given. For the second set, the one with the http://url in the request, these tries to find badly configured (not secured) proxy servers that could be wide open. These can then be used to hide the real origin of attacks/scans directed onto an other machine. The other machine will then see the attack as if it was coming from ...


2

Robot traffic does not affect page rank. Whilst robots can & may index your website, the sheer volume of robots hitting your website does not play any role whatsoever in affecting your page rank, the driving force behind Google's organic search traffic algorithm. Brian Dean shared a very, very comprehensive list of factors which influence this which can ...


2

PHP user agents are known as libraries. It should be no problem if you block them since legitimate crawlers use other string types, however, can't you check and block only the IP's? In this site, for example, you have ways to block only Chinese traffic.


2

Yes you can. In fact, I recommend it and others too. I would not work so hard on my regular expression to include version numbers- just ^PHP.*$ You will find that some agent names are consistently up to no good even though some will argue that they can be used for good and should not be blocked. That is a negative argument. I study these things and I ...


2

Bing may change titles in their SERPs (Just like Google). See Bing’s blog post How Does Bing Choose The Title For My Web Page?: Sometimes, despite a webmaster’s best efforts, Bing may choose to serve a title that is different to the title of a web site or document. Why "My social networks"? Maybe because it is your first heading on the homepage, which ...


2

If you redirect a page to another page, then Google will no longer index the redirected page. Implementing a redirect removes the content. If you are moving a site from one domain to another without changing the content, you can use the change of address feature in Google Webmaster Tools. When doing so, your new site will take the place of your old site ...


2

If you literally only have a few "groups" you want to block then you would do something like: User-agent: * Disallow: /lang/group1 Disallow: /lang/group2 ...and everything else would be allowed. This would work with all robots that obey the original "standard". Or, you could block all groups (group1, group2, etc.) and make an exception for "group3", like: ...


2

You can't. You have to wait for Google to re-crawl the page on whatever schedule it has determined they will crawl that page. It could be every hour, it could be once a month. Unfortunately you have no control over that.


2

Disallow: *?s= Bots following the original robots.txt specification would not be allowed to crawl URLs like these: http://example.com/*?s= http://example.com/*?s=foo http://example.com/*?s=/ So they interpret *, ? and = literally (i.e., these characters have to appear at the beginning of the URL path). But many bots use (their own) extensions to the ...


2

I would highly recommend the Screaming Frog SEO Spider. He gets you all URLs, Status Codes, Titles, Descriptions, Images on your Website, etc. It's really a great SEO tool, as far as I know most SEOs use it. You can export csv- and Excel-files, as you requested, and also you can create sitemaps, image-sitemaps, and so on. The problem here is that the free ...



Only top voted, non community-wiki answers of a minimum length are eligible