Tag Info

New answers tagged

0

Bing has a help document that says the following about "Unexpected Error": What does "Unexpected Error" mean? If you are seeing frequent "Unexpected Error" results in Fetch as Bingbot, this may indicate that our crawler is not able to connect to your server and not getting a server response at all. Your server may be unintentionally blocking Bingbot ...


0

It is possible to create a different directory for each user using FTP and IIS. To do so you need to use the "user isolation" option when setting up FTP. Given that you are sharing files with these users, IIS's FTP server sounds like it would be a good option for you. Mac and Linux users can access FTP because all modern browsers support the ftp:// ...


1

If your site already had 10 million pages indexed, I would say double-check that your site is crawlable and a sitemap is submitted through Google Webmaster Tools. Google does prefer sitemaps for larger sites such as yours. It takes time for Google to notice changes sometimes especially if there is a major change with a ton of pages. It can take as much as ...


1

You could try and promote the site a bit more and get more inbound links, and/or ensure that your server can cope with a bigger crawl. See below, (but the whole answer linked is worth a read) : There is also not a hard limit on our crawl. The best way to think about it is that the number of pages that we crawl is roughly proportional to your ...


0

set up a webmaster for your website, it is recommended by google. It will help you in finding errors in crawling.


2

So there are already some answers given. For the second set, the one with the http://url in the request, these tries to find badly configured (not secured) proxy servers that could be wide open. These can then be used to hide the real origin of attacks/scans directed onto an other machine. The other machine will then see the attack as if it was coming from ...


1

Okay. Most of these access (at least) are landscaping attempts to find potential vulnerabilities on your server. They are trying to fingerprint your server to know what web-based applications are installed. The first set and third set are clearly landscaping. The second set may be a result of forged request headers but still likely to be landscaping- not ...


3

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


1

If you're using Apache web-server you could use an .htaccess configuration to white-list by User Agent and prevent genuine bots from reaching your 'tarpit': RewriteEngine On RewriteCond %{HTTP_USER_AGENT} SEMrush [NC] RewriteRule .* - [F,L]


0

Facebook uses the facebookexternalhit user agent facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" Google+ uses a user agent containing "Google" Google (+https://developers.google.com/+/web/snippet/) You should grant them access by user agent.


2

You should be doing a 301 redirect from the .php URLs to the rewritten URLs. If that is not possible for some reason you need to add a canonical URL to the .php URL pointing to the rewritten URL.


0

Since your stated intention is to get accurate statistics as to how many human visitors I get, The approriate solution is to use a service like Google Analytics, or New Relic, once signed up you insert a snippet of javascript into your page (many engines such as wordpress can do this automatically or with a plugin) which sends information to the ...


0

The correct answer is to not mess with robots.txt and instead parse your logs, looking at the User-Agent header as mentioned in the comments. Google, Yahoo, etc. should identify themselves as bots using this header, and disallowing bots via robots.txt would be like driving a truck through your search engine ranking. As @adria said, there are tools out ...


1

The data from server logs is limited, and will unavoidably have a high noise to signal ratio, thanks to factors such as bots, caching, CDN. Analyzing page views is a task for page-tag based analytics.


22

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


3

It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to. However it is possible your site will still be displayed in search results, ...


3

Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...



Top 50 recent answers are included