Tag Info

New answers tagged

0

To answer the original question: no there is no way to mark a page as user-generated content. But "keyword density" is not a site-wide metric. Adding user-generated content will not affect the keyword density of your other pages and will not harm the ranking of your site (unless there is a large amount of spam). Adding some UGC to existing pages is very ...


0

Bing has a help document that says the following about "Unexpected Error": What does "Unexpected Error" mean? If you are seeing frequent "Unexpected Error" results in Fetch as Bingbot, this may indicate that our crawler is not able to connect to your server and not getting a server response at all. Your server may be unintentionally blocking Bingbot ...


0

It is possible to create a different directory for each user using FTP and IIS. To do so you need to use the "user isolation" option when setting up FTP. Given that you are sharing files with these users, IIS's FTP server sounds like it would be a good option for you. Mac and Linux users can access FTP because all modern browsers support the ftp:// ...


1

If your site already had 10 million pages indexed, I would say double-check that your site is crawlable and a sitemap is submitted through Google Webmaster Tools. Google does prefer sitemaps for larger sites such as yours. It takes time for Google to notice changes sometimes especially if there is a major change with a ton of pages. It can take as much as ...


1

You could try and promote the site a bit more and get more inbound links, and/or ensure that your server can cope with a bigger crawl. See below, (but the whole answer linked is worth a read) : There is also not a hard limit on our crawl. The best way to think about it is that the number of pages that we crawl is roughly proportional to your ...


0

set up a webmaster for your website, it is recommended by google. It will help you in finding errors in crawling.


2

So there are already some answers given. For the second set, the one with the http://url in the request, these tries to find badly configured (not secured) proxy servers that could be wide open. These can then be used to hide the real origin of attacks/scans directed onto an other machine. The other machine will then see the attack as if it was coming from ...


1

Okay. Most of these access (at least) are landscaping attempts to find potential vulnerabilities on your server. They are trying to fingerprint your server to know what web-based applications are installed. The first set and third set are clearly landscaping. The second set may be a result of forged request headers but still likely to be landscaping- not ...


4

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


1

If you're using Apache web-server you could use an .htaccess configuration to white-list by User Agent and prevent genuine bots from reaching your 'tarpit': RewriteEngine On RewriteCond %{HTTP_USER_AGENT} SEMrush [NC] RewriteRule .* - [F,L]


0

Facebook uses the facebookexternalhit user agent facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" Google+ uses a user agent containing "Google" Google (+https://developers.google.com/+/web/snippet/) You should grant them access by user agent.


2

You should be doing a 301 redirect from the .php URLs to the rewritten URLs. If that is not possible for some reason you need to add a canonical URL to the .php URL pointing to the rewritten URL.


0

Since your stated intention is to get accurate statistics as to how many human visitors I get, The approriate solution is to use a service like Google Analytics, or New Relic, once signed up you insert a snippet of javascript into your page (many engines such as wordpress can do this automatically or with a plugin) which sends information to the ...



Top 50 recent answers are included