Hot answers tagged web-crawlers
Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.
Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...
It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to. However it is possible your site will still be displayed in search results, ...
Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt
The data from server logs is limited, and will unavoidably have a high noise to signal ratio, thanks to factors such as bots, caching, CDN. Analyzing page views is a task for page-tag based analytics.
Usually all it takes is a single link from one site to another. Once your site is discoverable via links, it will eventually be crawled as the spiders move throughout the web. As for why you are just now seeing it, no idea. There could be a number of factors in play with this, including a simple frequency increase as the search engines decide that you ...
Did you follow the guidelines for removing the malicious codes and following the hacked site guidelines? The last step is submitting a request for review, which can be done here: https://support.google.com/webmasters/answer/2600725?hl=en To expedite the process, I would submit a fresh sitemap to Google, but I would give it 7 days.
Only top voted, non community-wiki answers of a minimum length are eligible