New answers tagged web-crawlers
The correct answer is to not mess with robots.txt and instead parse your logs, looking at the User-Agent header as mentioned in the comments. Google, Yahoo, etc. should identify themselves as bots using this header, and disallowing bots via robots.txt would be like driving a truck through your search engine ranking. As @adria said, there are tools out ...
The data from server logs is limited, and will unavoidably have a high noise to signal ratio, thanks to factors such as bots, caching, CDN. Analyzing page views is a task for page-tag based analytics.
Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.
It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to. However it is possible your site will still be displayed in search results, ...
Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt
Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...
Usually all it takes is a single link from one site to another. Once your site is discoverable via links, it will eventually be crawled as the spiders move throughout the web. As for why you are just now seeing it, no idea. There could be a number of factors in play with this, including a simple frequency increase as the search engines decide that you ...
Did you follow the guidelines for removing the malicious codes and following the hacked site guidelines? The last step is submitting a request for review, which can be done here: https://support.google.com/webmasters/answer/2600725?hl=en To expedite the process, I would submit a fresh sitemap to Google, but I would give it 7 days.
I have had several cases where rankings were lost due to technical problems: extended downtime canonical tags that point to the wrong thing noindex tags accidentally applied to all pages In my experience your rankings do eventually return. It can, however, take awhile. As a general rule, I would expect rankings to return in about three times the amount ...
Top 50 recent answers are included