Tag Info

Hot answers tagged

2

Disallow: *?s= Bots following the original robots.txt specification would not be allowed to crawl URLs like these: http://example.com/*?s= http://example.com/*?s=foo http://example.com/*?s=/ So they interpret *, ? and = literally (i.e., these characters have to appear at the beginning of the URL path). But many bots use (their own) extensions to the ...


2

I would highly recommend the Screaming Frog SEO Spider. He gets you all URLs, Status Codes, Titles, Descriptions, Images on your Website, etc. It's really a great SEO tool, as far as I know most SEOs use it. You can export csv- and Excel-files, as you requested, and also you can create sitemaps, image-sitemaps, and so on. The problem here is that the free ...


1

You can also try Site Visualizer tool. It has no limitations on number of URLs to crawl in trial version - it's fully-functional in 30 days. After downloading and installing (Standard version would be enough for your task), click Project -> New, specify a URL of the website you need to crawl, then click Start Crawl tool button. When the crawling is ...


1

I'm not sure why this is happening and how people are finding the link because I have tried searching for techanger and the calculator.php page hasn't shown up. I then searched for techanger calculator.php. I'm sure it's nothing to worry about because the 404 not found page that it brings me to isn't your own custom one. The only thing that I can think of ...


1

If you want to disallow all robots to crawl your site simply use: User-agent: * Disallow: / User-agent: * means that all robots should follow the rule that comes next. And Disallow: / prevents them to crawl any path. You can see more here on robotstxt.org. I think your Disallow: *?s= means that robots are not allowed to crawl URIs with parameters, but I'm ...


1

If your page starts with just a "loading" screen, then you may be in trouble. I don't think all search engines support javascript. Google states they do but Its best not to depend on it. Also, your method requires users to make a minimum of two requests to the server just to receive some useful content. This will be a problem if your pages are visited very ...


1

Alas if your data is worth getting it will be gotten. When we decided to handle IP addresses, the scrapers went horizontal on us and came back with thousands of IP addresses, each accessing us well under the limit we set. I am myself hunting for the newest hints on how to stop robots that do not respect any protection.


1

One way to do this is to use an image of text rather than plain text. It is possible that Google will eventually be smart enough to read the text out of the image, so it might not be completely future-proof, but it should work well for at least a while from now. There's a bunch of disadvantages to this approach. If a person is visually impaired, it's bad. ...



Only top voted, non community-wiki answers of a minimum length are eligible