Tag Info

Hot answers tagged

2

Disallow: *?s= Bots following the original robots.txt specification would not be allowed to crawl URLs like these: http://example.com/*?s= http://example.com/*?s=foo http://example.com/*?s=/ So they interpret *, ? and = literally (i.e., these characters have to appear at the beginning of the URL path). But many bots use (their own) extensions to the ...


2

I would highly recommend the Screaming Frog SEO Spider. He gets you all URLs, Status Codes, Titles, Descriptions, Images on your Website, etc. It's really a great SEO tool, as far as I know most SEOs use it. You can export csv- and Excel-files, as you requested, and also you can create sitemaps, image-sitemaps, and so on. The problem here is that the free ...


1

Your regex code in general is wrong. Try instead something like this: RewriteCond %{HTTP_USER_AGENT} (.*Baiduspider.*|.*DV.*CRAWLER.*) [NC] You are matching against a string in each iteration between the parenthesis () separated by the pipe character | whereas .* is a wild card that matches anything. Optionally you can use \s or \s+ for spaces but .* ...


1

Is there anything unusual or nefarious about private IP addresses in server logs? would like to know if there is a plausible explanation which points to private IPs in Apache access logs being a normal, "usual" thing for some reason Unusual, no. Nefarious, possibly, it really depends on what the traffic is attempting to do. from ...


1

You can also try Site Visualizer tool. It has no limitations on number of URLs to crawl in trial version - it's fully-functional in 30 days. After downloading and installing (Standard version would be enough for your task), click Project -> New, specify a URL of the website you need to crawl, then click Start Crawl tool button. When the crawling is ...


1

I'm not sure why this is happening and how people are finding the link because I have tried searching for techanger and the calculator.php page hasn't shown up. I then searched for techanger calculator.php. I'm sure it's nothing to worry about because the 404 not found page that it brings me to isn't your own custom one. The only thing that I can think of ...


1

If you want to disallow all robots to crawl your site simply use: User-agent: * Disallow: / User-agent: * means that all robots should follow the rule that comes next. And Disallow: / prevents them to crawl any path. You can see more here on robotstxt.org. I think your Disallow: *?s= means that robots are not allowed to crawl URIs with parameters, but I'm ...


1

If your page starts with just a "loading" screen, then you may be in trouble. I don't think all search engines support javascript. Google states they do but Its best not to depend on it. Also, your method requires users to make a minimum of two requests to the server just to receive some useful content. This will be a problem if your pages are visited very ...


1

One way to do this is to use an image of text rather than plain text. It is possible that Google will eventually be smart enough to read the text out of the image, so it might not be completely future-proof, but it should work well for at least a while from now. There's a bunch of disadvantages to this approach. If a person is visually impaired, it's bad. ...



Only top voted, non community-wiki answers of a minimum length are eligible