1
vote
1answer
302 views

301 redirect causing Bingbot to crash, redirection limit reached

Inside my .htaccess file, I have the following script: RewriteEngine on RewriteCond %{HTTPS} off RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] <IfModule mod_rewrite.c> ...
0
votes
2answers
419 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
5
votes
1answer
246 views

How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache

The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
2
votes
2answers
288 views

Tool to generate list of redirections (sitemap for .htaccess)

Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site? I'm building new version of some site, but I don't want to get 404 errors for the old pages. ...
1
vote
1answer
539 views

Robots.txt practices with .htaccess redirections (inherits)

I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
1answer
544 views

How does bing-bot( is that the right spider-name? ) and googlebot interpret 301 redirect?

I've been looking for documentation on how the Microsoft and Google bots interpret 301 redirects. It seems that google-bot stores documents on a url based index system. But I haven't been able to ...