2

I have a website (dynamic real estate website) that has been up for nearly a year, all of a sudden Bingbots are trying to access completely lower case versions of the main pages resulting in 404 errors.

The base url is lowercase but subsequent pages are a mixture.

https://example.com/Houses-for-sale-in-Suburb-STATE-3345.php

I have read up on rel canonical and 301 redirects and since have learned that using uppercase is not a good idea - although I decided to use them because states in my country are expressed in UPPERCASE

I am not sure how to handle this, if I convert all of the uppercase URLs to lower case then the bots will still try to access the uppercase versions.

  • What web server? There may be options to set depending upon the web server. Cheers!! – closetnoc Oct 28 '16 at 0:42
  • Just as a point of reference, I realize this it too late for the OP, however, never, never, never, use upper case or mixed case URLs unless you absolutely have no choice. Even then, do not do it! ;-) Cheers!! – closetnoc Oct 28 '16 at 3:00
  • Pretty sure it is a Centos server on shared host, accessed via CPanel without root privileges. – zod Oct 28 '16 at 3:30
  • I am assuming Apache. It may be a bit of work, however, this may help and make your site more usable overall: httpd.apache.org/docs/trunk/rewrite/rewritemap.html – closetnoc Oct 28 '16 at 3:43
  • Just tried it in .htaccess see @Grizzly link however I don't have access to httpd.conf, so of course got a 500 Server error – zod Oct 28 '16 at 5:23
1

Stephen Ostermiller gave me the answer to proceed with testing and to implement the following in my .htaccess

RewriteEngine On # Turn on the rewriting engine

Redirect 301 /houses-for-sale-in-suburb-state-3345.php http://example.com/Houses-for-sale-in-Suburb-STATE-3345.php  

I wrote a redirect for every single page that had CAPS in the URL, except for the dynamically created URL's, which are driven by RewriteRules in the .htaccess & put the Redirects above the RewriteRules

0

When I worked for TripAdvisor, we saw this problem all the time. TripAdvisor has mixed case URLs like:

https://www.tripadvisor.com/Hotels-g60763-New_York_City_New_York-Hotels.html

Many crawlers including BingBot would try to lower case the entire URL before fetching it. To fix the issue we made sure that all lowercase URLs would redirect to the mixed case version. So the following gives a 301 redirect:

https://www.tripadvisor.com/hotels-g60763-new_york_city_new_york-hotels.html

We used the ID in the URL to figure out the correct URL and redirect. It looks like you have a numeric id in your URL that you could use as well.

Because of this experience, I recommend that the canonical version of URLs always be all lowercase. I don't recommend using mixed case URLs because there are so many badly programmed crawlers.

StackExchange does something similar. If you request a butchered URL with the id still intact, you will get a redirect to the correct thing. You can test this by fetching the following URL for this question:

http://webmasters.stackexchange.com/questions/100613/BINGBOT-IS-DUMB

I also recommend that numeric ids go near the front of the URL because of truncation issues. You can correct for

/Hotels-g60763-New_Yor...

but not as easily for:

/Houses-for-sale-in-Su...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.