2

My Google search traffic has been growing consistently for months without any hiccups. Then a few days ago, Google dropped my traffic by about 80% in one day. The traffic would then later rebound several hours later or the next day, only to drop again for a little while, then come back.

This of course concerns me, and I couldn't figure out what the issue was at first.

I implemented another subdirectory on the site with a lot of new content a few weeks ago.

After running some tests, I realized that many of the links in the new subdirectory have foreign characters. Google then URLencodes them.

So for instance, I link to:

example.com/new-directory/रुला-देने-वाला-Ritesh-Pandey-का-2018-का-सबसे-हिट-Sad-Song-जनाजा-मेरा-Janaja-Mera-Riddhi-Enter.

And then Google URLencodes it as:

example.com/new-directory/%E0%A4%B0%E0%A5%81%E0%A4%B2%E0%A4%BE%20%E0%A4%A6%E0%A5%87%E0%A4%A8%E0%A5%87%20%E0%A4%B5%E0%A4%BE%E0%A4%B2%E0%A4%BE%20Ritesh%20Pandey%20%E0%A4%95%E0%A4%BE%202018%20%E0%A4%95%E0%A4%BE%20%E0%A4%B8%E0%A4%AC%E0%A4%B8%E0%A5%87%20%E0%A4%B9%E0%A4%BF%E0%A4%9F%20Sad%20Song%20-%20%E0%A4%9C%E0%A4%A8%E0%A4%BE%E0%A4%9C%E0%A4%BE%20%E0%A4%AE%E0%A5%87%E0%A4%B0%E0%A4%BE%20-%20Janaja%20Mera%20-%20Riddhi%20Enter.

The problem is that the second URL is too long and is causing server errors from too many redirects. When I fetch these URLs in search console, Google cannot render them.

Search console began reporting these errors about a week ago, which went unnoticed and so I think it's possible that this is what has caused the Google flux traffic drop.

There are millions of links on the site like this, and so to save the website I removed the new content directory, removed all of the links, and am 302 redirecting all pages in that directory to the homepage. Google is now able to fetch these URLs and is showing that they 302 redirect to the homepage.

I could shorten the links of URLencoded URLs and this should fix it as there are less redirects, but I'm afraid of putting the directory back. Because if Google has a bunch of these longtail URLs queued up, they'll continue to report these errors and I risk another traffic drop.

Are there any good solutions to this problem? I would like to be able to put the content back up and remove the 302s, but I don't know how to fix the server errors of too many redirects. I tried the URLs on two different servers and both had the same errors.

Edits:

Can I change LimitRequestLine in httpd.conf to allow for longer URLs? The httpd.conf file says "Direct modifications to the Apache configuration file WILL be lost upon subsequent regeneration of this configuration file, or an Apache update." If the configuration file regenerates/updates then these errors will keep happening. Httpd.conf says "To have your modifications retained, you should create/edit administrator-specific include files:". But I don't really know how to do that and am afraid I could ruin the server by messing it up.

Is it possible to remove URLencoding characters in htaccess before the server redirects? Or do the server URLencoded redirects happen before htaccess mod-rewrites?

2

Browsers should also encode URLs with international characters before fetching them. URLs can only contain ASCII by spec. International characters were shoehorned in via encoding. Browsers display the international characters to users, but always send the encoded version to the server.

This is defined by RFC 398. There is a slightly easier to read Wikipedia article about it.

International characters have to be encoded using the percent encoding of their UTF-8 values. That means that each single international character in the URL becomes 6 to 12 ASCII characters long. Your percent encoded URL is going to be much much longer than your unencoded URL.

Your server needs to be able to support fully percent encoded URLs. You need to either decrease the length of the URLs or find a server capable of handling them. It looks like you should be able to support URLs 2,000 characters long without any problems. IE should be the limiting factor. Apache server should be able to support URLs up to 4,000 characters long.

  • Thank you Stephen, this sounds right to me. I believe the issue is that my servers don't allow for long enough URLs. From what I'm reading, URL length can be changed through "urlrequestline" in apache settings. I believe it's in the httpd.conf file, but I don't have access to that file on my servers. Do you know of a way that I can change this, or do I simply not have enough control over my servers to do so? And do you have any other solutions? Even removing any urlencodings from a URL in htaccess before redirects would be helpful. Ty Stephen – Michael d May 8 '18 at 10:53
  • It looks like you need to set LimitRequestLine LimitRequestFieldSize according to confluence.atlassian.com/jirakb/… It is probably the latter that is causing problems on redirect. I'm not sure if it can be done in .htaccess. Probably not. You might have to have access to the main conf files. – Stephen Ostermiller May 8 '18 at 11:23
  • For LimitRequest, The httpd.conf file says "Direct modifications to the Apache configuration file WILL be lost upon subsequent regeneration of this configuration file, or an Apache update." If the configuration file regenerates/updates then these errors will keep happening. Httpd.conf says "To have your modifications retained, you should create/edit administrator-specific include files:". But I don't really know how to do that and am afraid I could ruin the server by messing it up. - Sounds like I should probably abort this plan and keep the 302/301 redirects? Ty Stephen – Michael d May 9 '18 at 2:32
  • If it is a Debian based system, it is suggesting you create a NEW conf file like /etc/apache2/conf/long-urls.conf and put the directive in that. Then enable it using a2enconf long-urls. Then restart the server. Making any changes to existing configuration files means that your changes could get overwritten when Apache gets upgraded by the package manager. – Stephen Ostermiller May 9 '18 at 8:44

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.