Googlebot URLencoding longtail URLs which is causing server errors

Question

My Google search traffic has been growing consistently for months without any hiccups. Then a few days ago, Google dropped my traffic by about 80% in one day. The traffic would then later rebound several hours later or the next day, only to drop again for a little while, then come back.

This of course concerns me, and I couldn't figure out what the issue was at first.

I implemented another subdirectory on the site with a lot of new content a few weeks ago.

After running some tests, I realized that many of the links in the new subdirectory have foreign characters. Google then URLencodes them.

So for instance, I link to:

example.com/new-directory/रुला-देने-वाला-Ritesh-Pandey-का-2018-का-सबसे-हिट-Sad-Song-जनाजा-मेरा-Janaja-Mera-Riddhi-Enter.

And then Google URLencodes it as:

example.com/new-directory/%E0%A4%B0%E0%A5%81%E0%A4%B2%E0%A4%BE%20%E0%A4%A6%E0%A5%87%E0%A4%A8%E0%A5%87%20%E0%A4%B5%E0%A4%BE%E0%A4%B2%E0%A4%BE%20Ritesh%20Pandey%20%E0%A4%95%E0%A4%BE%202018%20%E0%A4%95%E0%A4%BE%20%E0%A4%B8%E0%A4%AC%E0%A4%B8%E0%A5%87%20%E0%A4%B9%E0%A4%BF%E0%A4%9F%20Sad%20Song%20-%20%E0%A4%9C%E0%A4%A8%E0%A4%BE%E0%A4%9C%E0%A4%BE%20%E0%A4%AE%E0%A5%87%E0%A4%B0%E0%A4%BE%20-%20Janaja%20Mera%20-%20Riddhi%20Enter.

The problem is that the second URL is too long and is causing server errors from too many redirects. When I fetch these URLs in search console, Google cannot render them.

Search console began reporting these errors about a week ago, which went unnoticed and so I think it's possible that this is what has caused the Google flux traffic drop.

There are millions of links on the site like this, and so to save the website I removed the new content directory, removed all of the links, and am 302 redirecting all pages in that directory to the homepage. Google is now able to fetch these URLs and is showing that they 302 redirect to the homepage.

I could shorten the links of URLencoded URLs and this should fix it as there are less redirects, but I'm afraid of putting the directory back. Because if Google has a bunch of these longtail URLs queued up, they'll continue to report these errors and I risk another traffic drop.

Are there any good solutions to this problem? I would like to be able to put the content back up and remove the 302s, but I don't know how to fix the server errors of too many redirects. I tried the URLs on two different servers and both had the same errors.

Edits:

Can I change LimitRequestLine in httpd.conf to allow for longer URLs? The httpd.conf file says "Direct modifications to the Apache configuration file WILL be lost upon subsequent regeneration of this configuration file, or an Apache update." If the configuration file regenerates/updates then these errors will keep happening. Httpd.conf says "To have your modifications retained, you should create/edit administrator-specific include files:". But I don't really know how to do that and am afraid I could ruin the server by messing it up.

Is it possible to remove URLencoding characters in htaccess before the server redirects? Or do the server URLencoded redirects happen before htaccess mod-rewrites?

Answer 1

Browsers should also encode URLs with international characters before fetching them. URLs can only contain ASCII by spec. International characters were shoehorned in via encoding. Browsers display the international characters to users, but always send the encoded version to the server.

This is defined by RFC 398. There is a slightly easier to read Wikipedia article about it.

International characters have to be encoded using the percent encoding of their UTF-8 values. That means that each single international character in the URL becomes 6 to 12 ASCII characters long. Your percent encoded URL is going to be much much longer than your unencoded URL.

Your server needs to be able to support fully percent encoded URLs. You need to either decrease the length of the URLs or find a server capable of handling them. It looks like you should be able to support URLs 2,000 characters long without any problems. IE should be the limiting factor. Apache server should be able to support URLs up to 4,000 characters long.