So this is what happened.
One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of HTTP 400 errors (I deliberately made requests containing '
return 400 status) in URLs of the following format (replace #
with an actual number):
http://example.com/album/gallery/('+#+');
But in reality the URL should be in this form:
http://example.com/album/gallery/#
In each of my pages I used the noarchive attribute as follows:
<meta name="GOOGLEBOT" content="NOARCHIVE">
<meta name="ROBOTS" content="NOARCHIVE">
Since this incident, my maximum crawl rate google allows for my site is now only at 2 requests per second, and almost everyday I see only a couple of new entries similar to above malformed links in my GSC crawl errors report.
It's like somehow google has cached all the HTML output related to my site and relies solely on that HTML as if it was the truth, and it also ignores my "noarchive" which is supposed to prevent google from archiving content in the first place.
The only way I was able to solve this problem even though the malformed links on my site were replaced with valid links was to modify my apache configuration file to include a rewrite rule before the odd character filtering rules (that produced the HTTP 400 errors). This rewrite rule causes all malformed URLs (shown above) to redirect to the valid URLs.
The problem here is that I disabled use of .htaccess for speed and security reasons, so my action resulted in an apache graceful restart. I hate resorting to that and I hate lowering my security.
What I want to know is, could there be some way for me to explicitly tell google to cancel their cached version of my pages (a.k.a. forgive me for my accidental mistake) and re-crawl everything if I made a mistake rather than having it rely on its own cached data for at least several days? I used noarchive meta tag and that does not seem to work.
If I accidentally make faulty links in the future on my site, I don't want to have to go through modifying the apache configuration to redirect the links and gracefully restart apache and/or lower security to my site.
Heck, if there's a special URL I can use to reset web crawler scanning of my site, I'd use it right now.
disabled use of .htaccess for speed and security reasons
. The.htaccess
is considered one of the best ways to harden a website, not make it insecure. The htaccess is literately impossible to hack without the user gaining root access of the box. You do this by using chmod with read access only, or better yet you chown it to root and have group read enabled. The speed thing, shouldn't even be a factor, with good hosting you ain't gonna see a nano seconds difference. Your constantly creating work for yourself :)