1

So this is what happened.

One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of HTTP 400 errors (I deliberately made requests containing ' return 400 status) in URLs of the following format (replace # with an actual number):

http://example.com/album/gallery/('+#+');

But in reality the URL should be in this form:

http://example.com/album/gallery/#

In each of my pages I used the noarchive attribute as follows:

<meta name="GOOGLEBOT" content="NOARCHIVE">
<meta name="ROBOTS" content="NOARCHIVE">

Since this incident, my maximum crawl rate google allows for my site is now only at 2 requests per second, and almost everyday I see only a couple of new entries similar to above malformed links in my GSC crawl errors report.

It's like somehow google has cached all the HTML output related to my site and relies solely on that HTML as if it was the truth, and it also ignores my "noarchive" which is supposed to prevent google from archiving content in the first place.

The only way I was able to solve this problem even though the malformed links on my site were replaced with valid links was to modify my apache configuration file to include a rewrite rule before the odd character filtering rules (that produced the HTTP 400 errors). This rewrite rule causes all malformed URLs (shown above) to redirect to the valid URLs.

The problem here is that I disabled use of .htaccess for speed and security reasons, so my action resulted in an apache graceful restart. I hate resorting to that and I hate lowering my security.

What I want to know is, could there be some way for me to explicitly tell google to cancel their cached version of my pages (a.k.a. forgive me for my accidental mistake) and re-crawl everything if I made a mistake rather than having it rely on its own cached data for at least several days? I used noarchive meta tag and that does not seem to work.

If I accidentally make faulty links in the future on my site, I don't want to have to go through modifying the apache configuration to redirect the links and gracefully restart apache and/or lower security to my site.

Heck, if there's a special URL I can use to reset web crawler scanning of my site, I'd use it right now.

  • 2
    Almost spilt my tea when I read this: disabled use of .htaccess for speed and security reasons. The .htaccess is considered one of the best ways to harden a website, not make it insecure. The htaccess is literately impossible to hack without the user gaining root access of the box. You do this by using chmod with read access only, or better yet you chown it to root and have group read enabled. The speed thing, shouldn't even be a factor, with good hosting you ain't gonna see a nano seconds difference. Your constantly creating work for yourself :) – Simon Hayter Jun 12 '16 at 17:57
  • 1
    But out of interest why would the cache in Google bother you? It has no relation to rankings nor does it speed up indexing or reindexing. The cache is not realtime, its often days out of date purposely, because its useful for users to view page changes as well as having a offline copy. Just because the cache is older, doesn't mean that Google is not aware of any changes... Google updates rankings, and takes content changes into consideration a lot faster than those get displayed in the cache, don't use it gauge what Google is considering. – Simon Hayter Jun 12 '16 at 18:02
  • When a page is fetched, it is stored. This is where the cache comes from. However, if you use NOARCHIVE, it does not mean that Google will not store what is fetched and work from what it finds. Quite the contrary. The NOARCHIVE tag tells Google NOT to display the results in their cache. It is a simple binary setting that dictates whether a page is displayed or not. Google requires the storage of the page's code in order to rank your page. Assessing a pages value is not instantaneous. It can take days, weeks, or even longer depending. – closetnoc Jun 12 '16 at 20:23
  • Google cache bothers me because it affects my adsense. With the errors I was making maybe 3 cents a day with zero clicks. when I fixed them, my earnings go up and sometimes I see clicks. – Mike Jun 12 '16 at 21:27
  • 1
    FTP should be as secure as anything else as long as reflection attacks are mitigated which should be the case these days for almost all FTP servers. No-one should have direct write access to the web space nor should any user have read access. Instead, FTP should allow a user to upload files to a safe place only. From there, the user would then copy the files into place as a user with proper permission. For my systems, that requires root/sudo. As for cache, it represents the page that Google has indexed. It is not the cache that is a problem, rather the page would have to be refetched. – closetnoc Jun 13 '16 at 0:27
0

There is no way to tell Googlebot to forget about something it has crawled.

Your only recourse is to:

  1. Fix the problem with your HTML.
  2. Redirect any faulty URLs that were caused by the problem.
  3. Wait until Googlebot has recrawled all the pages with faulty HTML and all the bad links that those pages might have generated.

NOARCHIVE prevents Google from showing a cache of the page to users. It has no effect on whether they crawl it again, remember it internally, or use its links to crawl other pages. Google will always crawl links in a page unless that page has the NOFOLLOW attribute. However, NOFOLLOW cannot be applied retroactively.

I tend not to put much into my .htaccess files either, but I don't usually disable them entirely. I find that redirects are often best implemented in the programming logic that powers the web application rather than in .htaccess. You might consider moving your redirects into your software, however that isn't an option if your site is static.

  • I have followed your steps 1 and 2 which does work, but its a pain to always set up redirects to the correct URL when I publish an incorrect URL by accident and realize too late that it has been published. – Mike Jun 28 '16 at 0:04
  • The only other suggestion I have would be to run a crawler against your staging server before a release as part of your QA process. At least then broken links might get caught before going live. – Stephen Ostermiller Jun 28 '16 at 0:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.