3

I am seeing a bunch of 404 errors in the Crawl Errors section of Google Webmaster tools with "gibberish" URLs, that look like:

  • https://example.com/baqmZ9/P12V0aYabJIjnAhH5mYp/4w967mBzt9iuEWqTtRPxtuUZwaKK3WiHB6vhUJ0nGTUthJCFUThfurFOwe4FT3yJ/QjoVjrT5/k6+tQ==/

  • https://example.com/asKiRlXposNi466jsUy5g0G6nhezKqoUBl5AOIpGs4KxiSNraYmnxTUaQXtadWXRM7AZyECaVVwaNcKKn99RakcYQU3NELl6lv8KuC6Czzg==/

Where example.com is my site's domain name.

There are around 400 of them, and they do not have Linking information like most other crawl errors.

Any ideas as to what maybe causing this?

Improve this question
8
  • Are you using WP or other CMS?? How about any plug-ins or themes? Do these appear in any sitemap? This does not look like it would be a bad link from another site to me, but something that my be actually created somewhere through the CMS and possibly a plug-in or theme.
    – closetnoc
    Mar 15, 2015 at 23:29
  • Hi....we are using a custom built CMS. I was thinking it may be related to ad tracking javascript code possibly. Remarketing, AdButler. But see no such long URls in our sourcecode.
    – Martin
    Mar 15, 2015 at 23:40
  • Normally, these things come from links made by other sites- typically scraper data sites. But this URI does not seem to fit any examples I have seen. It is still possible that some other site is making bad links using a failed plug-in. Just make sure it is not your site. Once you are sure of this, you can them ignore them. 404 errors from links to your site are no big deal unless you begin to see thousands of them. Do not use Mark as Fixed and Google will remove them after a number of retries and then ignore the link for the future. It will take a while before they are removed.
    – closetnoc
    Mar 15, 2015 at 23:55
  • 1
    Do not use Mark as Fixed? Really? Why is that? Just wondering, as I had been marking things that don't make sense or that are fixed, as fixed.
    – Martin
    Mar 15, 2015 at 23:59
  • 1
    It resets the retry count. In effect, you are telling Google that they should be able to find the page and that you fixed it. But there is nothing to fix. People often treat 404 errors that Google lists as something to be fixed, when in fact, most of the time, it is the right cause/effect. If the pages are not there, then a 404 is correct and therefore nothing to be fixed. If you use Mark as Fixed, the page will continue to be listed (as long as you keep marking it) and Google cannot learn to ignore the link.
    – closetnoc
    Mar 16, 2015 at 0:04

2 Answers 2

Reset to default
1

I have an instant fix for you and I'm gonna assume your server is apache based.

Create an .htaccess file in the document root of your website or open up the existing one and add the following to the bottom of it after all other rewrite rules:

RewriteEngine On
RewriteRule ^uselessurl(.*)$ - [R=410,L,NC]

and replace uselessurl with anything after http://example.com/ thats in the url that you are sure is not supposed to be a valid URL that google thinks returns a 404. In the above code if you left it as is, then any url that starts with http://example.com/uselessurl will result in error 410.

When google sees error 410, it will report it once, then it wont report it again after, even if you indicate its fixed because 410 means gone. 404 means not found with the possible intention that the link might be available later.

Improve this answer
3
  • While this is good advice, it will require the OP to endure at least one 404 error and some time before he will know what to block. I have seen strings like this before. They can be rather random. A 404 is not a problem if you did not create it yourself- assuming that you are not getting thousands of them. So in this case, it really is safe to ignore a 404 that Google reports. Your suggestion does short-circuit some of what can be a minor nuisance.
    – closetnoc
    Mar 16, 2015 at 3:00
  • There are 400 or so random gibberish URL strings currently in WMT - I assume this would require a rule for each - a very time consuming task - and probably will be chasing our tails as more random gibberish URLs appear in WMT. Great advice though for the more consistent 404s.
    – Martin
    Mar 16, 2015 at 3:41
  • Martin, if they are all random yet start with the same set of characters but those same set of characters aren't used in valid pages, then my advice works well. For example, If your invalid URL's consist of http://example.com/abcdfklhgsdlgjsfdlgjsdlgj and http://example.com/abc435346346346363 yet you have no valid URL that starts with http://example.com/abc then use my answer but change uselessurl to abc
    – Mike -- No longer here
    Mar 16, 2015 at 18:11
1

The == at the end makes me think that this is something that has been base 64 encoded. The equals signs are used as padding at the end of base 64 encoding to make it come out to a multiple of four bytes.

Google could be picking it up from any number of sources:

  • Badly formatted data URI images.
  • JavaScript code containing base 64 data in strings (Google does scan JavaScript and it will crawl things in JavaScript that look like they could be URLs.
  • Other sites that are linking to your site incorrectly.

Here is what Google's John Mueller (who works on Webmaster Tools and Sitemaps) has to say about 404 errors that appear in Webmaster tools:

HELP! MY SITE HAS 939 CRAWL ERRORS!!1

I see this kind of question several times a week; you’re not alone - many websites have crawl errors.

  1. 404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html
  2. In some cases, crawl errors may come from a legitimate structural issue within your website or CMS. How you tell? Double-check the origin of the crawl error. If there's a broken link on your site, in your page's static HTML, then that's always worth fixing. (thanks +Martino Mosna)
  3. What about the funky URLs that are “clearly broken?” When our algorithms like your site, they may try to find more great content on it, for example by trying to discover new URLs in JavaScript. If we try those “URLs” and find a 404, that’s great and expected. We just don’t want to miss anything important (insert overly-attached Googlebot meme here). http://support.google.com/webmasters/bin/answer.py?answer=1154698
  4. You don’t need to fix crawl errors in Webmaster Tools. The “mark as fixed” feature is only to help you, if you want to keep track of your progress there; it does not change anything in our web-search pipeline, so feel free to ignore it if you don’t need it. http://support.google.com/webmasters/bin/answer.py?answer=2467403
  5. We list crawl errors in Webmaster Tools by priority, which is based on several factors. If the first page of crawl errors is clearly irrelevant, you probably won’t find important crawl errors on further pages. http://googlewebmastercentral.blogspot.ch/2012/03/crawl-errors-next-generation.html
  6. There’s no need to “fix” crawl errors on your website. Finding 404’s is normal and expected of a healthy, well-configured website. If you have an equivalent new URL, then redirecting to it is a good practice. Otherwise, you should not create fake content, you should not redirect to your homepage, you shouldn’t robots.txt disallow those URLs -- all of these things make it harder for us to recognize your site’s structure and process it properly. We call these “soft 404” errors. http://support.google.com/webmasters/bin/answer.py?answer=181708
  7. Obviously - if these crawl errors are showing up for URLs that you care about, perhaps URLs in your Sitemap file, then that’s something you should take action on immediately. If Googlebot can’t crawl your important URLs, then they may get dropped from our search results, and users might not be able to access them either.
Improve this answer
1
  • Damned good catch!!
    – closetnoc
    Mar 16, 2015 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.