17
I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link:
The site might previously have had those links
The site may be cloaking and serving those links only to Googlebot and not to visitors
It is a waste of resources, but it won't confuse Google and it won't hurt ...
answered May 21 '14 at 14:04
17
Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is because the 404 error indicates a temporary situation where a 410 error says the page is gone. So every time you mark a 404 as being fixed, you are in effect telling ...
12
This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go.
There are many proper ways to prevent indexing such as using robots.txt, meta tags or authentication.
10
Blocking No User Agent
Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search ...
9
If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers.
robots.txt
Include a robots.txt at the site's root including:
User-agent: *
Disallow: /
X-Robots-Tag
Include the following to your .htaccess to add an X-Robots-Tag ...
8
You should login to Google Webmaster Tools and do a fetch as Google, if the page returns a status 200 then you know that your pages are working as intended and what you're experiencing is just the Google cache service not working, which should resolve itself in time.
Google Cache is not Realtime
It's worth mentioning that the Google cache system runs ...
8
Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally.
To prevent the pages from showing up in the search engine, your 404 errors will do the job. It may take Google a day to remove the page from the index after Googlebot crawls it ...
answered Jun 3 '14 at 17:43
7
On our website we have recently implemented 410 errors for pages that have been removed permanently. We have around 40 million pages in the Google index and get crawled with 2 million requests per day by the Googlebot.
After cleaning up our database we found a large number of 404 errors kept showing up in the Crawl Errors on Webmaster Tools. When we ...
6
When you use the full url for ErrorDocument, Apache will always issue a redirect to that location. You want to change your ErrorDocument directive to a relative path instead.
ErrorDocument 404 /404.php
answered Feb 10 '13 at 11:37
6
This is unlikely to be "visitors" (real people) but is likely to be automated software testing for vulnerabilities in the software run by your website. I've seen these types of requests for years. The most common for my servers is requests for WordPress administration pages and Microsoft FrontPage extensions.
If you are not running the software, these ...
answered Jun 23 '14 at 17:34
6
You do not want to do a 301 redirect to a 404 page. The 404 HTTP response tells the user-agent (browser, search engine, etc) that the requested document cannot be found. If you send a 301 HTTP response then they will think the document has moved to the new location which is your 404 page. That is not accurate.
When you encounter a page not found send the ...
6
It isn't necessarily bad for SEO. You want to be careful of "sneaky redirects" (see https://support.google.com/webmasters/answer/2721217?hl=en), but this type of behavior should probably fall in acceptable territory.
Honestly, from an SEO perspective, you'd be better off redirecting the broken URL to a legit page on your site via a 301 server side redirect. ...
6
There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed.
If this was a PHP page then you could obviously just send a 404 Not Found header as part of the standard response - to make sure that it always returns a 404.
You could also use .htaccess (mod_rewrite) to force any direct requests to 404.html to also ...
5
I've finished what I can on the script so far (read comments of original question for detail and context). Source:
http://www.ionfish.org/projects/xml-spider/
Features:
ability to start from any point (resume crashed attempt?) since it tells you which number it's processing.
scans any sitemap public accessible
finds all the links it has to scan BEFORE ...
5
If your site is truly dynamic then you should not be generating img elements for non-existent images in the first place, thus avoiding 404's from the beginning.
Broken img elements are potentially a bad experience for users and for this reason you could be penalised by the search engines.
Will search engines see your JavaScript hidden images as "not for ...
5
Let's analyse your examples. All three have on thing in common: /enquiry/frmenquiry.aspx. That makes it worth googling for. One of the results is this question, but there are several others, a few of which actually display the HTML source code instead of the rendered HTML. That is a classic attack vector for compromising web sites.
I would surmise that your ...
5
With mod_alias activated, you can do that with the Redirect directive:
Redirect 404 /1234/page/
Redirect expects usually the target as 3rd argument, leaving it brings up the configured ErrorDocument. If a target is added it would result in an internal error.
As can be read in the docs:
If the status is between 300 and 399, the URL argument must be ...
5
Sitemap.xml helps your website to be well and fast indexed. By submitting this file to search engines, I don't see any good reason to ask them to index a page that doesn't exist (404 HTTP status).
5
It depends on why the 404 errors are coming up but in general terms, yes, high numbers of 404 errors can count against the quality metrics of your site and hurt your rankings.
Whenever Google spiders the web, they are following links from external sites to yours, your own internal structure, and your past indexed pages. While 404s themselves are not a ...
5
301 Redirecting is generally a bad idea when the pages are not relevant. It is believed that Google now determine relevant keywords on the page of links to that of the linked page. Too many non-relevant links you run the risk of being slapped by Google.
Also it's considered bad user experience for users to be greeted with a page they didn't expect (...
5
You do not need a custom error document in order to monitor 404's on your site - if that is the requirement. All the information is in your server access log. The HTTP response code (ie. 404, 403, 200, etc) and the URL of the request that produced that response.
The custom error document is a nice way of serving a meaningful response to users.
You can also ...
5
There are tons of scripts out there that optimistically scan random IP addresses on the internet to find vulnerabilities known in various kinds of software. 99.99% of the time, they find nothing (like on your site,) and that 0.01% of the time, the script will pwn the machine and do whatever the script controller wants. Typically, these scripts are run by ...
5
Google is likely to continue trying to crawl these pages for a long time. Webmasters make mistakes, or sites become unavailable for whatever reason, so Google won't remove content at the first sign of a 404.
Alternatively you could serve a 410 Gone instead. This is a much stronger (ie. deliberate) signal that the page has literally "gone" and is not coming ...
5
If another website is linking to those incorrect URLs (because of a typo) and Google is following those links they will get a 404 page. They may assume that page is supposed to exist because another site is linking to them and are reporting the error to you in case you need to fix it.
Naturally after a while they will stop attempting to crawl that URL ...
5
It seems like you might have something like this on your site:
<link rel='alternate' type='application/rss+xml' title='RSS' href='http://example.com/rss'>
So search engines, RSS readers, etc... are attempting to access your /rss directory.
5
Googlebot generally never stops crawling pages once it starts. It is possible that Googlebot will return to check on those URLs every once in a while indefinitely. For reference here is an article about Google's big memory for 404 pages.
You say that you already removed the URLs from the sitemap. That is good. If you hadn't already done so, that would ...
answered Feb 26 '16 at 15:13
5
The file which is being requested is a json file used by some websites to map certain sections of the site and certain pages to a native app so that going into those sections or pages will automatically open the native app and use the app instead of the browser for iOS devices. It appears as though Google is trying to see if you have this file on your site ...
5
A 302 to the home page is likely to be seen as a soft-404 anyway, so that's unlikely to be of any benefit.
A 404 can be perceived as temporary (a 410 is more permanent) and if the page doesn't exist then a 404 is certainly valid. There's not a lot you can do; if the page doesn't exist then it doesn't exist.
However, if this is really a "temporary" thing ...
5
Found this on a Site Lock Facebook page.
SiteLock - Website Security Thanks for the info. After looking into your account, you do have a free scanner provided by your hosting company. In order for us to verify that your 404 page is clean, we actually try to provoke a 404 error by making a request to a non-existent page (e.g. th1s_1s_a_4o4.html). Please ...
Only top voted, non community-wiki answers of a minimum length are eligible
