21
As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder.
Here are all the ones I know of:
favicon.ico
Gives your pages an icon in tabs, bookmarks, etc.
robots.txt
Useful if ...
17
Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is because the 404 error indicates a temporary situation where a 410 error says the page is gone. So every time you mark a 404 as being fixed, you are in effect telling ...
16
I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link:
The site might previously have had those links
The site may be cloaking and serving those links only to Googlebot and not to visitors
It is a waste of resources, but it won't confuse Google and it won't hurt ...
answered May 21 '14 at 14:04
12
This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go.
There are many proper ways to prevent indexing such as using robots.txt, meta tags or authentication.
11
There is more than Google in this world. A 410 unambiguously tells a bot that the file is gone. A 404 does not. A persistent bot might keep trying to find a 404 indefinitely whereas they might stop trying to find a 410 immediately which would make your server very happy.
10
Blocking No User Agent
Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search ...
9
If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers.
robots.txt
Include a robots.txt at the site's root including:
User-agent: *
Disallow: /
X-Robots-Tag
Include the following to your .htaccess to add an X-Robots-Tag ...
8
You should login to Google Webmaster Tools and do a fetch as Google, if the page returns a status 200 then you know that your pages are working as intended and what you're experiencing is just the Google cache service not working, which should resolve itself in time.
Google Cache is not Realtime
It's worth mentioning that the Google cache system runs ...
8
Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally.
To prevent the pages from showing up in the search engine, your 404 errors will do the job. It may take Google a day to remove the page from the index after Googlebot crawls it ...
answered Jun 3 '14 at 17:43
7
On our website we have recently implemented 410 errors for pages that have been removed permanently. We have around 40 million pages in the Google index and get crawled with 2 million requests per day by the Googlebot.
After cleaning up our database we found a large number of 404 errors kept showing up in the Crawl Errors on Webmaster Tools. When we ...
6
Besides there being more search engines than Google out there, there's also no reason to assume that Google won't ever change the way they treat 410 responses. Indeed, it seems that's already happened: the information Matt Cutts quotes in the video is from 2007, whereas this post from 2009 by John Mu on Google's Webmaster Central forums says otherwise:
"...
6
I think the only way to do so is using ErrorDocument directive:
ErrorDocument 404 index.php
because with RewriteRule you are "masking" the error serving another page to the request.
Yet you must know that ErrorDocument 404 won't change the URL of the page, thus you will have your homepage on a wrong URL.
6
When you use the full url for ErrorDocument, Apache will always issue a redirect to that location. You want to change your ErrorDocument directive to a relative path instead.
ErrorDocument 404 /404.php
answered Feb 10 '13 at 11:37
6
Sitemap.xml helps your website to be well and fast indexed. By submitting this file to search engines, I don't see any good reason to ask them to index a page that doesn't exist (404 HTTP status).
6
This is unlikely to be "visitors" (real people) but is likely to be automated software testing for vulnerabilities in the software run by your website. I've seen these types of requests for years. The most common for my servers is requests for WordPress administration pages and Microsoft FrontPage extensions.
If you are not running the software, these ...
answered Jun 23 '14 at 17:34
6
You do not want to do a 301 redirect to a 404 page. The 404 HTTP response tells the user-agent (browser, search engine, etc) that the requested document cannot be found. If you send a 301 HTTP response then they will think the document has moved to the new location which is your 404 page. That is not accurate.
When you encounter a page not found send the ...
6
It isn't necessarily bad for SEO. You want to be careful of "sneaky redirects" (see https://support.google.com/webmasters/answer/2721217?hl=en), but this type of behavior should probably fall in acceptable territory.
Honestly, from an SEO perspective, you'd be better off redirecting the broken URL to a legit page on your site via a 301 server side redirect. ...
6
There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed.
If this was a PHP page then you could obviously just send a 404 Not Found header as part of the standard response - to make sure that it always returns a 404.
You could also use .htaccess (mod_rewrite) to force any direct requests to 404.html to also ...
5
This isn't actually related to a firefox plugin, or any plugin at all. crossdomain.xml is part of the flash/flex specification. It's a method to allow/validate cross domain operations for flash and other Adobe products, Sliverlight also seems to use/obey the same adobe policy framework.
From Adobe's Cross-domain policy file specification
A cross-domain ...
5
I've finished what I can on the script so far (read comments of original question for detail and context). Source:
http://www.ionfish.org/projects/xml-spider/
Features:
ability to start from any point (resume crashed attempt?) since it tells you which number it's processing.
scans any sitemap public accessible
finds all the links it has to scan BEFORE ...
5
If your site is truly dynamic then you should not be generating img elements for non-existent images in the first place, thus avoiding 404's from the beginning.
Broken img elements are potentially a bad experience for users and for this reason you could be penalised by the search engines.
Will search engines see your JavaScript hidden images as "not for ...
5
Let's analyse your examples. All three have on thing in common: /enquiry/frmenquiry.aspx. That makes it worth googling for. One of the results is this question, but there are several others, a few of which actually display the HTML source code instead of the rendered HTML. That is a classic attack vector for compromising web sites.
I would surmise that your ...
5
With mod_alias activated, you can do that with the Redirect directive:
Redirect 404 /1234/page/
Redirect expects usually the target as 3rd argument, leaving it brings up the configured ErrorDocument. If a target is added it would result in an internal error.
As can be read in the docs:
If the status is between 300 and 399, the URL argument must be ...
5
It depends on why the 404 errors are coming up but in general terms, yes, high numbers of 404 errors can count against the quality metrics of your site and hurt your rankings.
Whenever Google spiders the web, they are following links from external sites to yours, your own internal structure, and your past indexed pages. While 404s themselves are not a ...
5
301 Redirecting is generally a bad idea when the pages are not relevant. It is believed that Google now determine relevant keywords on the page of links to that of the linked page. Too many non-relevant links you run the risk of being slapped by Google.
Also it's considered bad user experience for users to be greeted with a page they didn't expect (...
5
You do not need a custom error document in order to monitor 404's on your site - if that is the requirement. All the information is in your server access log. The HTTP response code (ie. 404, 403, 200, etc) and the URL of the request that produced that response.
The custom error document is a nice way of serving a meaningful response to users.
You can also ...
5
There are tons of scripts out there that optimistically scan random IP addresses on the internet to find vulnerabilities known in various kinds of software. 99.99% of the time, they find nothing (like on your site,) and that 0.01% of the time, the script will pwn the machine and do whatever the script controller wants. Typically, these scripts are run by ...
5
Google is likely to continue trying to crawl these pages for a long time. Webmasters make mistakes, or sites become unavailable for whatever reason, so Google won't remove content at the first sign of a 404.
Alternatively you could serve a 410 Gone instead. This is a much stronger (ie. deliberate) signal that the page has literally "gone" and is not coming ...
5
If another website is linking to those incorrect URLs (because of a typo) and Google is following those links they will get a 404 page. They may assume that page is supposed to exist because another site is linking to them and are reporting the error to you in case you need to fix it.
Naturally after a while they will stop attempting to crawl that URL ...
Only top voted, non community-wiki answers of a minimum length are eligible
