17
I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link:
The site might previously have had those links
The site may be cloaking and serving those links only to Googlebot and not to visitors
It is a waste of resources, but it won't confuse Google and it won't hurt ...
answered May 21 '14 at 14:04
17
Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is because the 404 error indicates a temporary situation where a 410 error says the page is gone. So every time you mark a 404 as being fixed, you are in effect telling ...
12
This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go.
There are many proper ways to prevent indexing such as using robots.txt, meta tags or authentication.
10
Blocking No User Agent
Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search ...
9
On our website we have recently implemented 410 errors for pages that have been removed permanently. We have around 40 million pages in the Google index and get crawled with 2 million requests per day by the Googlebot.
After cleaning up our database we found a large number of 404 errors kept showing up in the Crawl Errors on Webmaster Tools. When we ...
9
If you know before time when this page (job advert) will expire then consider including an unavailable_after META tag (or X-Robots-Tag HTTP response header) to inform search engines (ie. Google) before time:
<meta name="googlebot" content="unavailable_after: 30-Mar-2014 18:00:00 GMT">
Reference: Robots Exclusion Protocol: now with even more ...
9
If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers.
robots.txt
Include a robots.txt at the site's root including:
User-agent: *
Disallow: /
X-Robots-Tag
Include the following to your .htaccess to add an X-Robots-Tag ...
8
Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally.
To prevent the pages from showing up in the search engine, your 404 errors will do the job. It may take Google a day to remove the page from the index after Googlebot crawls it ...
answered Jun 3 '14 at 17:43
8
There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed.
If this was a PHP page then you could obviously just send a 404 Not Found header as part of the standard response - to make sure that it always returns a 404.
If you are on the Apache web server then you could also use .htaccess (mod_rewrite) to force any ...
7
You do not want to do a 301 redirect to a 404 page. The 404 HTTP response tells the user-agent (browser, search engine, etc) that the requested document cannot be found. If you send a 301 HTTP response then they will think the document has moved to the new location which is your 404 page. That is not accurate.
When you encounter a page not found send the ...
6
301 Redirecting is generally a bad idea when the pages are not relevant. It is believed that Google now determine relevant keywords on the page of links to that of the linked page. Too many non-relevant links you run the risk of being slapped by Google.
Also it's considered bad user experience for users to be greeted with a page they didn't expect (...
6
This is unlikely to be "visitors" (real people) but is likely to be automated software testing for vulnerabilities in the software run by your website. I've seen these types of requests for years. The most common for my servers is requests for WordPress administration pages and Microsoft FrontPage extensions.
If you are not running the software, these ...
answered Jun 23 '14 at 17:34
6
It isn't necessarily bad for SEO. You want to be careful of "sneaky redirects" (see https://support.google.com/webmasters/answer/2721217?hl=en), but this type of behavior should probably fall in acceptable territory.
Honestly, from an SEO perspective, you'd be better off redirecting the broken URL to a legit page on your site via a 301 server side redirect. ...
5
You do not need a custom error document in order to monitor 404's on your site - if that is the requirement. All the information is in your server access log. The HTTP response code (ie. 404, 403, 200, etc) and the URL of the request that produced that response.
The custom error document is a nice way of serving a meaningful response to users.
You can also ...
5
There are tons of scripts out there that optimistically scan random IP addresses on the internet to find vulnerabilities known in various kinds of software. 99.99% of the time, they find nothing (like on your site,) and that 0.01% of the time, the script will pwn the machine and do whatever the script controller wants. Typically, these scripts are run by ...
5
Google is likely to continue trying to crawl these pages for a long time. Webmasters make mistakes, or sites become unavailable for whatever reason, so Google won't remove content at the first sign of a 404.
Alternatively you could serve a 410 Gone instead. This is a much stronger (ie. deliberate) signal that the page has literally "gone" and is not coming ...
5
If another website is linking to those incorrect URLs (because of a typo) and Google is following those links they will get a 404 page. They may assume that page is supposed to exist because another site is linking to them and are reporting the error to you in case you need to fix it.
Naturally after a while they will stop attempting to crawl that URL ...
5
It seems like you might have something like this on your site:
<link rel='alternate' type='application/rss+xml' title='RSS' href='http://example.com/rss'>
So search engines, RSS readers, etc... are attempting to access your /rss directory.
5
Googlebot generally never stops crawling pages once it starts. It is possible that Googlebot will return to check on those URLs every once in a while indefinitely. For reference here is an article about Google's big memory for 404 pages.
You say that you already removed the URLs from the sitemap. That is good. If you hadn't already done so, that would ...
answered Feb 26 '16 at 15:13
5
The file which is being requested is a json file used by some websites to map certain sections of the site and certain pages to a native app so that going into those sections or pages will automatically open the native app and use the app instead of the browser for iOS devices. It appears as though Google is trying to see if you have this file on your site ...
5
A 302 to the home page is likely to be seen as a soft-404 anyway, so that's unlikely to be of any benefit.
A 404 can be perceived as temporary (a 410 is more permanent) and if the page doesn't exist then a 404 is certainly valid. There's not a lot you can do; if the page doesn't exist then it doesn't exist.
However, if this is really a "temporary" thing ...
5
Found this on a Site Lock Facebook page.
SiteLock - Website Security Thanks for the info. After looking into your account, you do have a free scanner provided by your hosting company. In order for us to verify that your 404 page is clean, we actually try to provoke a 404 error by making a request to a non-existent page (e.g. th1s_1s_a_4o4.html). Please ...
5
Nice question. The short answer is yes. As long as the error 404 URL has internal/external links pointing to them, Google will eventually visit the page. We can check this through:
Google Search Console: we can see errors from several months ago.
Analyzing server logs: we can see searchers visiting error 404 and 410 pages after those URLs were deindexed.
...
5
So I dealt with something similar a while back and it was a real pain. I was constantly getting requests for random pdfs and Mexican food recipes in my logs. The IPs were Googlebot, Bingbot, etc (their valid IPs too)
I found out about it initially when I saw a spammy looking domain show up in my backlink profile in Search Console and Ahrefs. In Ahrefs, ...
4
When a page is reported in WMT, it means that their bot actually tried to scan this page normally. When clicking on the error, you can see where the page is linked from. There may be invalid links to your website all around the web and you should research it and possibly notify the owners of these websites if relevant.
Seems the selected answer refers to ...
4
Google offers a fetch as Googlebot option as part of Google Webmaster Tools.
Use that tool to fetch your pages from Google's servers. That will show you if it is a problem with connectivity from Google or with the Googlebot user agent as opposed to downtime of your website when nobody can reach it periodically.
answered Nov 25 '13 at 2:59
4
Blocking by referrer won't help: Googlebot doesn't send a referrer header when it crawls, so it will never see your block.
Here is what Google's John Mueller (who works on Webmaster Tools and Sitemaps) has to say about 404 errors that appear in Webmaster tools:
HELP! MY SITE HAS 939 CRAWL ERRORS!!1
I see this kind of question several times a week; ...
answered Dec 4 '13 at 12:40
4
Redirects to the home page show up in Google Webmaster Tools as "soft 404 errors" in the error reports. Google will not pass Pagerank to what it considers to be "errors."
If you want to preserve Pagerank, you will need to find a more related replacement than just the home page.
answered Dec 24 '13 at 11:29
4
Is it bad to change slugs for SEO and will the Google crawled page
example.com/apple be broken?
Yes to both.
When you change the slug /apple to /banana, as you indicated, the URL changed from:
example.com/apple to example.com/banana
When URLs change, Google will need to find and crawl them again (URLs are the addresses of your pages used by both search ...
Only top voted, non community-wiki answers of a minimum length are eligible