17
votes
Accepted
Massive 404 attack with non existent URLs. How to prevent this?
I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link:
The site might previously have had those links
The ...
- 93.9k
17
votes
Accepted
Google is still crawling and indexing my old, dummy, test pages which now are 404 not found
Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is ...
- 32.5k
12
votes
How to make whole site headers send code 404?
This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go.
There are many ...
- 2,727
10
votes
Accepted
Can I block requests that have no user agent because they cause 404 errors?
Blocking No User Agent
Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to ...
- 32.6k
9
votes
How to make whole site headers send code 404?
If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers.
robots....
- 5,794
8
votes
Google is still crawling and indexing my old, dummy, test pages which now are 404 not found
Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally.
To prevent the ...
- 93.9k
8
votes
Accepted
Ok to use meta noindex on 404 pages?
There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed.
If this was a PHP page then you could obviously just send a 404 Not Found header as part of the ...
- 41.3k
7
votes
Accepted
Should I 301 permanent redirect to my custom 404 page?
You do not want to do a 301 redirect to a 404 page. The 404 HTTP response tells the user-agent (browser, search engine, etc) that the requested document cannot be found. If you send a 301 HTTP ...
- 85.8k
6
votes
Accepted
Error Log shows nosy visitors trying to access various admin pages
This is unlikely to be "visitors" (real people) but is likely to be automated software testing for vulnerabilities in the software run by your website. I've seen these types of requests for years. ...
- 93.9k
6
votes
Accepted
Redirect from 404 page to homepage after some seconds and SEO impact
It isn't necessarily bad for SEO. You want to be careful of "sneaky redirects" (see https://support.google.com/webmasters/answer/2721217?hl=en), but this type of behavior should probably fall in ...
- 1,411
6
votes
Accepted
Weird bot requests in logs
So I dealt with something similar a while back and it was a real pain. I was constantly getting requests for random pdfs and Mexican food recipes in my logs. The IPs were Googlebot, Bingbot, etc (...
- 4,743
5
votes
Accepted
Check for URLs that return my 404 error page
You do not need a custom error document in order to monitor 404's on your site - if that is the requirement. All the information is in your server access log. The HTTP response code (ie. 404, 403, 200,...
- 41.3k
5
votes
Massive 404 attack with non existent URLs. How to prevent this?
There are tons of scripts out there that optimistically scan random IP addresses on the internet to find vulnerabilities known in various kinds of software. 99.99% of the time, they find nothing (like ...
- 151
5
votes
Google is still crawling and indexing my old, dummy, test pages which now are 404 not found
Google is likely to continue trying to crawl these pages for a long time. Webmasters make mistakes, or sites become unavailable for whatever reason, so Google won't remove content at the first sign of ...
- 41.3k
5
votes
Accepted
Where Did this 404 Error Come From?
If another website is linking to those incorrect URLs (because of a typo) and Google is following those links they will get a 404 page. They may assume that page is supposed to exist because another ...
- 85.8k
5
votes
What is the proper way to indicate that my site doesn't have an RSS feed?
It seems like you might have something like this on your site:
<link rel='alternate' type='application/rss+xml' title='RSS' href='http://example.com/rss'>
So search engines, RSS readers, etc......
- 176
5
votes
Accepted
Accidentally submitted broken URLs in a sitemap. How can I stop Google crawling them?
Googlebot generally never stops crawling pages once it starts. It is possible that Googlebot will return to check on those URLs every once in a while indefinitely. For reference here is an article ...
- 93.9k
5
votes
Google bot hits on files 'apple-app-site-association' and 'assetlinks.json' showing up in Google Webmasters as pages with errors
The file which is being requested is a json file used by some websites to map certain sections of the site and certain pages to a native app so that going into those sections or pages will ...
- 10.1k
5
votes
Is it bad for SEO to return a 404 for a page that is temporarily unavailable instead of a redirect to the home page?
A 302 to the home page is likely to be seen as a soft-404 anyway, so that's unlikely to be of any benefit.
A 404 can be perceived as temporary (a 410 is more permanent) and if the page doesn't exist ...
- 41.3k
5
votes
Accepted
Is Google responsible for th1s_1s_a_4o4.html?
Found this on a Site Lock Facebook page.
SiteLock - Website Security Thanks for the info. After looking into your account, you do have a free scanner provided by your hosting company. In order ...
- 66
5
votes
Accepted
Is it necessary to handle every 404 reported in Google Search Console?
Is it necessary to handle every 404 reported in Google Search Console?
No.
You only need to resolve 404 errors in Google Search Console (GSC) if it's not meant to be a 404. ie. If your site is ...
- 41.3k
5
votes
Accepted
Does Google return to visit a page with 404 Error?
Nice question. The short answer is yes. As long as the error 404 URL has internal/external links pointing to them, Google will eventually visit the page. We can check this through:
Google Search ...
- 3,787
4
votes
Google Webmaster Tools reporting crawl error 404 (Not found) for a page which has never existed
When a page is reported in WMT, it means that their bot actually tried to scan this page normally. When clicking on the error, you can see where the page is linked from. There may be invalid links to ...
- 269
4
votes
When redirecting HTTP traffic to HTTPS, is it OK to redirect non-existant pages to a 404 error?
It is very common practice to redirect to error pages when putting in a large class of redirects. In addition to wholesale HTTP to HTTPS redirects, this often happens when:
You redirect from naked ...
- 93.9k
4
votes
Redirect from 404 page to homepage after some seconds and SEO impact
For worldwide compatibility, the answer is no.
The reason is because not everyone can read messages very quickly, and depending on the contents of your error page, it might take someone with poor ...
- 13.5k
4
votes
Accepted
.com/ vs .com when hosting a website
Some hostings might not have mapping for default file to be used as homepage, thus you might sometimes need to add it.
If this is Apache-based hosting(commonly Linux hostings use Apache), you will ...
- 156
4
votes
Accepted
Looking for explanation of Apache behavior on manipulating headers
how this workaround indeed works
PHP runs later in the request, so most of the time you can simply override any headers that Apache has already set in your PHP code. That's pretty much it.
(Aside: ...
- 41.3k
4
votes
Accepted
What to do with unwanted URLs that link to my website?
Unfortunately there's not much you can do. You can't prevent these other (hacked) sites from linking to you. Besides, once Google has found these (now erroneous) URLs, it is likely to continue ...
- 4,277
4
votes
Accepted
Should I redirect all strange URLs in Google Webmaster Tools?
If you never intended to have such urls in your site and they truly mean nothing for you, then it's better to keep serving them as a 404 - a page that is not found - not existing in the site.
Use ...
- 375
Only top scored, non community-wiki answers of a minimum length are eligible