Hot answers tagged

17 votes
Accepted

Massive 404 attack with non existent URLs. How to prevent this?

I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link: The site might previously have had those links The ...
user avatar
17 votes
Accepted

Google is still crawling and indexing my old, dummy, test pages which now are 404 not found

Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is ...
user avatar
  • 32.5k
12 votes

How to make whole site headers send code 404?

This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go. There are many ...
user avatar
  • 2,727
10 votes
Accepted

Can I block requests that have no user agent because they cause 404 errors?

Blocking No User Agent Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to ...
user avatar
  • 32.6k
9 votes

How to make whole site headers send code 404?

If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers. robots....
user avatar
  • 5,794
8 votes

Google is still crawling and indexing my old, dummy, test pages which now are 404 not found

Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally. To prevent the ...
user avatar
8 votes
Accepted

Ok to use meta noindex on 404 pages?

There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed. If this was a PHP page then you could obviously just send a 404 Not Found header as part of the ...
user avatar
  • 41.3k
7 votes
Accepted

Should I 301 permanent redirect to my custom 404 page?

You do not want to do a 301 redirect to a 404 page. The 404 HTTP response tells the user-agent (browser, search engine, etc) that the requested document cannot be found. If you send a 301 HTTP ...
user avatar
  • 85.8k
6 votes
Accepted

Error Log shows nosy visitors trying to access various admin pages

This is unlikely to be "visitors" (real people) but is likely to be automated software testing for vulnerabilities in the software run by your website. I've seen these types of requests for years. ...
user avatar
6 votes
Accepted

Redirect from 404 page to homepage after some seconds and SEO impact

It isn't necessarily bad for SEO. You want to be careful of "sneaky redirects" (see https://support.google.com/webmasters/answer/2721217?hl=en), but this type of behavior should probably fall in ...
user avatar
6 votes
Accepted

Weird bot requests in logs

So I dealt with something similar a while back and it was a real pain. I was constantly getting requests for random pdfs and Mexican food recipes in my logs. The IPs were Googlebot, Bingbot, etc (...
user avatar
  • 4,743
5 votes
Accepted

Check for URLs that return my 404 error page

You do not need a custom error document in order to monitor 404's on your site - if that is the requirement. All the information is in your server access log. The HTTP response code (ie. 404, 403, 200,...
user avatar
  • 41.3k
5 votes

Massive 404 attack with non existent URLs. How to prevent this?

There are tons of scripts out there that optimistically scan random IP addresses on the internet to find vulnerabilities known in various kinds of software. 99.99% of the time, they find nothing (like ...
user avatar
  • 151
5 votes

Google is still crawling and indexing my old, dummy, test pages which now are 404 not found

Google is likely to continue trying to crawl these pages for a long time. Webmasters make mistakes, or sites become unavailable for whatever reason, so Google won't remove content at the first sign of ...
user avatar
  • 41.3k
5 votes
Accepted

Where Did this 404 Error Come From?

If another website is linking to those incorrect URLs (because of a typo) and Google is following those links they will get a 404 page. They may assume that page is supposed to exist because another ...
user avatar
  • 85.8k
5 votes

What is the proper way to indicate that my site doesn't have an RSS feed?

It seems like you might have something like this on your site: <link rel='alternate' type='application/rss+xml' title='RSS' href='http://example.com/rss'> So search engines, RSS readers, etc......
user avatar
5 votes
Accepted

Accidentally submitted broken URLs in a sitemap. How can I stop Google crawling them?

Googlebot generally never stops crawling pages once it starts. It is possible that Googlebot will return to check on those URLs every once in a while indefinitely. For reference here is an article ...
user avatar
5 votes

Google bot hits on files 'apple-app-site-association' and 'assetlinks.json' showing up in Google Webmasters as pages with errors

The file which is being requested is a json file used by some websites to map certain sections of the site and certain pages to a native app so that going into those sections or pages will ...
user avatar
5 votes

Is it bad for SEO to return a 404 for a page that is temporarily unavailable instead of a redirect to the home page?

A 302 to the home page is likely to be seen as a soft-404 anyway, so that's unlikely to be of any benefit. A 404 can be perceived as temporary (a 410 is more permanent) and if the page doesn't exist ...
user avatar
  • 41.3k
5 votes
Accepted

Is Google responsible for th1s_1s_a_4o4.html?

Found this on a Site Lock Facebook page. SiteLock - Website Security Thanks for the info. After looking into your account, you do have a free scanner provided by your hosting company. In order ...
user avatar
  • 66
5 votes
Accepted

Is it necessary to handle every 404 reported in Google Search Console?

Is it necessary to handle every 404 reported in Google Search Console? No. You only need to resolve 404 errors in Google Search Console (GSC) if it's not meant to be a 404. ie. If your site is ...
user avatar
  • 41.3k
5 votes
Accepted

Does Google return to visit a page with 404 Error?

Nice question. The short answer is yes. As long as the error 404 URL has internal/external links pointing to them, Google will eventually visit the page. We can check this through: Google Search ...
user avatar
  • 3,787
4 votes

Google Webmaster Tools reporting crawl error 404 (Not found) for a page which has never existed

When a page is reported in WMT, it means that their bot actually tried to scan this page normally. When clicking on the error, you can see where the page is linked from. There may be invalid links to ...
user avatar
  • 269
4 votes

When redirecting HTTP traffic to HTTPS, is it OK to redirect non-existant pages to a 404 error?

It is very common practice to redirect to error pages when putting in a large class of redirects. In addition to wholesale HTTP to HTTPS redirects, this often happens when: You redirect from naked ...
user avatar
4 votes

Redirect from 404 page to homepage after some seconds and SEO impact

For worldwide compatibility, the answer is no. The reason is because not everyone can read messages very quickly, and depending on the contents of your error page, it might take someone with poor ...
user avatar
4 votes
Accepted

.com/ vs .com when hosting a website

Some hostings might not have mapping for default file to be used as homepage, thus you might sometimes need to add it. If this is Apache-based hosting(commonly Linux hostings use Apache), you will ...
user avatar
  • 156
4 votes
Accepted

Looking for explanation of Apache behavior on manipulating headers

how this workaround indeed works PHP runs later in the request, so most of the time you can simply override any headers that Apache has already set in your PHP code. That's pretty much it. (Aside: ...
user avatar
  • 41.3k
4 votes
Accepted

What to do with unwanted URLs that link to my website?

Unfortunately there's not much you can do. You can't prevent these other (hacked) sites from linking to you. Besides, once Google has found these (now erroneous) URLs, it is likely to continue ...
user avatar
  • 4,277
4 votes
Accepted

Should I redirect all strange URLs in Google Webmaster Tools?

If you never intended to have such urls in your site and they truly mean nothing for you, then it's better to keep serving them as a 404 - a page that is not found - not existing in the site. Use ...
user avatar
  • 375

Only top scored, non community-wiki answers of a minimum length are eligible