1

I am facing a problem. THE Problem is that in google search there some spam links that are of type HTTP and I would like to return 410 for those urls that actually do not exist. but for people who type mywebsite.com in url (if they do not add https:// in the beginning they do receive 410 status). if i modify htaccess to redirect all the http requests to https, then all those spam links (there are tons of them and all of those are different pattern such as: /spstyaaliti4csf6ne.desiringly, /rd9ot2iob.twiceuttered, /pi6odntdezo17dia.deranger, /disanidasi34 and so on.) gets redirected to https and then responds with 404 (but tried to check on httpstatus.io as google bot it responds 301 301 200. which should not be the correct behavior.

htaccess file has these lines: those are for https forwarding

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteCond %{HTTP_HOST} ^(www\.)?example\.lt$ [NC]
RewriteRule ^(.*)$ https://www.example.lt/$1 [L,R=301]

and this were for 410

RewriteEngine on
RewriteCond %{HTTPS} !=on [NC]
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [R=410,L]

How can I deal with those redirects and 410? I want normal people typing in browser example.lt (without need to write https:// beforehand to access my website) and those spam URLs to return 410.

2
  • To be clear, you are saying that there are URLs where if you visit them directly you get a 404, but if you are redirected to them you get a 200? Jan 27, 2022 at 23:11
  • if i remove the first htaccess 4 rows, then those urls return 410. if i do not, then they are being redirected to https which is not 410. and i do not know how to solve normal people entering my web url in browser without entering https before hand. they get 410 (without the first part of htaccess 4 lines) and that is not what i would like to achieve. i want normal people to access my url without needing to type https, and those http spam links to be returning 410 (without a redirect to https). is it even possible?
    – Au Gie
    Jan 28, 2022 at 5:36

1 Answer 1

1

Ideally you would fix your site to not return a 200 status after the redirect. Pages that don't exist on your site should return a "404 Not Found" status. However, even if you are return 200 with a "not found" message, Google is smart enough to treat that like a 404. Google calls that a "soft 404" error and treats it just like a real 404 error. Of course having your site return the correct error code makes it unambiguous for search engine crawlers and testing tools like the one you tried to use.

It's fine if you redirect to HTTPS for spam links and then serve a 404 error or a soft-404. It is not required to configure your server serve a 410 Gone status before redirecting. Google treats "410 Gone" status and "404 Not Found" status pages nearly identically: it removes them from the search results. The only difference is that 410 pages get removed immediately while 404 pages have a 24 hour grace period. Googlebot is perfectly able to process a redirect to a 404, 410 page, or soft 404, and remove the URL from search results.


A better 410 Gone rule

Mod rewrite does not use the redirect flag ([R=410]) for setting the 410 status. Rather, it has its own flag for "gone": [G]. When you write a rule for "410 Gone" you should use the gone flag.

One approach to showing 410 for the spam URLs this would be to use a regular expression to try to match the those URLs specifically. Based on the examples you gave, you could use something like this:

# If the request is not for a valid directory
RewriteCond %{REQUEST_FILENAME} !-d
# If the request is not for a valid file
RewriteCond %{REQUEST_FILENAME} !-f
# If the request is not for a valid link
RewriteCond %{REQUEST_FILENAME} !-l
# 410 gone for a long string of lowercase letters 
# and/or numbers followed by an optional long extension
# to handle spam URLs like /spstyaaliti4csf6ne.desiringly
RewriteRule ^/?[a-z0-9]{12,30}(\.[a-z0-9]{8,30})?$ - [G,L]

This rule set would have to go at the top of your .htaccess file before any of your other rules.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.