How to ignore/redirect all URLs matching a certain string

Question

I am using the WordPress plugin, Timely All-in-One events calendar. Unfortunately it is creating a plethora of duplicate URLs which end in strings like

https://www.example.com/events/action~agenda/page_offset~-2/request_format~json/cat_ids~4

or

https://www.example.com/events/action~oneday/exact_date~2-4-2019/

As a consequence of these URL directives each being for a different calendar view but containing the same webpage title and content, some search engines are seeing this as duplicate content. Whilst robots.txt is setup to tell bots to ignore the URLs containing said strings, some crawlers are ignoring robots.txt. I have also disabled the various different calendar views so there is now only the agenda view but even in spite of this, bots continue to crawl these URLs.

Therefore, is it possible to use an Apache .htaccess directive to tell the server to direct any requests containing /action~ to either remove the string from the URL so the browser just reads /events/ or to redirect/forward the URLs to another page?

There are over 500 of these URLs so I ideally would like a quick remedy!

Answer 1

(I kinda feel there should be a WordPress specific way to resolve this, but anyway...)

Appropriate rel="canonical" tags in the head section should have resolved any duplicate content issues.

If these URLs no longer exist then arguably they should return a 404. You can serve this using mod_rewrite at the top of your .htaccess file, before any existing mod_rewrite directives:

RewriteRule ^events/action~ - [R=404]

If you want to send a "410 Gone" instead, change the R=404 flag to G.

To redirect to /events/ instead, you would change the directive to read:

RewriteRule ^events/action~ /events/ [R=302,L]

302 being a temporary redirect, 301 for permanent. (But only changed to 301 once you have thoroughly tested, to avoid caching issues.)

However, if you have blocking directives in robots.txt then no bot that obeys robots.txt will see these responses. For instance, if Google has previously indexed these URLs then you should consider removing the block in robots.txt so Google can see the 4xx response.