Preventing URLs ending in “/” from search engine indexes with robots.txt or rewrite rule

Question

Is there a programmatic way to easily deal with the issue of search engines considering pages where the page works with and without the trailing slash duplicates like via a robots.txt or Nginx rewrite rule?

Examples of "duplicate content" according to search engines:

https://www.jumpingthruhoops.com/filepath
https://www.jumpingthruhoops.com/filepath/

Maybe something like this for the rewrite rule, except I'm not sure what to use as far as where to redirect to (hence the question mark below)

rewrite ^/$ ? permanent;

We use Magento 1.13, where some content already has canonical links generated automatically. So if either the robots or rewrite rules won't work, I'll probably have to try to find an extension that performs that task on pages that don't already have canonical links. Or if anyone has a better approach, I'd be very interested in hearing about it.

Answer 1

rewrite ^/$ ? permanent;

This would only match the document root (a valid request) - this isn't a URL that ends with a slash, but a URL that starts with a slash.

Try something like the following instead, to match any request URL that ends with a slash:

rewrite ^(.+)/$ $1 redirect;

$1 is a backreference to the original URL-path, less the trailing slash.

Change the redirect flag to permanent once you have tested that it works OK - to avoid any caching issues.