3

I want to block this URL: /services/online.html?dir=asc&limit=12&order=price but I want to allow this one: /services/online.html.

I have added this on the robots.txt:

Allow: /services/online.html
Disallow: /*?dir=

When I try to check on the Robot Tester it is still allowed. What could be the right code for it?

2

You should know that URL

/services/online.html?dir=asc&limit=12&order=price

serving the same content as

/services/online.html?limit=12&dir=asc&order=price

as

/services/online.html?order=price&dir=asc&limit=12

and so on, and so on. Position of params in query string doesn't mean anything for a web server. So disallow just query starting with

?dir=

doesn't solve all your problems. So maybe you should try something like this

Disallow: /services/online.html?

But personally I would go for canonical tags, as pointed by danielwill786. It is recommended method. You can read about it here from Google.

|improve this answer|||||
1
Allow: /services/online.html
Disallow: /*?dir=

The most specific rule (based on the length of the path argument) wins when resolving Allow: / Disallow: conflicts - regardless of the order of the directives in the file. So, for the given URL, the first rule wins because it is the most specific path that matches the requested URL.

To resolve this you can either make the Disallow: directive more specific (ie. longer), for example:

Allow: /services/online.html
Disallow: /services/online.html?dir=

Or, simply remove the Allow: directive altogether, since allow is the default action. So, all you would seem to need is:

Disallow: /*?dir=

A URL of the form /services/online.html would be implicitly allowed since it doesn't contain a query string that starts ?dir=.

|improve this answer|||||
  • as i can see it I could also do Disallow: /services/*.html?dir=* ? – Alen Jul 22 '16 at 8:15
  • Yes, you could - as mentioned, you just need to make the rule more specific - there are many possibilities. But do you need the Allow directive at all? In the example you've given it just seems to add unnecessary complexity? The Allow directive is not part of the original "standard", although is supported by all the major SE's - there might be bots crawling your site that do not support it. – MrWhite Jul 22 '16 at 8:43
0

It appears that you are looking forward to block URL with appearing with sorting parameters. The best thing to do is add canonical tags. It the recommended method

Coming to the above, The robots test in webmaster often takes time to update and display the new data add in live robots.txt page. Please confirm that it has been updated.

|improve this answer|||||
  • I did change the robots.txt now but on the webmaster the robots is still same and on that point that "takes time to update". . . you are right. – Alen Jul 22 '16 at 8:17
0

There is some decent advice already in the existing answers but to answer your original question...

You could disallow all URL parameters like so if you wished:-

# Don't crawl URL's that contain sorting/narrowing parameters...
Disallow: *&cat=*
Disallow: *&dir=*
Disallow: *&p=*
Disallow: *&limit=*
Disallow: *&mode=*
Disallow: *&pg=*
Disallow: *&order=*
Disallow: *?cat=*
Disallow: *?dir=*
Disallow: *?p=*
Disallow: *?limit=*
Disallow: *?mode=*
Disallow: *?pg=*
Disallow: *?order=*

Although, it would be best to handle these in Google's Search Console where you can give Google specific instructions for each URL parameter that exists on your website.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.