Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

A SEO firm suggested changing the robots.txt to:

User-agent: *

Disallow: /*/
Allow: /ims/

I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ims folder. I understand the wildcard, but I'm confused by the slashes and don't know how they would play out in conjunction with the wildcard.

* Update * I didn't mention that there is a sitemap listed in the robots.txt file, but according to one tech blogger, (ETA: not true, see below) he realized that sitemaps trump robots exclusions. So, even though this says in Google Webmaster Tools that everything with a trailing slash will not be indexed, the sitemap contains the important links.

I did notice that the link count on Google went from 360 to 336, and the sitemap links under the URL scaled back to 3 from 6. I'm not sure the cause or what links were removed, though. Perhaps it cleaned out garbage.

I'm still clueless why they would add in 'Allow: /ims/', that seems pointless.

And a quick list of what would index according to the robots rules above (without the sitemap) using /*/:

domain.com Indexed domain.com/page.html Indexed domain.com/folder Indexed domain.com/folder/ Not Indexed domain.com/folder/page.html Not Indexed

* Update: The Results are In * Okay, for the sake of those wondering about using robots.txt to exclude everything and hoping that the sitemap will override, here is what happened.

The SEO firm (not someone I hired, and a company I suggested against upon minimal research) basically set up a filter that the three major search engines followed. All indexed pages that were in a folder disappeared, except those which allowed worked without the trailing slash (domain.com/folder was indexed, but domain.com/folder/ was not). The only difference is that Google specifically used the sitemap declaration to pickup all the links and ignored the robots exclusion.

On top of that, the sitemap links (which appear under the first search page result) vanished from the search engines. Eventually, I removed the disallow and allow lines and now I've seen a few of the sitemap links return a few days later.

share

migration rejected from stackoverflow.com Nov 12 '13 at 17:02

This question came from our site for professional and enthusiast programmers. Votes, comments, and answers are locked due to the question being closed here, but it may be eligible for editing and reopening on the site where it originated.

closed as off-topic by Stephen Ostermiller Nov 12 '13 at 17:02

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because it is unlikely to help anybody else. Examples include individual website reviews for SEO purposes and general website troubleshooting." – Stephen Ostermiller
If this question can be reworded to fit the rules in the help center, please edit the question.

1  

1 Answer 1

The /*/ is a match all pattern. For every subfolder under /.

But if you pay this SEO firm i would fire them. Because as with nearly every http allow/disallow rules "First-Match-Rules".

So this two lines are in the incorrect order. Because /*/ will also match /ims/ and the first rule is "disallow" so /ims/ is not crawled either.

quote from the rfc:

To evaluate if access to a URL is allowed, a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used. If no match is found, the default assumption is that the URL is allowed.

RFC

share
    
Are you sure that Google follows robots.txt rules in order? Their own robots.txt documentation suggests that only rule length matters. (See final section titled "order of precedence".) –  Nick Jun 28 '11 at 20:49
    
@Nick In the case of google it is even more worse. They state : "At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined.". –  fyr Jun 28 '11 at 20:54