A SEO firm suggested changing the robots.txt to:
User-agent: * Disallow: /*/ Allow: /ims/
I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ims folder. I understand the wildcard, but I'm confused by the slashes and don't know how they would play out in conjunction with the wildcard.
* Update * I didn't mention that there is a sitemap listed in the robots.txt file, but according to one tech blogger, (ETA: not true, see below) he realized that sitemaps trump robots exclusions. So, even though this says in Google Webmaster Tools that everything with a trailing slash will not be indexed, the sitemap contains the important links.
I did notice that the link count on Google went from 360 to 336, and the sitemap links under the URL scaled back to 3 from 6. I'm not sure the cause or what links were removed, though. Perhaps it cleaned out garbage.
I'm still clueless why they would add in 'Allow: /ims/', that seems pointless.
And a quick list of what would index according to the robots rules above (without the sitemap) using /*/:
domain.com Indexed domain.com/page.html Indexed domain.com/folder Indexed domain.com/folder/ Not Indexed domain.com/folder/page.html Not Indexed
* Update: The Results are In * Okay, for the sake of those wondering about using robots.txt to exclude everything and hoping that the sitemap will override, here is what happened.
The SEO firm (not someone I hired, and a company I suggested against upon minimal research) basically set up a filter that the three major search engines followed. All indexed pages that were in a folder disappeared, except those which allowed worked without the trailing slash (domain.com/folder was indexed, but domain.com/folder/ was not). The only difference is that Google specifically used the sitemap declaration to pickup all the links and ignored the robots exclusion.
On top of that, the sitemap links (which appear under the first search page result) vanished from the search engines. Eventually, I removed the disallow and allow lines and now I've seen a few of the sitemap links return a few days later.