Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
3answers
802 views

What is a good robots.txt for WP?

What is the "best" setup for robots.txt? I'm using the following permalink structure in Wordpress: /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ...
2
votes
2answers
250 views

robots.txt and disallow

Which one is correct? User-agent: * Disallow: index.php or User-agent: * Disallow: /index.php inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30
1
vote
1answer
148 views

Should a single page fetcher like TinEye obey robots.txt?

I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" ...
1
vote
3answers
437 views

Preventing Spiders on One Subdomain?

We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, ...
8
votes
4answers
473 views

Robots denied by domain is still listed in search results

So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results ?, or any other similar question). However, if ...
0
votes
1answer
151 views

Why is “mac” disallowed in microsoft.com/robots.txt? [closed]

At http://www.microsoft.com/robots.txt, why are there entries like: Disallow: /*/mac/help.mspx Disallow: /*/mac/help.mspx? Disallow: /*/mactopia/help.mspx? .... Could there be a real reason to not ...
3
votes
2answers
221 views

Should I disallow robots on A/B test pages?

I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php. Should I be putting in rules to my ...
1
vote
1answer
851 views

Different robots.txt for two different domains point to same folder

I have the following two domains: domain.com test.domain.com Both point to same folder which is "public_html". What I want is a different robots.txt file for each domain. So when someone ...
4
votes
3answers
602 views

600 visitors per day, 20 backlinks but still not referenced by Google

i've launch a website on wednesday (9 August) There's already in 4 days 12,000 viewed pages 1,400 visits 20 to 25 backlincks a sitemap.xml (130 URL) english language / french language - url like ...
3
votes
5answers
2k views

Disallow robots.txt from being accessed in a browser but still accessible by spiders?

We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
4
votes
2answers
84 views

robots.txt - just a guess about wild-card

so if I disallow tempPage, does it mean tempPage_1, temp_Page_2, tempPage_x are also disallowed? I tried to google this up, but I don't know...
10
votes
3answers
435 views

Does Google Preview obey Robots.txt?

Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky. Is this the case and is there a way to ...
3
votes
3answers
438 views

PageRank: will links pointing to pages protected by robots.txt still count?

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e. ...
4
votes
1answer
379 views

Should I block search engines with robots.txt on my Facebook app?

There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
1
vote
1answer
108 views

What is USH.HTML and why do some robot try to access it?

I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is. Is this a type of robots.txt file or something specific ...
4
votes
1answer
513 views

Does the line-ending format of robots.txt matter?

Simple question: Should I make sure to use Unix line endings for my robots.txt, or does it not matter?
22
votes
6answers
355 views

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? Or can the lack of one be harmful?
4
votes
2answers
343 views

What is the best configuration for robots.txt

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders? Is there a general setup that should always be used?