Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
1
vote
1answer
137 views
Should a single page fetcher like TinEye obey robots.txt?
I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" ...
1
vote
3answers
358 views
Preventing Spiders on One Subdomain?
We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, ...
8
votes
4answers
420 views
Robots denied by domain is still listed in search results
So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results ?, or any other similar question).
However, if ...
0
votes
1answer
137 views
Why is “mac” disallowed in microsoft.com/robots.txt? [closed]
At http://www.microsoft.com/robots.txt, why are there entries like:
Disallow: /*/mac/help.mspx
Disallow: /*/mac/help.mspx?
Disallow: /*/mactopia/help.mspx?
....
Could there be a real reason to not ...
3
votes
2answers
196 views
Should I disallow robots on A/B test pages?
I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php.
Should I be putting in rules to my ...
1
vote
1answer
620 views
Different robots.txt for two different domains point to same folder
I have the following two domains:
domain.com
test.domain.com
Both point to same folder which is "public_html". What I want is a different robots.txt file for each domain. So when someone ...
4
votes
3answers
579 views
600 visitors per day, 20 backlinks but still not referenced by Google
i've launch a website on wednesday (9 August)
There's already in 4 days
12,000 viewed pages
1,400 visits
20 to 25 backlincks
a sitemap.xml (130 URL)
english language / french language - url like ...
3
votes
5answers
2k views
Disallow robots.txt from being accessed in a browser but still accessible by spiders?
We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
2
votes
2answers
1k views
Robots.txt and Regular Expressions
If I place a regular expression in robots.txt will it be readable for the web crawlers?
1
vote
2answers
3k views
What's the proper way to handle Allow and Disallow in robots.txt?
I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the ...
4
votes
2answers
80 views
robots.txt - just a guess about wild-card
so if I disallow tempPage, does it mean tempPage_1, temp_Page_2, tempPage_x are also disallowed? I tried to google this up, but I don't know...
10
votes
3answers
404 views
Does Google Preview obey Robots.txt?
Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky.
Is this the case and is there a way to ...
3
votes
3answers
308 views
PageRank: will links pointing to pages protected by robots.txt still count?
If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e.
...
4
votes
1answer
332 views
Should I block search engines with robots.txt on my Facebook app?
There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
1
vote
1answer
105 views
What is USH.HTML and why do some robot try to access it?
I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is.
Is this a type of robots.txt file or something specific ...
3
votes
3answers
4k views
How to stop certain urls from being indexed
When I type site:example.com (using my domain obviously),
I get several link errors showing up in the listing. Typically, they are of the form:
www.example.com/some/fixed/path/admin/unblockUser/11
...
4
votes
1answer
292 views
Does the line-ending format of robots.txt matter?
Simple question: Should I make sure to use Unix line endings for my robots.txt, or does it not matter?
22
votes
6answers
311 views
If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?
If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?
Or can the lack of one be harmful?
4
votes
2answers
279 views
What is the best configuration for robots.txt
What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders?
Is there a general setup that should always be used?
