Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
2answers
149 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
2
votes
1answer
2k views

Google Webmasters Tools: Denied by robots.txt

I have a website: http://46.105.30.3/ And I'm trying to get testing it using the Fetch as Googlebot option in the Google Webmasters Tools. When I try it it returns Denied by robots.txt. My ...
0
votes
1answer
474 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
1
vote
2answers
299 views

robots.txt ignored

I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories. By default it had Disallow: ...
7
votes
2answers
163 views

Robots.txt Issue

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following: # go away ...
1
vote
2answers
465 views

Robots.txt not working because my site is still searchable

I put a robots.txt in my server This is the code that I putted in User-agent: * Disallow: / I just want my entire site won't be searchable in google. I also put in the NO INDEX, NO FOLLOW on the ...
1
vote
2answers
511 views

Should a mobile-only site use robots.txt?

I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
2
votes
2answers
3k views

robots.txt for a CodeIgniter based website?

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application public sql style system design images ...
2
votes
2answers
132 views

Robots.txt syntax -> Can this be done?

When creating a robots.txt, are the following the same? # Slurp User-agent: Slurp Disallow: / and # Slurp User-agent: Slurp Disallow: / It seems like for someone disallowing a large number of ...
4
votes
5answers
110 views

removing robots.txt

Will removing robots.txt make the website 'searchable' again. Curious if you had a robots.txt to avoid indexing, but changed your mind and wanted to get indexed.
2
votes
2answers
131 views

Pointing to Google that a specific website is for mobiles

I have 2 websites: let's call them A and B. A is a Desktop optimized website. B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A How ...
0
votes
2answers
133 views

Will robots.txt block crawling things like menu and footer links?

If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
1
vote
1answer
733 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
5
votes
1answer
150 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
1
vote
2answers
150 views

Robots.txt and specific file inclusion

Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
3
votes
2answers
4k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
2
votes
1answer
111 views

Adsense and blocking robots from web pages

If I use a meta tag to make a page 'noindex' does this mean that my adsense ads will not be targeted?
1
vote
2answers
130 views

Problem with my robots.txt

I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
1
vote
2answers
113 views

Where should the robots.txt file be?

If I have a subdomain: subdir.a.com that redirects to a.com/myfolder/ should the robots.txt file be in: subdir.a.com/robots.txt or in a.com/myfolder/robots.txt
3
votes
1answer
791 views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
1
vote
1answer
126 views

To disallow indexing the category and tag listings in a blog

Mark Wilson says that category and tag listings in a blog should be disallowed in order to prevent duplicate content. I understand this. However, I want to put internal links on keywords in the blog ...
1
vote
1answer
76 views

Search and Tags for robots.txt

After setting up a blog at blogspot.com I added the site to my webmasters tools of google. I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
3
votes
1answer
442 views

Quick Question, robots.txt Disallow: /*/ does what exactly? [closed]

A SEO firm suggested changing the robots.txt to: User-agent: * Disallow: /*/ Allow: /ims/ I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
2
votes
1answer
3k views

How do I disallow an entire directory with robots.txt?

I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example http://someurl.com/first_url ...
1
vote
1answer
58 views

Blocking URLs and canonical questions

We are running a Magento store set up and was looking to block all pages with the exception of a select few. It seems the only way we can do this is with blocking direct paths to files that are in ...
5
votes
1answer
544 views

wget not respecting my robots.txt. Is there an interceptor?

I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little. ...
2
votes
3answers
277 views

After adding *.xls to robots.txt how long will the results continue to show up in google

Is there an estimated amount of time before the document will not appear in google results? We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...
3
votes
1answer
3k views

Search engines that do _not_ respect robots.txt?

Are there any search engines that do not care about robots.txt?
1
vote
1answer
1k views

How to hide my website from search engines using robots.txt? [duplicate]

Possible Duplicate: Robots denied by domain is still listed in search results I want to hide my website from all search engines for that i write robots.txt in that robots.txt i write code ...
2
votes
1answer
148 views

How can I let robots access my site?

I have a webpage on my university's server, which means I cannot access the root directory and change files there. Is there some other way to give access to robots to my page? Thanks.
2
votes
3answers
852 views

What is a good robots.txt for WP?

What is the "best" setup for robots.txt? I'm using the following permalink structure in Wordpress: /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ...
2
votes
2answers
261 views

robots.txt and disallow

Which one is correct? User-agent: * Disallow: index.php or User-agent: * Disallow: /index.php inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30
1
vote
1answer
150 views

Should a single page fetcher like TinEye obey robots.txt?

I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" ...
1
vote
3answers
439 views

Preventing Spiders on One Subdomain?

We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, ...
8
votes
4answers
485 views

Robots denied by domain is still listed in search results

So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results ?, or any other similar question). However, if ...
0
votes
1answer
164 views

Why is “mac” disallowed in microsoft.com/robots.txt? [closed]

At http://www.microsoft.com/robots.txt, why are there entries like: Disallow: /*/mac/help.mspx Disallow: /*/mac/help.mspx? Disallow: /*/mactopia/help.mspx? .... Could there be a real reason to not ...
3
votes
2answers
241 views

Should I disallow robots on A/B test pages?

I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php. Should I be putting in rules to my ...
1
vote
1answer
913 views

Different robots.txt for two different domains point to same folder

I have the following two domains: domain.com test.domain.com Both point to same folder which is "public_html". What I want is a different robots.txt file for each domain. So when someone ...
4
votes
3answers
612 views

600 visitors per day, 20 backlinks but still not referenced by Google

i've launch a website on wednesday (9 August) There's already in 4 days 12,000 viewed pages 1,400 visits 20 to 25 backlincks a sitemap.xml (130 URL) english language / french language - url like ...
3
votes
5answers
2k views

Disallow robots.txt from being accessed in a browser but still accessible by spiders?

We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
4
votes
2answers
85 views

robots.txt - just a guess about wild-card

so if I disallow tempPage, does it mean tempPage_1, temp_Page_2, tempPage_x are also disallowed? I tried to google this up, but I don't know...
10
votes
3answers
442 views

Does Google Preview obey Robots.txt?

Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky. Is this the case and is there a way to ...
3
votes
3answers
455 views

PageRank: will links pointing to pages protected by robots.txt still count?

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e. ...
4
votes
1answer
381 views

Should I block search engines with robots.txt on my Facebook app?

There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
1
vote
1answer
108 views

What is USH.HTML and why do some robot try to access it?

I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is. Is this a type of robots.txt file or something specific ...
4
votes
1answer
573 views

Does the line-ending format of robots.txt matter?

Simple question: Should I make sure to use Unix line endings for my robots.txt, or does it not matter?
23
votes
6answers
365 views

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? Or can the lack of one be harmful?
4
votes
2answers
345 views

What is the best configuration for robots.txt

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders? Is there a general setup that should always be used?