Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
0
votes
1answer
356 views
Will google remove the old caches if noarchive is added to robots.txt
If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
24
votes
2answers
2k views
Files I should have on every website
I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs.
So far I have the ...
7
votes
2answers
153 views
Robots.txt Issue
So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following:
# go away
...
1
vote
2answers
275 views
robots.txt ignored
I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories.
By default it had Disallow: ...
1
vote
2answers
254 views
Robots.txt not working because my site is still searchable
I put a robots.txt in my server
This is the code that I putted in
User-agent: *
Disallow: /
I just want my entire site won't be searchable in google.
I also put in the NO INDEX, NO FOLLOW on the ...
1
vote
2answers
416 views
Should a mobile-only site use robots.txt?
I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
1
vote
6answers
77 views
Uploading a non-finished website
I have a pretty basic question. I developed a neet little website wich I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
2
votes
2answers
2k views
robots.txt for a CodeIgniter based website?
I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.
public_html
application
public
sql
style
system
design
images
...
0
votes
1answer
536 views
Restricting crawler activity to certain directories with robots.txt
I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers.
In my robots.txt, I have ...
2
votes
2answers
129 views
Robots.txt syntax -> Can this be done?
When creating a robots.txt, are the following the same?
# Slurp
User-agent: Slurp
Disallow: /
and
# Slurp User-agent: Slurp Disallow: /
It seems like for someone disallowing a large number of ...
1
vote
2answers
62 views
Serious Googlebot spike
One of the sites I maintain has seen googlebot use 23gb of bandwidth this month.
This seems utterly mental to me, what can I do about it?
I have reduced the crawl rate on webmaster tools, and ...
2
votes
2answers
241 views
Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?
Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
1
vote
1answer
285 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
0
votes
2answers
106 views
Will robots.txt block crawling things like menu and footer links?
If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
2
votes
2answers
120 views
Pointing to Google that a specific website is for mobiles
I have 2 websites: let's call them A and B.
A is a Desktop optimized website.
B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A
How ...
1
vote
1answer
525 views
“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists
In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow".
I did ...
4
votes
1answer
131 views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
1
vote
2answers
107 views
Robots.txt and specific file inclusion
Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
4
votes
3answers
550 views
nofollow doesn't stop crawling?
In my website, most of the pages has a linkedin sharing button written in this way:
<li>
<a id="share_linkedin"
...
2
votes
1answer
93 views
Adsense and blocking robots from web pages
If I use a meta tag to make a page 'noindex' does this mean that my adsense ads will not be targeted?
1
vote
5answers
217 views
Should I add a “nofollow” attribute to download links, or disallow the URLs in robots.txt?
I have a download link very similar to Opera's one - it's just a script that sends the file. It doesn't have an extension and there's no obvious way to tell that it's actually a download link.
So ...
1
vote
2answers
118 views
Problem with my robots.txt
I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
1
vote
2answers
105 views
Where should the robots.txt file be?
If I have a subdomain:
subdir.a.com that redirects to a.com/myfolder/
should the robots.txt file be in:
subdir.a.com/robots.txt or in
a.com/myfolder/robots.txt
3
votes
1answer
374 views
Can a sitemap index contain other sitemap indexes?
I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
1
vote
1answer
405 views
Robots.txt practices with .htaccess redirections (inherits)
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place.
We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
1answer
69 views
Search and Tags for robots.txt
After setting up a blog at blogspot.com I added the site to my webmasters tools of google.
I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
4
votes
2answers
903 views
How do you disallow root in robots.txt, but allow a subdirectory?
Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?
2
votes
1answer
235 views
How do I disallow an entire directory with robots.txt?
I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example
http://someurl.com/first_url
...
1
vote
1answer
46 views
Blocking URLs and canonical questions
We are running a Magento store set up and was looking to block all pages with the exception of a select few.
It seems the only way we can do this is with blocking direct paths to files that are in ...
3
votes
2answers
132 views
How to temporarely remove a website from search engines?
we are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain.
now we are being already referenced by bots (google, yahoo...) and we dont want ...
5
votes
1answer
386 views
wget not respecting my robots.txt. Is there an interceptor?
I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little.
...
3
votes
1answer
319 views
Quick Question, robots.txt Disallow: /*/ does what exactly?
A SEO firm suggested changing the robots.txt to:
User-agent: *
Disallow: /*/
Allow: /ims/
I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
3
votes
4answers
93 views
Stopping Google index some web pages I have
I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?
1
vote
1answer
1k views
Strange robots.txt - how and why did it get there?
I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
3
votes
2answers
192 views
How can I allow search engines to index my invite only website in ruby on rails?
I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication:
before_filter ...
2
votes
3answers
293 views
Google is indexing unintended content that is either unpublished or a secret part of site
Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that I'm working on - www.domain.com/m/ <-- note the M.
How is that even possible? Only thing I can ...
2
votes
3answers
250 views
After adding *.xls to robots.txt how long will the results continue to show up in google
Is there an estimated amount of time before the document will not appear in google results?
We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...
2
votes
1answer
1k views
Search engines that do _not_ respect robots.txt?
Are there any search engines that do not care about robots.txt?
1
vote
1answer
845 views
How to hide my website from search engines using robots.txt? [duplicate]
Possible Duplicate:
Robots denied by domain is still listed in search results
I want to hide my website from all search engines for that i write robots.txt in that robots.txt i write code ...
2
votes
1answer
131 views
How can I let robots access my site?
I have a webpage on my university's server, which means I cannot access the root directory and change files there.
Is there some other way to give access to robots to my page?
Thanks.
2
votes
2answers
217 views
robots.txt and disallow
Which one is correct?
User-agent: *
Disallow: index.php
or
User-agent: *
Disallow: /index.php
inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30
1
vote
1answer
137 views
Should a single page fetcher like TinEye obey robots.txt?
I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" ...
1
vote
3answers
357 views
Preventing Spiders on One Subdomain?
We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, ...
1
vote
1answer
1k views
Google Webmasters Tools: Denied by robots.txt
I have a website: http://46.105.30.3/
And I'm trying to get testing it using the Fetch as Googlebot option in the Google Webmasters Tools.
When I try it it returns Denied by robots.txt.
My ...
8
votes
4answers
419 views
Robots denied by domain is still listed in search results
So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results ?, or any other similar question).
However, if ...
0
votes
1answer
137 views
Why is “mac” disallowed in microsoft.com/robots.txt? [closed]
At http://www.microsoft.com/robots.txt, why are there entries like:
Disallow: /*/mac/help.mspx
Disallow: /*/mac/help.mspx?
Disallow: /*/mactopia/help.mspx?
....
Could there be a real reason to not ...
3
votes
2answers
196 views
Should I disallow robots on A/B test pages?
I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php.
Should I be putting in rules to my ...
1
vote
1answer
616 views
Different robots.txt for two different domains point to same folder
I have the following two domains:
domain.com
test.domain.com
Both point to same folder which is "public_html". What I want is a different robots.txt file for each domain. So when someone ...
1
vote
2answers
518 views
SEO: disallowing Google from indexing forms?
From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not?
I mean my forms hardly ever contains keyword/keyphrases.
So I'm wondering what's the point of letting ...
1
vote
1answer
104 views
To disallow indexing the category and tag listings in a blog
Mark Wilson says that category and tag listings in a blog should be disallowed in order to prevent duplicate content. I understand this.
However, I want to put internal links on keywords in the blog ...
