Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
2
votes
1answer
205 views
Stop bots from crawling old links with extensions
I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist.
So I'm trying to find out if there is a ...
2
votes
1answer
392 views
Why am I getting this message “Some important page is blocked by robots.txt”?
My site's URL is: www.hackinguniverse.org
From some day in Google Webmaster tool a message is showing that says "Some important page is blocked by robots.txt".
My robots.txt is:
User-agent: ...
1
vote
1answer
869 views
Why google isn't updating my site title in search results? [duplicate]
Possible Duplicate:
Google doesn't seem to update the description or title of my homepage
I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
2
votes
2answers
241 views
Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?
Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
0
votes
1answer
181 views
Is “robots.txt” necessarily needed? [duplicate]
Possible Duplicate:
Do I need robots.txt?
From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
1
vote
2answers
238 views
How do I have Google not index any of my site, except my domain?
Disallow: /
I want Google to index the index page, but nothing else. Will this achieve it?
3
votes
2answers
192 views
How can I allow search engines to index my invite only website in ruby on rails?
I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication:
before_filter ...
0
votes
3answers
112 views
Robots are still disallowed even without a robots.txt file
I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has ...
3
votes
3answers
116 views
Do crawlers require a robots.txt file before indexing content?
Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?
0
votes
1answer
533 views
Restricting crawler activity to certain directories with robots.txt
I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers.
In my robots.txt, I have ...
2
votes
2answers
364 views
Image Hot-links & Google's Instant Previews
I'm trying to adjust my site:
1. stop Image Hot-links
2. block Googlebot-Image (because I do not want my images to be searchable)
3. allow Google's Instant Previews
I've tried this but the Instant ...
0
votes
1answer
147 views
Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?
I'm not sure if this is a problem specific to how my hosting is structured or not.
Let's say the main domain for my account is maindomain.com (for purpose of discussion)
I am building other sites, ...
0
votes
2answers
1k views
How to control what appears on google search results?
So I have just built a site and all is well... but I want to control what appears on google search results.
Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
2
votes
2answers
121 views
Does Yahoo Slurp crawl hammer a site?
A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
1
vote
1answer
1k views
Google Webmasters Tools: Denied by robots.txt
I have a website: http://46.105.30.3/
And I'm trying to get testing it using the Fetch as Googlebot option in the Google Webmasters Tools.
When I try it it returns Denied by robots.txt.
My ...
0
votes
1answer
351 views
Will google remove the old caches if noarchive is added to robots.txt
If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
1
vote
2answers
274 views
robots.txt ignored
I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories.
By default it had Disallow: ...
7
votes
2answers
153 views
Robots.txt Issue
So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following:
# go away
...
1
vote
2answers
254 views
Robots.txt not working because my site is still searchable
I put a robots.txt in my server
This is the code that I putted in
User-agent: *
Disallow: /
I just want my entire site won't be searchable in google.
I also put in the NO INDEX, NO FOLLOW on the ...
1
vote
2answers
62 views
Serious Googlebot spike
One of the sites I maintain has seen googlebot use 23gb of bandwidth this month.
This seems utterly mental to me, what can I do about it?
I have reduced the crawl rate on webmaster tools, and ...
1
vote
2answers
415 views
Should a mobile-only site use robots.txt?
I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
1
vote
6answers
77 views
Uploading a non-finished website
I have a pretty basic question. I developed a neet little website wich I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
2
votes
2answers
2k views
robots.txt for a CodeIgniter based website?
I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.
public_html
application
public
sql
style
system
design
images
...
2
votes
2answers
129 views
Robots.txt syntax -> Can this be done?
When creating a robots.txt, are the following the same?
# Slurp
User-agent: Slurp
Disallow: /
and
# Slurp User-agent: Slurp Disallow: /
It seems like for someone disallowing a large number of ...
4
votes
5answers
107 views
removing robots.txt
Will removing robots.txt make the website 'searchable' again.
Curious if you had a robots.txt to avoid indexing, but changed your mind and wanted to get indexed.
2
votes
2answers
120 views
Pointing to Google that a specific website is for mobiles
I have 2 websites: let's call them A and B.
A is a Desktop optimized website.
B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A
How ...
0
votes
2answers
106 views
Will robots.txt block crawling things like menu and footer links?
If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
1
vote
1answer
522 views
“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists
In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow".
I did ...
4
votes
1answer
130 views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
1
vote
2answers
107 views
Robots.txt and specific file inclusion
Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
2
votes
2answers
4k views
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
2
votes
1answer
93 views
Adsense and blocking robots from web pages
If I use a meta tag to make a page 'noindex' does this mean that my adsense ads will not be targeted?
1
vote
2answers
117 views
Problem with my robots.txt
I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
1
vote
2answers
105 views
Where should the robots.txt file be?
If I have a subdomain:
subdir.a.com that redirects to a.com/myfolder/
should the robots.txt file be in:
subdir.a.com/robots.txt or in
a.com/myfolder/robots.txt
3
votes
1answer
362 views
Can a sitemap index contain other sitemap indexes?
I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
3
votes
2answers
132 views
How to temporarely remove a website from search engines?
we are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain.
now we are being already referenced by bots (google, yahoo...) and we dont want ...
1
vote
1answer
104 views
To disallow indexing the category and tag listings in a blog
Mark Wilson says that category and tag listings in a blog should be disallowed in order to prevent duplicate content. I understand this.
However, I want to put internal links on keywords in the blog ...
1
vote
1answer
69 views
Search and Tags for robots.txt
After setting up a blog at blogspot.com I added the site to my webmasters tools of google.
I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
3
votes
1answer
318 views
Quick Question, robots.txt Disallow: /*/ does what exactly?
A SEO firm suggested changing the robots.txt to:
User-agent: *
Disallow: /*/
Allow: /ims/
I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
2
votes
1answer
233 views
How do I disallow an entire directory with robots.txt?
I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example
http://someurl.com/first_url
...
1
vote
1answer
46 views
Blocking URLs and canonical questions
We are running a Magento store set up and was looking to block all pages with the exception of a select few.
It seems the only way we can do this is with blocking direct paths to files that are in ...
4
votes
1answer
381 views
wget not respecting my robots.txt. Is there an interceptor?
I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little.
...
3
votes
4answers
93 views
Stopping Google index some web pages I have
I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?
2
votes
3answers
250 views
After adding *.xls to robots.txt how long will the results continue to show up in google
Is there an estimated amount of time before the document will not appear in google results?
We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...
2
votes
1answer
1k views
Search engines that do _not_ respect robots.txt?
Are there any search engines that do not care about robots.txt?
1
vote
1answer
837 views
How to hide my website from search engines using robots.txt? [duplicate]
Possible Duplicate:
Robots denied by domain is still listed in search results
I want to hide my website from all search engines for that i write robots.txt in that robots.txt i write code ...
2
votes
1answer
131 views
How can I let robots access my site?
I have a webpage on my university's server, which means I cannot access the root directory and change files there.
Is there some other way to give access to robots to my page?
Thanks.
2
votes
3answers
543 views
What is a good robots.txt for WP?
What is the "best" setup for robots.txt?
I'm using the following permalink structure in Wordpress: /%category%/%postname%/.
My robots.txt currently looks like this (copied from somewhere a long time ...
2
votes
2answers
216 views
robots.txt and disallow
Which one is correct?
User-agent: *
Disallow: index.php
or
User-agent: *
Disallow: /index.php
inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30
1
vote
1answer
137 views
Should a single page fetcher like TinEye obey robots.txt?
I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" ...
