Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
24
votes
2answers
2k views
Files I should have on every website
I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs.
So far I have the ...
22
votes
6answers
309 views
If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?
If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?
Or can the lack of one be harmful?
10
votes
3answers
403 views
Does Google Preview obey Robots.txt?
Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky.
Is this the case and is there a way to ...
9
votes
3answers
269 views
Robots.txt: do I need to disallow a page which is not linked anywhere?
There are some pages on my websites that I want the user to be able to visit only if I give him/her the url.
If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
8
votes
3answers
2k views
Does Google cache robots.txt?
I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...
8
votes
4answers
417 views
Robots denied by domain is still listed in search results
So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results ?, or any other similar question).
However, if ...
7
votes
2answers
153 views
Robots.txt Issue
So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following:
# go away
...
6
votes
4answers
306 views
robots.txt not updated
I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
6
votes
2answers
73 views
What kind of redirect (301 or 302) for an email links tracker?
We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application ...
5
votes
4answers
191 views
Robots.txt vs Sitemap — Who wins in a Conflict
If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
5
votes
2answers
1k views
Why do Google search results include pages disallowed in robots.txt?
I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this:
User-Agent: *
Disallow: /email
Yet I recently noticed that Google ...
5
votes
2answers
115 views
Robots.txt and pattern matching
Adding this to my robots.txt
User-agent: *
Disallow: /*action=*$
How does robots not recognizing wild cards handle this?
5
votes
1answer
333 views
Do I reference all my *.xml.gz sitemap files in robots.txt
I have a website with over 1 million url's. As such, I have 1 sitemap.xml file which acts as an index to many sitemap0000.xml.gz, sitemap0001.xml.gz ... sitemap0030.xml.gz etc. files.
My question is: ...
5
votes
1answer
211 views
should I block wordpress archive pages from search engines?
I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives.
Google has indexed the site fully.
Since the archives currently display full posts I've thought that I shouldn't ...
5
votes
3answers
274 views
Does loading images from a server that disallows robots affect SEO?
Lets say a page on example.com has all of its images coming from img.example.com:
<img src="http://img.example.com/keyword.jpg" alt="keyword"/>
while robots.txt on img.example.com disallows ...
4
votes
3answers
579 views
600 visitors per day, 20 backlinks but still not referenced by Google
i've launch a website on wednesday (9 August)
There's already in 4 days
12,000 viewed pages
1,400 visits
20 to 25 backlincks
a sitemap.xml (130 URL)
english language / french language - url like ...
4
votes
2answers
103 views
Preventing search engines to crawl on specific folders
I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow:
...
4
votes
1answer
130 views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
4
votes
2answers
893 views
How do you disallow root in robots.txt, but allow a subdirectory?
Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?
4
votes
1answer
291 views
Does the line-ending format of robots.txt matter?
Simple question: Should I make sure to use Unix line endings for my robots.txt, or does it not matter?
4
votes
2answers
278 views
What is the best configuration for robots.txt
What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders?
Is there a general setup that should always be used?
4
votes
3answers
72 views
SEO Removal Time?
I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here.
I believe it will be the solution but what i still need to ...
4
votes
1answer
2k views
Google Webmaster Central tells me that robots is blocking access to the sitemap
This is my robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.mydomain.org/sitemap.xml.gz
But Google Webmaster Central tells me that robots is blocking ...
4
votes
3answers
205 views
Bad Bot blocking Revisited
I've read a lot about bad bot blocking, php scripts, .htaccess techniques, etc...
Is this a valid method?
Since .htacces can rewrite and send a bad bot a 403 deny or forward to something like spam ...
4
votes
3answers
2k views
In Robots.txt, how do I disallow all pages except for the index page?
I want google to index my index page, but do not index any other pages.
User-agent: *
Disallow: /
So far I have this. However...when I do this, Google does not index my index page. When I search ...
4
votes
5answers
107 views
removing robots.txt
Will removing robots.txt make the website 'searchable' again.
Curious if you had a robots.txt to avoid indexing, but changed your mind and wanted to get indexed.
4
votes
3answers
543 views
nofollow doesn't stop crawling?
In my website, most of the pages has a linkedin sharing button written in this way:
<li>
<a id="share_linkedin"
...
4
votes
2answers
80 views
robots.txt - just a guess about wild-card
so if I disallow tempPage, does it mean tempPage_1, temp_Page_2, tempPage_x are also disallowed? I tried to google this up, but I don't know...
4
votes
1answer
568 views
How do I deal with content scrapers? [duplicate]
Possible Duplicate:
How to protect SHTML pages from crawlers/spiders/scrapers?
My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling ...
4
votes
1answer
381 views
wget not respecting my robots.txt. Is there an interceptor?
I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little.
...
4
votes
1answer
332 views
Should I block search engines with robots.txt on my Facebook app?
There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
4
votes
3answers
265 views
Google indexing page with parameters but page is Disallowed in robots.txt
I have the following in robots.txt:
User-agent: *
Disallow: /refer.php
User-agent: NinjaBot
Allow: /
Sitemap: http://www.mysite.com/sitemap.xml
The refer.php file does various things depending on ...
3
votes
2answers
185 views
Robots.txt on one line
Can I combine robots.txt rules on one line, separated by whitespace?
Will this work?
User-agent: * Disallow: /
3
votes
3answers
4k views
How to stop certain urls from being indexed
When I type site:example.com (using my domain obviously),
I get several link errors showing up in the listing. Typically, they are of the form:
www.example.com/some/fixed/path/admin/unblockUser/11
...
3
votes
4answers
501 views
Hide a Subdomain from Google/Search Engine SEO Result?
I have a subomain which I do not want to be listed in any search engine result. Let say I have:
http://www.example.com
http://child.example.com
How can I hide all URLs of child.example.com domain ...
3
votes
4answers
93 views
Stopping Google index some web pages I have
I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?
3
votes
2answers
192 views
How can I allow search engines to index my invite only website in ruby on rails?
I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication:
before_filter ...
3
votes
5answers
2k views
Disallow robots.txt from being accessed in a browser but still accessible by spiders?
We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
3
votes
3answers
116 views
Do crawlers require a robots.txt file before indexing content?
Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?
3
votes
1answer
362 views
Can a sitemap index contain other sitemap indexes?
I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
3
votes
2answers
5k views
Can I invoke Google to check my robots.txt?
I read the answers in this question, but they still leave my question open: does Google cache robots.txt?
I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt.
...
3
votes
2answers
442 views
Should I prevent search engines indexing tag/category pages?
On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...
3
votes
3answers
303 views
PageRank: will links pointing to pages protected by robots.txt still count?
If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e.
...
3
votes
3answers
110 views
SEO consideration for duplicate sites
I am building a brochure-ware website for a company that sells products all across the world. They need the site to ask the user what region they are in before using the site; there are 5 regions. ...
3
votes
2answers
132 views
How to temporarely remove a website from search engines?
we are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain.
now we are being already referenced by bots (google, yahoo...) and we dont want ...
3
votes
2answers
125 views
Impact of blocking site search from googlebot, but listing content pages in sitemap.xml
We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
3
votes
2answers
196 views
Should I disallow robots on A/B test pages?
I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php.
Should I be putting in rules to my ...
3
votes
3answers
214 views
Is my robots.txt working as it should?
I want crawlers to have access to http://www.example.com but not http://www.example.com/
My robots.txt is as follows:
User-agent: *
Allow: /$
Disallow: /
My site is in Google search results, but I ...
3
votes
1answer
523 views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
1answer
318 views
Quick Question, robots.txt Disallow: /*/ does what exactly?
A SEO firm suggested changing the robots.txt to:
User-agent: *
Disallow: /*/
Allow: /ims/
I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
