Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

1
vote
1answer
38 views

Prohibit search engines before going live

I want to prohibit search engines to index my web site before going live with it. If I simple do a robots.txt with: User-agent: * Disallow: / Would that cause any issues later when I decide to ...
5
votes
3answers
93 views

SEO Removal Time?

I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here. I believe it will be the solution but what i still need to ...
0
votes
1answer
78 views

How to overcome duplicate URLS

I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls. From: /psd-layouts.html to /psd-web-layouts.html Now my question is that, I had ...
0
votes
2answers
130 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
48 views

Question about duplicate groups (agents) in robots.txt

I'm looking at a corner case with robots.txt, and am curious if you have any guidance. Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file. I've ...
6
votes
4answers
795 views

Robots.txt vs Sitemap — Who wins in a Conflict

If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
2
votes
2answers
1k views

Will 301 redirection to a Google Indexed URL can remove the URL from Google index?

I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
2
votes
2answers
240 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
2
votes
2answers
308 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
2
votes
2answers
278 views

Robots.txt on one line

Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /
5
votes
1answer
115 views

Handling SEO for Infinite pages that cause external slow API calls

I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
1
vote
1answer
522 views

Block access to robots.txt file .htaccess

I read in this forum that this .htaccess code should block access to the robot.txt file: RewriteCond %{HTTP_USER_AGENT} ^Mozilla RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely) RewriteRule ...
4
votes
2answers
564 views

Should I prevent search engines indexing tag/category pages?

On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...
4
votes
2answers
353 views

Impact of blocking site search from googlebot, but listing content pages in sitemap.xml

We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
3
votes
1answer
2k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
0
votes
1answer
129 views

Hiding a particulat page from search engines not to index [duplicate]

Possible Duplicate: Blocking Search Engines to Crawl Parts of a Website? I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
6
votes
4answers
4k views

Hide a Subdomain from Google/Search Engine SEO Result?

I have a subomain which I do not want to be listed in any search engine result. Let say I have: http://www.example.com http://child.example.com How can I hide all URLs of child.example.com domain ...
3
votes
3answers
345 views

Is my robots.txt working as it should?

I want crawlers to have access to http://www.example.com but not http://www.example.com/ My robots.txt is as follows: User-agent: * Allow: /$ Disallow: / My site is in Google search results, but I ...
3
votes
3answers
690 views

Google is indexing unintended content that is either unpublished or a secret part of site

Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that I'm working on - www.domain.com/m/ <-- note the M. How is that even possible? Only thing I can ...
1
vote
2answers
730 views

SEO: disallowing Google from indexing forms?

From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not? I mean my forms hardly ever contains keyword/keyphrases. So I'm wondering what's the point of letting ...
2
votes
1answer
351 views

Googlebot can't access my site when crawling from rootdomain

I can't explain why I get this message for my rootdomain result in Google : trocmalin.com/ A description for this result is not available because of this site's robots.txt – learn more. ...
0
votes
1answer
2k views

SEO not indexing my site: “A description for this result is not available because of this site's robots.txt – learn more.” [duplicate]

When googling my site I get following text under the result for my domain: "A description for this result is not available because of this site's robots.txt – learn more." My robots.txt, ...
4
votes
1answer
662 views

How do I deal with content scrapers? [duplicate]

Possible Duplicate: How to protect SHTML pages from crawlers/spiders/scrapers? My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling ...
4
votes
2answers
166 views

Preventing search engines to crawl on specific folders

I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow: ...
2
votes
2answers
685 views

Network unreachable: robots.txt unreachable

I am trying to add a valid sitemap to Google Webmaster. Yet, it says: Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at ...
4
votes
3answers
328 views

Bad Bot blocking Revisited

I've read a lot about bad bot blocking, php scripts, .htaccess techniques, etc... Is this a valid method? Since .htacces can rewrite and send a bad bot a 403 deny or forward to something like spam ...
0
votes
0answers
451 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
2
votes
2answers
135 views

Prevent crawler that doesn't honoring robots.txt

I have some problem, when I try to write robots.txt for my site ... I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
1
vote
2answers
328 views

Can search engine robots read file with permission 640?

I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing www.mysite.com/robots.txt. As per the following google group ...
1
vote
1answer
188 views

What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?

I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
1
vote
1answer
44 views

If I block a page in robots.txt would I still be able to see the page in my browser? [closed]

What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?
2
votes
3answers
300 views

Hiding php includes from search spiders?

Quick and simple question. I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are ...
0
votes
1answer
383 views

How to secure robots.txt file?

I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: * Disallow: */* Sitemap: ...
1
vote
1answer
149 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
1
vote
5answers
510 views

Should I add a “nofollow” attribute to download links, or disallow the URLs in robots.txt?

I have a download link very similar to Opera's one - it's just a script that sends the file. It doesn't have an extension and there's no obvious way to tell that it's actually a download link. So ...
0
votes
1answer
55 views

If I blocked a directory, is a 301 error expected?

I blocked a directory in my site that is not secret but meant for internal discussions that are not relevant for search engine results. After running "Search as Google" at Google Webmaster Tools, the ...
1
vote
4answers
773 views

Can I include a robots meta tag outside of the head in HTML snippets indeded to be AJAXed?

I have a number of files in my site which are not intended for independent viewing, but rather to be AJAXed into content within the site. They obviously don't meet HTML standards (no body, head, etc.) ...
2
votes
2answers
189 views

How to get search engines to properly index an ajax driven search page

I have an ajax-driven search page that will allow users to search through a large collection of records. Each search result points to index.php?id=xyz (where xyz is the id of the record). The initial ...
3
votes
3answers
138 views

SEO consideration for duplicate sites

I am building a brochure-ware website for a company that sells products all across the world. They need the site to ask the user what region they are in before using the site; there are 5 regions. ...
1
vote
3answers
120 views

robots.txt file Help needed

What if I put same robots.txt file in multiple directories and folders? User-agent: * Disallow: /otherdirname/dirname2/ Can it'll affect anyhow ?? And how much time it generally takes to refine ...
7
votes
4answers
841 views

robots.txt not updated

I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
5
votes
2answers
264 views

Robots.txt and pattern matching

Adding this to my robots.txt User-agent: * Disallow: /*action=*$ How does robots not recognizing wild cards handle this?
5
votes
2answers
3k views

How do you disallow root in robots.txt, but allow a subdirectory?

Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?
1
vote
0answers
14 views

Mobile site robots.txt [duplicate]

Possible Duplicate: Should I block indexing of my mobile site? Should I allow webcrawlers to index the mobile versions of my websites? Is that a good practice from a SEO perspective? ...
10
votes
3answers
392 views

Robots.txt: do I need to disallow a page which is not linked anywhere?

There are some pages on my websites that I want the user to be able to visit only if I give him/her the url. If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
3
votes
1answer
674 views

How do I disallow robots to access links on a certain port?

I have a private web server set up on a non-conventional port (say 6677). All people can have access to the whole site's urls from 6677. The links by mysite:6677 are also indexed by Google, which is ...
1
vote
2answers
222 views

Robots.txt and “Bad” Robots

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
6
votes
3answers
325 views

Does loading images from a server that disallows robots affect SEO?

Lets say a page on example.com has all of its images coming from img.example.com: <img src="http://img.example.com/keyword.jpg" alt="keyword"/> while robots.txt on img.example.com disallows ...
1
vote
2answers
911 views

robots.txt, how effective is it and how long does it take?

We recently updated the site to a single page site using jQuery to slide between "pages". So we now have only index.php. When you search the company on engines such as Google, you get the site and a ...
1
vote
1answer
186 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...