Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
2answers
50 views

Does Google crawl page when G+ logo is embedded?

I have a robots.txt file in the root to prevent search engines indexing any of my sites. I have a social media management application embedded on my site that includes all major social media channels ...
2
votes
1answer
114 views

What is the difference between Disallow and Allow?

I had 2 questions in my mind: what is the difference between Allow and Disallow? which one should I use to allow Google bot for AdSense? User-agent: Mediapartners-Google Disallow: User-agent: ...
2
votes
1answer
70 views

To allow or disallow alphabetical content directory pages?

We have a content that publishes fresh content 2-3 times a day. The site has category pages for tags and each alphabet. They both list the title with links to all the content within that category or ...
2
votes
1answer
717 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
2
votes
1answer
58 views

what should be my initial address for sitemap and robots.txt if I'm using redirection

What should be my initial adress when using this site: http://www.xml-sitemaps.com? I have a folder inside my root where I have all my files. For example I have domain.com and my start directory is ...
2
votes
2answers
134 views

Prevent crawler that doesn't honoring robots.txt

I have some problem, when I try to write robots.txt for my site ... I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
2
votes
2answers
189 views

How to get search engines to properly index an ajax driven search page

I have an ajax-driven search page that will allow users to search through a large collection of records. Each search result points to index.php?id=xyz (where xyz is the id of the record). The initial ...
2
votes
1answer
560 views

Edit robots.txt on google sites

Getting many 404 errors on my website for the following URLs: /system/app/pages/admin/revisions* I was going to put Disallow in robots.txt, but it seems there is no way to edit robots.txt in google ...
2
votes
1answer
419 views

HTTP 303 redirection and robots.txt

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
48 views

What is the difference between Disallow: /? and Disallow: /*?* in robots.txt?

We have many parameters in our URL and I wonder what will do the parameters pasted below. Will they disallow all the parameters after ?? Please tell me what does this mean: Disallow: /? Disallow: ...
2
votes
1answer
259 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
2
votes
1answer
316 views

How to Keep “Good” Bots That Ignore Robots & Nofollow From Getting Into Honeypots?

We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
2
votes
1answer
64 views

Robots and Sitemaps

I have some problems regarding indexing. Google indexed so much in the list. Now I wanted to remove everything from my site. If I have disallow all in robots.txt and send my sitemap.xml. Would ...
2
votes
3answers
343 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
2
votes
1answer
178 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
2
votes
1answer
248 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: ...
2
votes
2answers
789 views

Restrict all robots except Googlebot and Yandexbot

I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages. All other bots will not be accessed my pages. Is this robots.txt ...
2
votes
1answer
308 views

Is there a reason msnbot crawls in spikes?

I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
2
votes
1answer
77 views

GoogleBot can't reach my site

For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
2answers
112 views

Robots.txt Blocking Blog Tags?

Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled? User-agent: ...
2
votes
2answers
1k views

Will 301 redirection to a Google Indexed URL can remove the URL from Google index?

I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
2
votes
2answers
237 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
2
votes
1answer
351 views

Googlebot can't access my site when crawling from rootdomain

I can't explain why I get this message for my rootdomain result in Google : trocmalin.com/ A description for this result is not available because of this site's robots.txt – learn more. ...
2
votes
1answer
269 views

What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]

Possible Duplicate: How to Remove URLs from Google Search Engine In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
2
votes
2answers
135 views

Robots.txt syntax -> Can this be done?

When creating a robots.txt, are the following the same? # Slurp User-agent: Slurp Disallow: / and # Slurp User-agent: Slurp Disallow: / It seems like for someone disallowing a large number of ...
2
votes
1answer
154 views

How can I let robots access my site?

I have a webpage on my university's server, which means I cannot access the root directory and change files there. Is there some other way to give access to robots to my page? Thanks.
2
votes
3answers
928 views

What is a good robots.txt for WP?

What is the "best" setup for robots.txt? I'm using the following permalink structure in Wordpress: /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ...
2
votes
3answers
182 views

Should I use both Robots.txt and META NAME=“ROBOTS”?

I am currently using the Robots.txt file (the standard one of Joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior. Is there a reason to ...
2
votes
2answers
48 views

Question about duplicate groups (agents) in robots.txt

I'm looking at a corner case with robots.txt, and am curious if you have any guidance. Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file. I've ...
2
votes
2answers
305 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
2
votes
2answers
234 views

How to block pagination when the URL is just one number?

I'm trying to avoid duplicate content on a website by blocking pagination via robots.txt, but it turns out the URL doesn't have a parameter passed into the URL I can simply block, it's just a number. ...
1
vote
2answers
306 views

robots.txt ignored

I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories. By default it had Disallow: ...
1
vote
2answers
137 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
1
vote
3answers
392 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
6answers
96 views

Uploading a non-finished website

I have a pretty basic question. I developed a neat little website which I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
1
vote
3answers
217 views

Right way to allow all access to website in robots.txt?

user-agent: * disallow: Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?
1
vote
3answers
463 views

Preventing Spiders on One Subdomain?

We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, ...
1
vote
4answers
1k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
2answers
120 views

Allow users to access my beta subdomain but pass link juice to the main site

My site can be accessed by either http://example.com or http://beta.example.com. The sub domain beta is the same site but a different version and I want users to be able to use the beta but any links ...
1
vote
3answers
3k views

How to fix robots.txt file error in my GWT?

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file. My blog’s robots.txt file ...
1
vote
3answers
120 views

robots.txt file Help needed

What if I put same robots.txt file in multiple directories and folders? User-agent: * Disallow: /otherdirname/dirname2/ Can it'll affect anyhow ?? And how much time it generally takes to refine ...
1
vote
2answers
134 views

Problem with my robots.txt

I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
1
vote
1answer
1k views

Different robots.txt for two different domains point to same folder

I have the following two domains: domain.com test.domain.com Both point to same folder which is "public_html". What I want is a different robots.txt file for each domain. So when someone ...
1
vote
1answer
35 views

Does repetition of same rules for each major crawler bot makes sense in robots.txt?

SEO is a bit of arcane topic and sometimes I encounter strange practics which seemingly aren't backed up by any advice but still persists. Now I mean something like this: User-agent: Googlebot ...
1
vote
3answers
100 views

How much time for robots.txt to remove my website from google

I created a robots.txt file in my root but the website is still indexed by Google. How much time for robots.txt to remove my website from Google?
1
vote
3answers
365 views

In WordPress, should I “Allow search engines to index this site” during development?

I am going to create a web page, and I will do it online. Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
1
vote
2answers
321 views

Can search engine robots read file with permission 640?

I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing www.mysite.com/robots.txt. As per the following google group ...
1
vote
2answers
535 views

Robots.txt not working because my site is still searchable

I put a robots.txt in my server This is the code that I putted in User-agent: * Disallow: / I just want my entire site won't be searchable in google. I also put in the NO INDEX, NO FOLLOW on the ...
1
vote
5answers
498 views

Should I add a “nofollow” attribute to download links, or disallow the URLs in robots.txt?

I have a download link very similar to Opera's one - it's just a script that sends the file. It doesn't have an extension and there's no obvious way to tell that it's actually a download link. So ...
1
vote
2answers
115 views

Where should the robots.txt file be?

If I have a subdomain: subdir.a.com that redirects to a.com/myfolder/ should the robots.txt file be in: subdir.a.com/robots.txt or in a.com/myfolder/robots.txt