Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

3
votes
2answers
13k views

Allow all robots robots.txt

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
6
votes
4answers
5k views

In Robots.txt, how do I disallow all pages except for the index page?

I want google to index my index page, but do not index any other pages. User-agent: * Disallow: / So far I have this. However...when I do this, Google does not index my index page. When I search ...
1
vote
1answer
384 views

Preventing Twitterbot to access website

I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And ...
3
votes
2answers
102 views

Sitemap specified in robots.txt not found by Google

I would like Google to find my sitemap.xml via robots.txt, rather than having to manually upload it. My robots.txt looks like this: User-agent: * Disallow: Sitemap: ...
1
vote
1answer
356 views

“Syntax not understood” error in Google Webmaster Tools robots.txt analysis when I use RewriteRule

In order to redirect my www.site.com to non-www, I employed this bit of code found somewhere on the web. RewriteCond %{HTTP_HOST} ^www.website.com [NC] RewriteRule ^(.*)$ http://website.com/$1 ...
4
votes
3answers
317 views

Will the content of a PDF on our website affect SEO?

I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings? We want to put some PDFs on our website but don't want to see our results take a ...
2
votes
2answers
602 views

How much Google takes to deindex pages after disallowed from robots?

By accident I have let Google index lots of junk pages. Now I have added them to disallow in robots.txt. Will Google completely remove the pages after seeing the are disallowed from robots? Please ...
1
vote
2answers
129 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
1
vote
3answers
379 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
2answers
128 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
0
votes
1answer
296 views

Need help with mybb forum robots.txt file

I have installed the mybb forum in subdomain of my site. The link to my site is here. but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt ...
0
votes
1answer
61 views

How to use wildcards to remove print pages from search engines?

Our site has pages that are print-friendly, which I would like to remove from search engines. Here's an example: ...
3
votes
1answer
241 views

How to block FeedReader from fetching my content to their site?

As you all can see from the picture below, my site's content is duplicated by FeedReader and indexed at Google. When I clicked at the FeedReader link, it uses some sort of iFrame to draw content from ...
4
votes
2answers
2k views

Is my whole site is blocked by this Disallow: /

This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site. User-agent: Baiduspider Disallow: / User-agent: * Disallow: ...
1
vote
3answers
339 views

In WordPress, should I “Allow search engines to index this site” during development?

I am going to create a web page, and I will do it online. Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
2
votes
2answers
177 views

What does “Allow: /” mean in a robots.txt file?

Please help me to understand the following code in robots.txt: Allow: /
0
votes
1answer
122 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
0
votes
2answers
87 views

Pre-launch Disallow in robots.txt?

My team is launching a new site soon. Is it worth blocking spiders from indexing the landing page right now? What effects will this have on my long term SEO results. Is this question even ...
-2
votes
1answer
259 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...
0
votes
1answer
758 views

Google Webmaster Tools is showing incorrect warnings - blocked by robots.txt

I'm getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt when they are not. Has any one come across this or a way to resolve this before ? ...
2
votes
2answers
139 views

robots.txt: block all webpages except a few number of webpages?

I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all the crawlers? User-agent: * Disallow: / Allow: /$ Allow: /a/$ Allow: ...
5
votes
1answer
190 views

How to fix thousands of 404's?

The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" ...
1
vote
1answer
149 views

Robots.txt Disallow command

How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error. The URL which been crawled incorrectly is: ...
2
votes
2answers
1k views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
1
vote
2answers
419 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
2
votes
1answer
296 views

Is there a reason msnbot crawls in spikes?

I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
2
votes
2answers
49 views

Does Google crawl page when G+ logo is embedded?

I have a robots.txt file in the root to prevent search engines indexing any of my sites. I have a social media management application embedded on my site that includes all major social media channels ...
0
votes
2answers
49 views

robots.txt dissalow url containing string with a '/' at the end

I have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
1
vote
6answers
95 views

Uploading a non-finished website

I have a pretty basic question. I developed a neat little website which I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
1
vote
2answers
611 views

How to submit sitemap when your website has partial https? - Error: “Not in Domain”

My website is an eCommerce that is set up to do HTTP for the item browsing portion, but HTTPS for things like shopping cart, contact us, etc.. (anything that has forms on it). I've submitted my ...
0
votes
1answer
174 views

Wlll links into pages that are restricted by Disallow rules in robots.txt help SEO?

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable. If I get incoming ...
1
vote
1answer
280 views

RSS feeds and robots.txt

I have RSS feeds on my site. I've decided to follow Stack Exchange and disallow my RSS feeds in the robots.txt. I don't want search engine to display the RSS feed page to people, that's not really a ...
2
votes
3answers
239 views

Is it safe to block redirected (but still linked) URLs with robots.txt?

I have a website that has all URLs optimized and 301 redirected from nasty URLs to clean ones. However, everywhere throughout the site the unclean URLs are linked in menus, content, products, etc. ...
3
votes
3answers
127 views

robots.txt dissalow url containing string with a '/' at the end

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
0
votes
2answers
151 views

Google Webmaster Tools robots test not working

Within Webmaster Tools I have supplied my test content: User-agent: * Disallow:/admin/ Disallow: /tag/ When I specify the URL to test against, for example: http://www.site.com/tag/ It gives me ...
1
vote
1answer
86 views

Best way to prevent Google from indexing a directory [duplicate]

I've researched many methods on how to prevent Google/other search engines from crawling a specific directory. The two most popular ones I've seen are: Adding it into the robots.txt file: Disallow: ...
3
votes
2answers
792 views

Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?

As I understand, having repetitive content is a detriment to search engine placement. Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
2
votes
1answer
70 views

To allow or disallow alphabetical content directory pages?

We have a content that publishes fresh content 2-3 times a day. The site has category pages for tags and each alphabet. They both list the title with links to all the content within that category or ...
3
votes
2answers
417 views

Grapeshot crawler ignoring robots.txt

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad ...
0
votes
1answer
131 views

Which token from a long User-Agent should I use in robots.txt?

The definition of User-Agent states that several tokens can be included, as deemed necessary by the client. I want to block certain bots via robots.txt and I am confused as to which part of the ...
0
votes
1answer
46 views

My first robots.txt [closed]

I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each other and I do NOT want that ...
2
votes
2answers
2k views

Robots.txt and Regular Expressions

If I place a regular expression in robots.txt will it be readable for the web crawlers?
1
vote
3answers
1k views

Robots.txt never downloaded but some blocked URLs in GWT

There is something I don't understand in Google Webmaster Tools (GWT) for my WordPress site. In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked ...
5
votes
1answer
86 views

Do we need to block repeated pages content for SEO relevance

I have multiple purchase pages with the same content like: product1red.php product2green.php Should I block them with robots.txt ?
2
votes
2answers
225 views

How to block pagination when the URL is just one number?

I'm trying to avoid duplicate content on a website by blocking pagination via robots.txt, but it turns out the URL doesn't have a parameter passed into the URL I can simply block, it's just a number. ...
2
votes
1answer
143 views

Robots.txt to block a parameter instead of a directory

It's my understanding that URLs are of the format example.com/something/somethingelse. It's my understanding that parameters follow the URL with a question mark example.com?l=fr_FR My CMS website ...
1
vote
1answer
486 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
1
vote
1answer
145 views

Google lists redirected URL instead of target URL

We've recently moved much of our site to a Web CMS which published to www1 rather than the old www. To clean up the search results, we've added an exclusion in robots.txt for the old hostname and ...
2
votes
1answer
112 views

What is the difference between Disallow and Allow?

I had 2 questions in my mind: what is the difference between Allow and Disallow? which one should I use to allow Google bot for AdSense? User-agent: Mediapartners-Google Disallow: User-agent: ...
1
vote
1answer
61 views

Removed pages from website - do I need to tell Google or will they fall off in time anyway?

This is a follow up question from here: Duplicate title tags due to translations Since asking the question I have removed the duplicated content that was being generated by language/translations on ...