Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
2answers
584 views

How much Google takes to deindex pages after disallowed from robots?

By accident I have let Google index lots of junk pages. Now I have added them to disallow in robots.txt. Will Google completely remove the pages after seeing the are disallowed from robots? Please ...
1
vote
2answers
120 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
1
vote
3answers
367 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
2answers
126 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
0
votes
1answer
290 views

Need help with mybb forum robots.txt file

I have installed the mybb forum in subdomain of my site. The link to my site is here. but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt ...
0
votes
1answer
60 views

How to use wildcards to remove print pages from search engines?

Our site has pages that are print-friendly, which I would like to remove from search engines. Here's an example: ...
3
votes
1answer
239 views

How to block FeedReader from fetching my content to their site?

As you all can see from the picture below, my site's content is duplicated by FeedReader and indexed at Google. When I clicked at the FeedReader link, it uses some sort of iFrame to draw content from ...
4
votes
2answers
2k views

Is my whole site is blocked by this Disallow: /

This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site. User-agent: Baiduspider Disallow: / User-agent: * Disallow: ...
1
vote
3answers
302 views

In WordPress, should I “Allow search engines to index this site” during development?

I am going to create a web page, and I will do it online. Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
2
votes
2answers
164 views

What does “Allow: /” mean in a robots.txt file?

Please help me to understand the following code in robots.txt: Allow: /
0
votes
1answer
114 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
0
votes
2answers
87 views

Pre-launch Disallow in robots.txt?

My team is launching a new site soon. Is it worth blocking spiders from indexing the landing page right now? What effects will this have on my long term SEO results. Is this question even ...
-2
votes
1answer
255 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...
0
votes
1answer
747 views

Google Webmaster Tools is showing incorrect warnings - blocked by robots.txt

I'm getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt when they are not. Has any one come across this or a way to resolve this before ? ...
2
votes
2answers
133 views

robots.txt: block all webpages except a few number of webpages?

I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all the crawlers? User-agent: * Disallow: / Allow: /$ Allow: /a/$ Allow: ...
5
votes
1answer
186 views

How to fix thousands of 404's?

The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" ...
1
vote
1answer
143 views

Robots.txt Disallow command

How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error. The URL which been crawled incorrectly is: ...
2
votes
2answers
1k views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
1
vote
2answers
409 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
2
votes
1answer
288 views

Is there a reason msnbot crawls in spikes?

I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
2
votes
2answers
49 views

Does Google crawl page when G+ logo is embedded?

I have a robots.txt file in the root to prevent search engines indexing any of my sites. I have a social media management application embedded on my site that includes all major social media channels ...
0
votes
2answers
48 views

robots.txt dissalow url containing string with a '/' at the end

I have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
1
vote
6answers
95 views

Uploading a non-finished website

I have a pretty basic question. I developed a neat little website which I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
1
vote
2answers
609 views

How to submit sitemap when your website has partial https? - Error: “Not in Domain”

My website is an eCommerce that is set up to do HTTP for the item browsing portion, but HTTPS for things like shopping cart, contact us, etc.. (anything that has forms on it). I've submitted my ...
0
votes
1answer
171 views

Wlll links into pages that are restricted by Disallow rules in robots.txt help SEO?

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable. If I get incoming ...
1
vote
1answer
255 views

RSS feeds and robots.txt

I have RSS feeds on my site. I've decided to follow Stack Exchange and disallow my RSS feeds in the robots.txt. I don't want search engine to display the RSS feed page to people, that's not really a ...
2
votes
3answers
225 views

Is it safe to block redirected (but still linked) URLs with robots.txt?

I have a website that has all URLs optimized and 301 redirected from nasty URLs to clean ones. However, everywhere throughout the site the unclean URLs are linked in menus, content, products, etc. ...
3
votes
3answers
126 views

robots.txt dissalow url containing string with a '/' at the end

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
0
votes
2answers
148 views

Google Webmaster Tools robots test not working

Within Webmaster Tools I have supplied my test content: User-agent: * Disallow:/admin/ Disallow: /tag/ When I specify the URL to test against, for example: http://www.site.com/tag/ It gives me ...
1
vote
1answer
84 views

Best way to prevent Google from indexing a directory [duplicate]

I've researched many methods on how to prevent Google/other search engines from crawling a specific directory. The two most popular ones I've seen are: Adding it into the robots.txt file: Disallow: ...
2
votes
2answers
764 views

Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?

As I understand, having repetitive content is a detriment to search engine placement. Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
2
votes
1answer
68 views

To allow or disallow alphabetical content directory pages?

We have a content that publishes fresh content 2-3 times a day. The site has category pages for tags and each alphabet. They both list the title with links to all the content within that category or ...
3
votes
2answers
411 views

Grapeshot crawler ignoring robots.txt

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad ...
0
votes
1answer
127 views

Which token from a long User-Agent should I use in robots.txt?

The definition of User-Agent states that several tokens can be included, as deemed necessary by the client. I want to block certain bots via robots.txt and I am confused as to which part of the ...
0
votes
1answer
45 views

My first robots.txt [closed]

I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each other and I do NOT want that ...
2
votes
2answers
2k views

Robots.txt and Regular Expressions

If I place a regular expression in robots.txt will it be readable for the web crawlers?
1
vote
3answers
1k views

Robots.txt never downloaded but some blocked URLs in GWT

There is something I don't understand in Google Webmaster Tools (GWT) for my WordPress site. In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked ...
5
votes
1answer
86 views

Do we need to block repeated pages content for SEO relevance

I have multiple purchase pages with the same content like: product1red.php product2green.php Should I block them with robots.txt ?
2
votes
2answers
215 views

How to block pagination when the URL is just one number?

I'm trying to avoid duplicate content on a website by blocking pagination via robots.txt, but it turns out the URL doesn't have a parameter passed into the URL I can simply block, it's just a number. ...
2
votes
1answer
140 views

Robots.txt to block a parameter instead of a directory

It's my understanding that URLs are of the format example.com/something/somethingelse. It's my understanding that parameters follow the URL with a question mark example.com?l=fr_FR My CMS website ...
1
vote
1answer
472 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
1
vote
1answer
142 views

Google lists redirected URL instead of target URL

We've recently moved much of our site to a Web CMS which published to www1 rather than the old www. To clean up the search results, we've added an exclusion in robots.txt for the old hostname and ...
2
votes
1answer
111 views

What is the difference between Disallow and Allow?

I had 2 questions in my mind: what is the difference between Allow and Disallow? which one should I use to allow Google bot for AdSense? User-agent: Mediapartners-Google Disallow: User-agent: ...
1
vote
1answer
61 views

Removed pages from website - do I need to tell Google or will they fall off in time anyway?

This is a follow up question from here: Duplicate title tags due to translations Since asking the question I have removed the duplicated content that was being generated by language/translations on ...
1
vote
1answer
38 views

Cascading matches in robots.txt

I was wondering if it's possible to let bots match two groups of User-Agent: blocks. I have this block of robots.txt User-Agent: AhrefsBot # https://ahrefs.com/robot/ Crawl-Delay: 5 User-agent: * ...
1
vote
2answers
175 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
2
votes
2answers
1k views

Correct usage of robots.txt file (“Googlebot-Image” and “Mediapartners-Google”)

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress ...
0
votes
1answer
240 views

Sitemaps: rel=“alternate” hreflang=“x” naming

I'm creating a Sitemap for rel="alternate" hreflang="x", but no where in Google's guidelines do they mention how it should be named. Also can we reference it separately in the robots.txt file, or does ...
3
votes
3answers
7k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
3
votes
1answer
1k views

Does Google respect a redirect header for robots.txt to a different file name?

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found". Does this mean that, if I create a 301 redirect for ...