Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

1
vote
2answers
214 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
1
vote
2answers
739 views

SEO: disallowing Google from indexing forms?

From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not? I mean my forms hardly ever contains keyword/keyphrases. So I'm wondering what's the point of letting ...
0
votes
1answer
2k views

SEO not indexing my site: “A description for this result is not available because of this site's robots.txt – learn more.” [duplicate]

When googling my site I get following text under the result for my domain: "A description for this result is not available because of this site's robots.txt – learn more." My robots.txt, ...
0
votes
1answer
386 views

How to secure robots.txt file?

I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: * Disallow: */* Sitemap: ...
0
votes
2answers
563 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
0
votes
1answer
20 views

Robots.txt problem in subdirectory website (locale) and “/” character

We have two sites mainsite.com and mainsite.com/de where /de shows the german site. We rely on 1 robots.txt file. It contains all the disallow links for mainsite.com that we set-up and copied it to ...
0
votes
1answer
40 views

Why is Google saying that the User-Agent line is invalid?

I have the following robots.txt file: User-agent: * Sitemap: https://m.mysite.com.au/sitemap.xml and when I use the Webmaster Tools to validate it I get it showing that the user agent line is ...
0
votes
3answers
68 views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=
0
votes
1answer
37 views

Does the following robots.txt block all pages

I want to knew if the following robots.txt file block the whole pages from being indexed or not. User-agent: * Disallow: Disallow: /cgi-bin/
0
votes
1answer
31 views

surprising visitors on my wordpress site? [closed]

Hi I have a blog in wordpress where I plan to put some c programming tutorials never got time to finish it actually. but still site is live. Yesterday I received an email from my hosting that I might ...
0
votes
1answer
25 views

Comment Out Link Shows in Webmaster

I had "Privacy Policy" link on my old website. When I launched my new website, I put the link in footer but disallowed that to robots.txt by adding this code: <?php /*?><div ...
0
votes
1answer
387 views

How my robots.txt should look like for a single page app

I can understand how to disallow bots to crawl some pages/folder in normal application. For example for google-bot it is nicely described here. But what should I do if I have a single page ...
0
votes
1answer
170 views

Google say: robots.txt not valid for subdomains?

I'm very confused about Google's robots.txt documentation Examples of valid robots.txt URLs: Not valid for http://other.example.com/ https://example.com/ http://example.com:8181/ ...
0
votes
1answer
302 views

How to prevent webpages translated with Google Translate Plugin from being indexed?

I've been reading the information here https://support.google.com/translate/#2534535 about the Google Translate plugin. At first I thought that if I used the plugin my site wouldn't be indexed, but ...
0
votes
3answers
152 views

Robots are still disallowed even without a robots.txt file

I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has ...
0
votes
2answers
149 views

Will robots.txt block crawling things like menu and footer links?

If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
0
votes
2answers
934 views

Baiduspider is crawling my site even when forbidden by robots.txt, how do I prevent it? [duplicate]

My site has heavy traffic because some bot. I checked access_log, some bot Baiduspider go to my site 10-20 times per minute. I do not need Chinese traffic. I have searched and read ...
0
votes
2answers
83 views

What is the empirical way to tell that Google has removed a URL from the index?

After merging two sites, one with ~40 URLs and one with ~700 URLs (due to a forum) the result was that overall traffic decreased by ~50%. I want to get the traffic back, as it was organic and a ...
0
votes
1answer
112 views

Hiding page content from Google when it is duplicated across several sites

I would like to hide some content from Google. Because a client has multiple websites with the same content then he gets a warning about duplicate content on some of them (multiple countries, same ...
0
votes
1answer
473 views

Google Webmaster Tool: “Severe health issues are found on your site” [closed]

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...
0
votes
1answer
124 views

How to block certain URL from robots.txt file? [closed]

This is simple question but also I have a confusion. Below is the code which I wrote in my robots.txt: User-Agent: * Disallow: /phpmyadmin/ Disallow: /login/ Disallow: /newsletter/ Disallow: ...
0
votes
2answers
70 views

Confused for writing robots.txt file

I am getting DNS error from last few 5 days in my Google Webmaster Tools. That's the reason I would like to confirm my robots.txt file. Is the below robots.txt file is correct or not? User-agent: * ...
0
votes
1answer
344 views

Noindex, Nofollow affect “Google Custom Search” results?

I am using Google Custom Search on my site, if I set pages to "noindex" or "nofollow", does it also affect the custom search results on my site?
0
votes
1answer
407 views

Is “robots.txt” necessarily needed? [duplicate]

Possible Duplicate: Do I need robots.txt? From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
0
votes
1answer
26 views

URLs with parameters & canonical tags. Block them via robots.txt vs GSC URL parameters?

I have some URLs with parameters, the parameters aren't of the filtering kind, they don't create millions of duplicates. The parameters are simple being used for tracking purpose, content doesn't ...
0
votes
2answers
31 views

User-agent line needed in robots.txt?

I'm about to deploy a website for testing and I want to let all the indexing bots to not index my site during this phase. According to Google Webmasters Help section apparently you only need the ...
0
votes
1answer
70 views

Am I doing .htaccess right to block google, bing and alexa bots?

I do not need bots at all. I bought my traffic and I do not want my sites to show up on search engine or anything. So I added this: <IfModule mod_setenvif.c> # remove most popular bot ...
0
votes
1answer
98 views

Google Bot spending way too much following old 301 redirects

We changed our URL structure a couple of years ago but we still have a lot of backlinks pointing to old URLs. This causes Google bot to keep crawling old paths again and again and spending a lot of ...
0
votes
1answer
76 views

Will adding X-Robots-Tag:noindex remove an already indexed page from searchresults

I have a site that has already been indexed by google and shows in the search results. After it had already been indexed I added the X-Robots-Tag:noindex header to all responses. Is that enough for ...
0
votes
1answer
58 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
0
votes
1answer
60 views

Remove page from Search Result Bunch [duplicate]

When I do a branded search for a website the structure shows like ( see image ). I would love to remove Privacy Policy page or Manipulate the structure I see now ( showing pages I want to ). How I can ...
0
votes
1answer
271 views

Clicks counting and crawler bots

I am currently running a small affiliate-program for Facebook users. We use an auto-poster to publish links to fan pages. Every hit is stored in our database and we have included a 24 hour reload ...
0
votes
1answer
65 views

How to use wildcards to remove print pages from search engines?

Our site has pages that are print-friendly, which I would like to remove from search engines. Here's an example: ...
0
votes
1answer
320 views

Need help with mybb forum robots.txt file

I have installed the mybb forum in subdomain of my site. The link to my site is here. but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt ...
0
votes
1answer
144 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
0
votes
2answers
52 views

robots.txt dissalow url containing string with a '/' at the end

I have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
0
votes
1answer
193 views

Wlll links into pages that are restricted by Disallow rules in robots.txt help SEO?

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable. If I get incoming ...
0
votes
1answer
51 views

My first robots.txt [closed]

I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each other and I do NOT want that ...
0
votes
1answer
334 views

Sitemaps: rel=“alternate” hreflang=“x” naming

I'm creating a Sitemap for rel="alternate" hreflang="x", but no where in Google's guidelines do they mention how it should be named. Also can we reference it separately in the robots.txt file, or does ...
0
votes
2answers
3k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
0
votes
1answer
52 views

Do I need disallow other websites directory in robots.txt?

I have a hosting which has several websites in www folder. I set up a WordPress website in root directory for main domain. Do I need disallow other websites directory in robots.txt? I think these ...
0
votes
1answer
100 views

Searches for “more information about text formats”

I have a site using Drupal as CMS, and got "infected" badly with spammer bots. Moreover, recently I signed up for Google's Webmaster Tools, and I see that one of the top search queries that result in ...
0
votes
1answer
67 views

Google indexed page a day before also reflecting in search but today everything vanish

We had robots.txt which disallow all robots as we were in development. We are live now. We change robots.txt as per our requirement a day before. Submit indexes using Google Webmaster Tools index ...
0
votes
2answers
155 views

Google Webmaster Tools robots test not working

Within Webmaster Tools I have supplied my test content: User-agent: * Disallow:/admin/ Disallow: /tag/ When I specify the URL to test against, for example: http://www.site.com/tag/ It gives me ...
0
votes
1answer
79 views

How to overcome duplicate URLS

I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls. From: /psd-layouts.html to /psd-web-layouts.html Now my question is that, I had ...
0
votes
2answers
132 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
0
votes
1answer
781 views

Google Webmaster Tools is showing incorrect warnings - blocked by robots.txt

I'm getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt when they are not. Has any one come across this or a way to resolve this before ? ...
0
votes
1answer
55 views

If I blocked a directory, is a 301 error expected?

I blocked a directory in my site that is not secret but meant for internal discussions that are not relevant for search engine results. After running "Search as Google" at Google Webmaster Tools, the ...
0
votes
1answer
286 views

wordpress robots.txt is this reasonable?

this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms. all the page urls like http://mydomain.com/it/page1, ...
0
votes
1answer
186 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion) I am building other sites, ...