Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
2
votes
2answers
19 views
What is the correct way to write my “robots.txt” file?
I have written the following code inside my robots.txt file:
User-Agent: Googlebot
Disallow:
User-agent: Mediapartners-Google
Disallow:
Sitemap: http://example.com/sitemap.xml
Is ...
4
votes
3answers
6k views
Can I invoke Google to check my robots.txt?
I read the answers in this question, but they still leave my question open: does Google cache robots.txt?
I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt.
...
2
votes
4answers
51 views
Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]
I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
0
votes
2answers
25 views
Google Webmaster Tools robots test not working
Within Webmaster Tools I have supplied my test content:
User-agent: *Disallow:/admin/
Disallow: /tag/
When I specify the URL to test against, for example:
http://www.site.com/tag/
It gives me ...
3
votes
3answers
4k views
How to stop certain urls from being indexed
When I type site:example.com (using my domain obviously),
I get several link errors showing up in the listing. Typically, they are of the form:
www.example.com/some/fixed/path/admin/unblockUser/11
...
2
votes
1answer
30 views
Why does my sitemap have a “.gz” extension, and how can I edit it?
An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
But other WordPress blogs have lots of ...
1
vote
1answer
20 views
To allow or disallow alphabetical content directory pages?
We have a content that publishes fresh content 2-3 times a day. The site has category pages for tags and each alphabet. They both list the title with links to all the content within that category or ...
6
votes
1answer
329 views
Should I block Wordpress archive pages from search engines?
I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives.
Google has indexed the site fully.
Since the archives currently display full posts I've thought that I shouldn't ...
1
vote
1answer
420 views
Robots.txt practices with .htaccess redirections (inherits)
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place.
We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
2answers
47 views
Robots.txt disallowing URLs
I need to disallow some URLs on my site but I am not sure how to do that. I have a site that has products and reviews. When someone makes a review, the site generates a URL automatically like this:
...
2
votes
2answers
108 views
Will duplicate international (i18n) content hinder SEO rankings?
Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
2
votes
1answer
317 views
HTTP 303 redirection and robots.txt
On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
0
votes
1answer
44 views
Addon domains and SEO
I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page.
Can I tell bots not to index the main domain WITHOUT ...
2
votes
1answer
59 views
GoogleBot can't reach my site
For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
1answer
99 views
How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.
I ...
2
votes
1answer
78 views
“Google couldn't access your site because of a server connectivity issue”
In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site".
Under the Current Status it has:
DNS - SUCCESS
Server Connectivity ...
4
votes
3answers
334 views
Google indexing page with parameters but page is Disallowed in robots.txt
I have the following in robots.txt:
User-agent: *
Disallow: /refer.php
User-agent: NinjaBot
Allow: /
Sitemap: http://www.mysite.com/sitemap.xml
The refer.php file does various things depending on ...
2
votes
1answer
67 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
3answers
54 views
Googlebot is blocked from my domain
My robots.txt is:
User-agent: *
Disallow: /
Sitemap: http://example.com/sitemap.xml
Sitemap: http://example.com/sitemap.xml.gz
It has been the same for well over a week now, but when I go to ...
2
votes
2answers
64 views
How can an exception be created for Facebook on robots.txt?
I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
2
votes
1answer
33 views
what should be my initial address for sitemap and robots.txt if I'm using redirection
What should be my initial adress when using this site: http://www.xml-sitemaps.com?
I have a folder inside my root where I have all my files.
For example I have domain.com and my start directory is ...
2
votes
3answers
69 views
Should I use both Robots.txt and META NAME=“ROBOTS”?
I am currently using the Robots.txt file (the standard one of joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior.
Is there a reason to ...
4
votes
1answer
2k views
Google Webmaster Central tells me that robots is blocking access to the sitemap
This is my robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.mydomain.org/sitemap.xml.gz
But Google Webmaster Central tells me that robots is blocking ...
1
vote
3answers
562 views
Robots.txt never downloaded but some blocked URLs in GWT
There is something I don't understand in Google Webmaster Tools (GWT) for my Wordpress site.
In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked ...
1
vote
2answers
43 views
Robots.txt Blocking Blog Tags?
Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled?
User-agent: ...
1
vote
1answer
33 views
Prohibit search engines before going live
I want to prohibit search engines to index my web site before going live with it.
If I simple do a robots.txt with:
User-agent: *
Disallow: /
Would that cause any issues later when I decide to ...
4
votes
3answers
74 views
SEO Removal Time?
I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here.
I believe it will be the solution but what i still need to ...
0
votes
1answer
42 views
How to overcome duplicate URLS
I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls.
From: /psd-layouts.html to /psd-web-layouts.html
Now my question is that, I had ...
0
votes
2answers
58 views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
28 views
Question about duplicate groups (agents) in robots.txt
I'm looking at a corner case with robots.txt, and am curious if you have any guidance.
Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file.
I've ...
1
vote
1answer
1k views
Strange robots.txt - how and why did it get there?
I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
5
votes
4answers
210 views
Robots.txt vs Sitemap — Who wins in a Conflict
If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
1
vote
3answers
132 views
Google Indexing Pages with noindex meta tag
I got a message in Google Webmaster Tools about "Googlebot found an extremely high number of URLs on your site" with a long list of example URLs.
For some of the pages listed, there was nothing to ...
1
vote
2answers
143 views
Will 301 redirection to a Google Indexed URL can remove the URL from Google index?
I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
2
votes
2answers
158 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
4
votes
3answers
2k views
In Robots.txt, how do I disallow all pages except for the index page?
I want google to index my index page, but do not index any other pages.
User-agent: *
Disallow: /
So far I have this. However...when I do this, Google does not index my index page. When I search ...
2
votes
1answer
129 views
Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?
As I understand, having repetitive content is a detriment to search engine placement.
Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
2
votes
2answers
170 views
Would a 403 page, a blank page and a page redirect be indexed by search robots?
I am wondering if a disallow directive needs to be issued to robots on pages that will generate:
a 403 forbidden error;
a blank page, or;
a page redirect (to log in page for example).
Would not ...
3
votes
2answers
194 views
Robots.txt on one line
Can I combine robots.txt rules on one line, separated by whitespace?
Will this work?
User-agent: * Disallow: /
1
vote
3answers
165 views
URL blocked by robots.txt in webmaster for blogger site
Remove: Disallow: /search
From where can i remove this line. I am using blogger and in webmaster it shows my many pages are blocked. From one of your answers i know that i need to remove this text ...
3
votes
1answer
93 views
Handling SEO for Infinite pages that cause external slow API calls
I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
8
votes
3answers
2k views
Does Google cache robots.txt?
I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...
2
votes
9answers
4k views
Need to Stop Bots from Killing my Webserver
I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
1
vote
1answer
305 views
Block access to robots.txt file .htaccess
I read in this forum that this .htaccess code should block access to the robot.txt file:
RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely)
RewriteRule ...
6
votes
2answers
83 views
What kind of redirect (301 or 302) for an email links tracker?
We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application ...
3
votes
2answers
450 views
Should I prevent search engines indexing tag/category pages?
On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...
3
votes
2answers
140 views
Impact of blocking site search from googlebot, but listing content pages in sitemap.xml
We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
1
vote
2answers
285 views
How to submit sitemap when your website has partial https? - Error: “Not in Domain”
My website is an eCommerce that is set up to do HTTP for the item browsing portion, but HTTPS for things like shopping cart, contact us, etc.. (anything that has forms on it).
I've submitted my ...
3
votes
1answer
563 views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
0
votes
1answer
109 views
Hiding a particulat page from search engines not to index [duplicate]
Possible Duplicate:
Blocking Search Engines to Crawl Parts of a Website?
I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
