Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
2
votes
1answer
32 views
How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.
I ...
2
votes
1answer
39 views
“Google couldn't access your site because of a server connectivity issue”
In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site".
Under the Current Status it has:
DNS - SUCCESS
Server Connectivity ...
4
votes
3answers
271 views
Google indexing page with parameters but page is Disallowed in robots.txt
I have the following in robots.txt:
User-agent: *
Disallow: /refer.php
User-agent: NinjaBot
Allow: /
Sitemap: http://www.mysite.com/sitemap.xml
The refer.php file does various things depending on ...
2
votes
1answer
48 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
3answers
40 views
Googlebot is blocked from my domain
My robots.txt is:
User-agent: *
Disallow: /
Sitemap: http://example.com/sitemap.xml
Sitemap: http://example.com/sitemap.xml.gz
It has been the same for well over a week now, but when I go to ...
1
vote
1answer
405 views
Robots.txt practices with .htaccess redirections (inherits)
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place.
We have a hosting account that enacts primary and add-on domains. All of our domains ...
2
votes
2answers
86 views
Will duplicate international (i18n) content hinder SEO rankings?
Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
2
votes
2answers
40 views
How can an exception be created for Facebook on robots.txt?
I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
0
votes
1answer
39 views
Addon domains and SEO
I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page.
Can I tell bots not to index the main domain WITHOUT ...
2
votes
1answer
30 views
what should be my initial address for sitemap and robots.txt if I'm using redirection
What should be my initial adress when using this site: http://www.xml-sitemaps.com?
I have a folder inside my root where I have all my files.
For example I have domain.com and my start directory is ...
2
votes
3answers
62 views
Should I use both Robots.txt and META NAME=“ROBOTS”?
I am currently using the Robots.txt file (the standard one of joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior.
Is there a reason to ...
1
vote
1answer
44 views
GoogleBot can't reach my site
For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
4
votes
1answer
2k views
Google Webmaster Central tells me that robots is blocking access to the sitemap
This is my robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.mydomain.org/sitemap.xml.gz
But Google Webmaster Central tells me that robots is blocking ...
1
vote
3answers
498 views
Robots.txt never downloaded but some blocked URLs in GWT
There is something I don't understand in Google Webmaster Tools (GWT) for my Wordpress site.
In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked ...
1
vote
2answers
31 views
Robots.txt Blocking Blog Tags?
Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled?
User-agent: ...
1
vote
1answer
33 views
Prohibit search engines before going live
I want to prohibit search engines to index my web site before going live with it.
If I simple do a robots.txt with:
User-agent: *
Disallow: /
Would that cause any issues later when I decide to ...
4
votes
3answers
72 views
SEO Removal Time?
I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here.
I believe it will be the solution but what i still need to ...
0
votes
1answer
38 views
How to overcome duplicate URLS
I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls.
From: /psd-layouts.html to /psd-web-layouts.html
Now my question is that, I had ...
0
votes
2answers
53 views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
26 views
Question about duplicate groups (agents) in robots.txt
I'm looking at a corner case with robots.txt, and am curious if you have any guidance.
Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file.
I've ...
1
vote
1answer
1k views
Strange robots.txt - how and why did it get there?
I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
5
votes
4answers
191 views
Robots.txt vs Sitemap — Who wins in a Conflict
If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
3
votes
2answers
5k views
Can I invoke Google to check my robots.txt?
I read the answers in this question, but they still leave my question open: does Google cache robots.txt?
I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt.
...
1
vote
3answers
98 views
Google Indexing Pages with noindex meta tag
I got a message in Google Webmaster Tools about "Googlebot found an extremely high number of URLs on your site" with a long list of example URLs.
For some of the pages listed, there was nothing to ...
1
vote
2answers
105 views
Will 301 redirection to a Google Indexed URL can remove the URL from Google index?
I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
2
votes
2answers
156 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
4
votes
3answers
2k views
In Robots.txt, how do I disallow all pages except for the index page?
I want google to index my index page, but do not index any other pages.
User-agent: *
Disallow: /
So far I have this. However...when I do this, Google does not index my index page. When I search ...
2
votes
1answer
105 views
Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?
As I understand, having repetitive content is a detriment to search engine placement.
Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
2
votes
2answers
162 views
Would a 403 page, a blank page and a page redirect be indexed by search robots?
I am wondering if a disallow directive needs to be issued to robots on pages that will generate:
a 403 forbidden error;
a blank page, or;
a page redirect (to log in page for example).
Would not ...
3
votes
2answers
185 views
Robots.txt on one line
Can I combine robots.txt rules on one line, separated by whitespace?
Will this work?
User-agent: * Disallow: /
1
vote
3answers
132 views
URL blocked by robots.txt in webmaster for blogger site
Remove: Disallow: /search
From where can i remove this line. I am using blogger and in webmaster it shows my many pages are blocked. From one of your answers i know that i need to remove this text ...
3
votes
1answer
87 views
Handling SEO for Infinite pages that cause external slow API calls
I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
8
votes
3answers
2k views
Does Google cache robots.txt?
I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...
2
votes
9answers
4k views
Need to Stop Bots from Killing my Webserver
I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
1
vote
1answer
296 views
Block access to robots.txt file .htaccess
I read in this forum that this .htaccess code should block access to the robot.txt file:
RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely)
RewriteRule ...
6
votes
2answers
74 views
What kind of redirect (301 or 302) for an email links tracker?
We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application ...
3
votes
2answers
442 views
Should I prevent search engines indexing tag/category pages?
On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...
5
votes
1answer
219 views
should I block wordpress archive pages from search engines?
I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives.
Google has indexed the site fully.
Since the archives currently display full posts I've thought that I shouldn't ...
3
votes
2answers
126 views
Impact of blocking site search from googlebot, but listing content pages in sitemap.xml
We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
1
vote
2answers
258 views
How to submit sitemap when your website has partial https? - Error: “Not in Domain”
My website is an eCommerce that is set up to do HTTP for the item browsing portion, but HTTPS for things like shopping cart, contact us, etc.. (anything that has forms on it).
I've submitted my ...
3
votes
1answer
525 views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
0
votes
1answer
107 views
Hiding a particulat page from search engines not to index [duplicate]
Possible Duplicate:
Blocking Search Engines to Crawl Parts of a Website?
I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
3
votes
4answers
512 views
Hide a Subdomain from Google/Search Engine SEO Result?
I have a subomain which I do not want to be listed in any search engine result. Let say I have:
http://www.example.com
http://child.example.com
How can I hide all URLs of child.example.com domain ...
3
votes
3answers
214 views
Is my robots.txt working as it should?
I want crawlers to have access to http://www.example.com but not http://www.example.com/
My robots.txt is as follows:
User-agent: *
Allow: /$
Disallow: /
My site is in Google search results, but I ...
3
votes
1answer
70 views
Need help with 200,000 + 404's
The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" ...
2
votes
3answers
293 views
Google is indexing unintended content that is either unpublished or a secret part of site
Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that I'm working on - www.domain.com/m/ <-- note the M.
How is that even possible? Only thing I can ...
1
vote
2answers
516 views
SEO: disallowing Google from indexing forms?
From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not?
I mean my forms hardly ever contains keyword/keyphrases.
So I'm wondering what's the point of letting ...
2
votes
1answer
274 views
Googlebot can't access my site when crawling from rootdomain
I can't explain why I get this message for my rootdomain result in Google :
trocmalin.com/
A description for this result is not available because of this site's robots.txt – learn more.
...
4
votes
3answers
547 views
nofollow doesn't stop crawling?
In my website, most of the pages has a linkedin sharing button written in this way:
<li>
<a id="share_linkedin"
...
2
votes
1answer
438 views
Prevent crawling of mobile site
I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users.
The mobile content is in /mobile/
The regular site will redirect mobile users to the ...
