Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
24
votes
2answers
2k views
Files I should have on every website
I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs.
So far I have the ...
1
vote
2answers
125 views
Google Webmaster Tools complains about missing robots.txt
Nine days ago, I got a message Google Webmaster Tools:
Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt.
Well, but I don't have a robots.txt on ...
1
vote
1answer
212 views
Complex Disallow pattern in robots.txt
I have a URL like this:
www.example.com/freelance-jobs-new-york
I had a problem and many duplicated pages have been created like this:
www.example.com/freelance-jobs-new-york-php-php
...
0
votes
1answer
501 views
SEO not indexing my site: “A description for this result is not available because of this site's robots.txt – learn more.” [duplicate]
When googling my site I get following text under the result for my domain:
"A description for this result is not available because of this site's
robots.txt – learn more."
My robots.txt, ...
4
votes
1answer
569 views
How do I deal with content scrapers? [duplicate]
Possible Duplicate:
How to protect SHTML pages from crawlers/spiders/scrapers?
My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling ...
4
votes
2answers
104 views
Preventing search engines to crawl on specific folders
I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow:
...
2
votes
2answers
205 views
Network unreachable: robots.txt unreachable
I am trying to add a valid sitemap to Google Webmaster. Yet, it says:
Network unreachable: robots.txt unreachableWe were unable to crawl
your Sitemap because we found a robots.txt file at ...
4
votes
3answers
205 views
Bad Bot blocking Revisited
I've read a lot about bad bot blocking, php scripts, .htaccess techniques, etc...
Is this a valid method?
Since .htacces can rewrite and send a bad bot a 403 deny or forward to something like spam ...
1
vote
0answers
79 views
Methods for keeping no-good scrapers off the site? [duplicate]
Possible Duplicate:
Tactics for dealing with misbehaving robots
I want to keep no-good scrapers (aka. bad bots) that steal content and consume bandwidth off my site. At the same time, I do ...
0
votes
0answers
140 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
1
vote
2answers
134 views
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent.
I decreased crawl rate from Google webmaster tools.
But I'm supposing to ...
2
votes
2answers
86 views
Prevent crawler that doesn't honoring robots.txt
I have some problem, when I try to write robots.txt for my site ...
I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
1
vote
2answers
86 views
Can search engine robots read file with permission 640?
I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing www.mysite.com/robots.txt.
As per the following google group ...
1
vote
1answer
48 views
What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?
I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
0
votes
1answer
156 views
Noindex, Nofollow affect “Google Custom Search” results?
I am using Google Custom Search on my site, if I set pages to "noindex" or "nofollow", does it also affect the custom search results on my site?
0
votes
1answer
214 views
google webmaster showing warnings - blocked by robots.txt, but its not?
im getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt but its not, has any one come across this or a way to resolve this before ?
the ...
1
vote
1answer
284 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
1
vote
1answer
40 views
If I block a page in robots.txt would I still be able to see the page in my browser? [closed]
What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?
2
votes
3answers
165 views
Hiding php includes from search spiders?
Quick and simple question.
I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are ...
0
votes
1answer
245 views
How to secure robots.txt file?
I would like for User-agents to index my relative pages only without accessing any directory on my server.
As initial thought, i had this version in mind:
User-agent: *
Disallow: */*
Sitemap: ...
1
vote
1answer
116 views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
0
votes
0answers
262 views
blocking bad bots with robots.txt in 2012 [closed]
does it still work good? I have this:
# Generated using http://solidshellsecurity.com services
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
...
1
vote
5answers
215 views
Should I add a “nofollow” attribute to download links, or disallow the URLs in robots.txt?
I have a download link very similar to Opera's one - it's just a script that sends the file. It doesn't have an extension and there's no obvious way to tell that it's actually a download link.
So ...
0
votes
1answer
49 views
If I blocked a directory, is a 301 error expected?
I blocked a directory in my site that is not secret but meant for internal discussions that are not relevant for search engine results. After running "Search as Google" at Google Webmaster Tools, the ...
2
votes
3answers
4k views
How to remove old robots.txt from google as old file block the whole site
I have a website which still shows old robots.txt in the google webmaster tools.
User-agent: *
Disallow: /
Which is blocking Googlebot. I have removed old file updated new robots.txt file with ...
0
votes
4answers
165 views
Can I include a robots meta tag outside of the head in HTML snippets indeded to be AJAXed?
I have a number of files in my site which are not intended for independent viewing, but rather to be AJAXed into content within the site. They obviously don't meet HTML standards (no body, head, etc.) ...
2
votes
2answers
104 views
How to get search engines to properly index an ajax driven search page
I have an ajax-driven search page that will allow users to search through a large collection of records. Each search result points to index.php?id=xyz (where xyz is the id of the record). The initial ...
1
vote
1answer
122 views
How to block FeedReader from fetching my content to their site?
As you all can see from the picture below, my site's content is duplicated by FeedReader and indexed at Google. When I clicked at the FeedReader link, it uses some sort of iFrame to draw content from ...
3
votes
3answers
111 views
SEO consideration for duplicate sites
I am building a brochure-ware website for a company that sells products all across the world. They need the site to ask the user what region they are in before using the site; there are 5 regions. ...
1
vote
2answers
597 views
How to resolve “Google can't find your site's robots.txt” error?
I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
1
vote
3answers
107 views
robots.txt file Help needed
What if I put same robots.txt file in multiple directories and folders?
User-agent: *
Disallow: /otherdirname/dirname2/
Can it'll affect anyhow ??
And how much time it generally takes to refine ...
6
votes
4answers
308 views
robots.txt not updated
I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
0
votes
0answers
12 views
URL blocked in robots.txt but still showing up on Google search [duplicate]
Possible Duplicate:
Why do Google search results include pages disallowed in robots.txt?
In my robots.txt I am disallowing a lot of URLs. Google webmaster tools says there're +750 URL ...
5
votes
2answers
118 views
Robots.txt and pattern matching
Adding this to my robots.txt
User-agent: *
Disallow: /*action=*$
How does robots not recognizing wild cards handle this?
2
votes
1answer
147 views
Edit robots.txt on google sites
Getting many 404 errors on my website for the following URLs:
/system/app/pages/admin/revisions*
I was going to put Disallow in robots.txt, but it seems there is no way to edit robots.txt in google ...
4
votes
2answers
895 views
How do you disallow root in robots.txt, but allow a subdirectory?
Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?
0
votes
0answers
65 views
Facebook crawling [closed]
Does a normal person is allowed to crawl all the publically visible links of facebook for research purpose with the help of automated bots?
And whether there is any API provided by ...
1
vote
0answers
14 views
Mobile site robots.txt [duplicate]
Possible Duplicate:
Should I block indexing of my mobile site?
Should I allow webcrawlers to index the mobile versions of my websites? Is that a good practice from a SEO perspective?
...
5
votes
2answers
1k views
Why do Google search results include pages disallowed in robots.txt?
I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this:
User-Agent: *
Disallow: /email
Yet I recently noticed that Google ...
9
votes
3answers
269 views
Robots.txt: do I need to disallow a page which is not linked anywhere?
There are some pages on my websites that I want the user to be able to visit only if I give him/her the url.
If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
2
votes
1answer
176 views
How do I disallow robots to access links on a certain port?
I have a private web server set up on a non-conventional port (say 6677). All people can have access to the whole site's urls from 6677. The links by mysite:6677 are also indexed by Google, which is ...
2
votes
1answer
302 views
HTTP 303 redirection and robots.txt
On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
1
vote
2answers
128 views
Robots.txt and “Bad” Robots
I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
5
votes
3answers
274 views
Does loading images from a server that disallows robots affect SEO?
Lets say a page on example.com has all of its images coming from img.example.com:
<img src="http://img.example.com/keyword.jpg" alt="keyword"/>
while robots.txt on img.example.com disallows ...
1
vote
2answers
285 views
robots.txt, how effective is it and how long does it take?
We recently updated the site to a single page site using jQuery to slide between "pages".
So we now have only index.php.
When you search the company on engines such as Google, you get the site and a ...
1
vote
1answer
110 views
Do or can robots cause considerable performance issues?
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Our setup:
Lets say I have site ...
0
votes
0answers
48 views
Need to know how to configure robots.txt for an age gate/age verification website
is it the normal allow all in the file? I need to make sure the crawlers can get through the gate. Thanks!
1
vote
1answer
222 views
Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent
I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site.
What's the symptom? Google + Bing crawl, but don't index any of the pages.
Let me be clear: I'm ...
0
votes
1answer
260 views
wordpress robots.txt is this reasonable？
this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms.
all the page urls like http://mydomain.com/it/page1, ...
2
votes
1answer
171 views
What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]
Possible Duplicate:
How to Remove URLs from Google Search Engine
In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
