Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

0
votes
1answer
293 views

wordpress robots.txt is this reasonable?

this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms. all the page urls like http://mydomain.com/it/page1, ...
0
votes
1answer
3k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
0
votes
1answer
991 views

Restricting crawler activity to certain directories with robots.txt

I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers. In my robots.txt, I have ...
0
votes
2answers
98 views

Pre-launch Disallow in robots.txt?

My team is launching a new site soon. Is it worth blocking spiders from indexing the landing page right now? What effects will this have on my long term SEO results. Is this question even ...
0
votes
0answers
11 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
0
votes
0answers
19 views

Robots.txt Query String - Couldn't Disallow [duplicate]

I have url : http://www.website.com/shopping/books/?b=9 I disallowed /?b=9 from robots.txt as User-agent: * Disallow: /?b=9 But when i test this from Google Webmaster, on Robots.txt Test Tools, ...
0
votes
0answers
9 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
0answers
15 views

Why does my staging site show up in Google even though my robots.txt file shouldn't allow it? [duplicate]

I'm having a lot of trouble with something that should be simple. I don't want my development site indexed by Google. According to Google's instructions, and just about everything else I can find says ...
0
votes
0answers
50 views

Google reports that robots.txt is blocking site - but I don't think it is

I have a site map at http://www.example.com/sitemap_index.xml that has been sumbitted to Google Webmaster Tools, however... Google is reporting that my sitemap is being blocked by robots.txt. My ...
0
votes
0answers
74 views

can't remove blocks on my resources in google webmaster tools

I received an alert from Google Webmaster Tools a couple weeks ago that stated: "Google systems have recently detected an issue with your homepage that affects how well our algorithms render and ...
0
votes
0answers
219 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
0
votes
0answers
448 views

Google webmaster tool Severe health issues are found in your property

I'm getting this message in GWT : Severe health issues are found in your property , Some important page is blocked by robots.txt. My robots.txt User-agent: * Disallow: /wp-admin/ Sitemap: ...
0
votes
0answers
71 views

SEO: Deindex member profile pages?

I have a forum where users can register and automatically a profile page is created, like here in Stackexchange. The problem with these pages is that sometimes users creates accounts at different ...
0
votes
1answer
49 views

“Verify live version” not available in Google Webmaster Tools

I was searching how to submit a robots.txt file using Google Webmaster Tools and I see no option of Verify live version, which is the forth step in submitting a robots.txt file to Google. I went ...
0
votes
1answer
372 views

explicitly whitelist every page using robots.txt

I need to allow only specific pages at some site. Let's say: / /page1 /page2 ... Now I don't get how to whitelist the root page at robots.txt Disallow: / # block everything Allow: /page1 ...
0
votes
0answers
15 views

How to prevent Google (and others) from indexing the site under construction? [duplicate]

Recently I faced with the situation that the wrong site pages (with wrong URLs, wrong content) was indexed by Google during the site developing. Should I place robots.txt with User-agent: * Disallow: ...
0
votes
0answers
576 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
0
votes
1answer
133 views

Hiding a particulat page from search engines not to index [duplicate]

Possible Duplicate: Blocking Search Engines to Crawl Parts of a Website? I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
-2
votes
1answer
339 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...