Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

1
vote
1answer
20 views

Allowing Google to index images on s3

We host our sites' static content on S3. We also have a very open robots.txt; User-agent: * Allow: * This is because in Webmaster tools I get thousands of warnings Sitemap contains urls which are ...
3
votes
2answers
41 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
4
votes
1answer
54 views

Google shows every page of my WordPress site but I only want the home page indexed

So to start with, I have Google crawl my (WordPress) website with the search console, now I can see my pages in the search engine. But it shows every page separately. Like my /contact, /services and ...
1
vote
0answers
22 views

Is sitemap in sitemap_index getting indexed by Google webmaster?

it sounds a silly question, but I actually could not find a reference to it. I have created a sitemap_index with about 100-ish sitemaps. No errors. I submitted only the sitemap_index to Google ...
1
vote
1answer
21 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: ...
2
votes
1answer
44 views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
0answers
52 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
2
votes
0answers
23 views

How to make nginx override web app route for sitemap and robots?

I would like to make nginx serve sitemap and robots. My problem is that route http://example.com/ is handled by flask, configured to handle request with certain patterns - ...
3
votes
1answer
148 views

Can HTML sitemaps be listed in robots.txt just like XML sitemaps?

I will have two sitemaps for my website. The first sitemap is the .xml version for the search engines. The second sitemap is a sitemap.php on my website for the users. I know I can list the ...
3
votes
1answer
49 views

Robots.txt problem

Two weeks ago I changed the address of my site. I forgot to renew my domain, and for a few days my site was unreachable. This resulted in google indexing my site inproperly. There is a robot.txt error ...
2
votes
2answers
32 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
3
votes
2answers
48 views

Google can't access non-existent robots.txt

I set up a website a few weeks ago and I'm trying to get Google to crawl it. When logging into Google's Search Console (Webmaster Tools) and within: Crawl > Crawl Errors It reports: Google ...
1
vote
2answers
41 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
2answers
64 views

How to block subdomains under my main domain

I want all my subdomains to not be indexed under my domain, that means when I search in Google for example.com to not appear in index test.example.com, but when I search test.example.com to be ...
2
votes
2answers
96 views

Google Search Console: Severe health issues, some important page is blocked by robots.txt

I have just added a new property to the Google search console. I've added it as http://example.com and http://www.example.com. I receive the following error message: Severe health issues are found ...
1
vote
1answer
37 views

The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix?

We have a website: www.example.com which redirects to https://www.example.com. Therefore when you type www.example.com/robots.txt you are redirected to https://www.example.com/robots.txt. Another ...
3
votes
1answer
30 views

cookies and useragent is not enough ? to emulate browser?

I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt Let's take an example, I need to get this page: ...
1
vote
1answer
46 views

Should I block a dynamically generated data results page in robots.txt?

I have a tiny two page site: the parameters page and results page. The results page can either work with URL parameters or hidden POST ones. The output is a giant table of data that contains rankings ...
2
votes
0answers
44 views

Google Webmaster Tools reports URLs blocked by robots.txt on my AVS tube website

I added a tube (with AVS script) in Google Webmaster Tools, but the issue is with the robots.txt file which is blocking everything. Here is my AVS tube robots.txt: User-agent: * Allow: / Disallow: ...
2
votes
1answer
31 views

Google indexed work-in-progress page. how to delete it from Google index?

I developed a page under the following URL: example.org/dev Since the development is finished, I moved the page to the root at example.org/ For some reason, Google indexed the page during ...
6
votes
5answers
583 views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
1
vote
1answer
32 views

Robots.txt disallow all except one directory and access to sitemap

robots.txt containts following instructions: User-agent: * Allow: /public/ Disallow: / Sitemap: http://sitedomain.com/sitemapindex.xml it seems that in this case sitemapindex won't be accessible ...
0
votes
2answers
38 views

Sitemap XML title in search results

I have a website that uses an XML sitemap and a robots.txt file. If I do a Google search of: site: mywebsite.co.uk "sitemap" I see the usual Google response, similar to the following image, ...
8
votes
2answers
806 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
2
votes
2answers
59 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
4
votes
1answer
56 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
5
votes
3answers
62 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
1
vote
1answer
32 views

Remove website pages from search results, but keep them crawled

I would like to remove all pages of my website from the search results, except the home page. Is there a way to do this and keep the pages crawlable? Or is there no way except robots.txt?
3
votes
1answer
57 views

How do I disallow variable parent directories and subdirectory with robots.txt?

I want to disallow robots to index some pages of my site. For example, I want to disallow the following pages: /category/name1/page/1 /category/name1/page/2 /category/etc/page/3 But I want to allow ...
1
vote
1answer
92 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
4
votes
4answers
124 views

How do I disallow a specific query string in robots.txt?

I have the URL http://www.example.com/shopping/books/?b=9 and the following robots.txt file: User-agent: * Disallow: /?b=9 But when I test this in Google Webmaster Tool's robots.txt tester it ...
0
votes
0answers
19 views

Robots.txt Query String - Couldn't Disallow [duplicate]

I have url : http://www.website.com/shopping/books/?b=9 I disallowed /?b=9 from robots.txt as User-agent: * Disallow: /?b=9 But when i test this from Google Webmaster, on Robots.txt Test Tools, ...
2
votes
2answers
129 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
2
votes
1answer
38 views

Using canonical links to eliminate “similar” search results in google

In our site, we have links as such: /{{category-name}}.html - will display all products /category-name/filters/{{some-filter}}.html - will display filtered products The problem we are facing is ...
0
votes
0answers
9 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
1answer
94 views

Using robots.txt to block sessionID URLs

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
2
votes
1answer
65 views

Remove several URL of website from Google

I want to remove multiple URL in my website (developped in ASP MVC) All these URL have the format www.mysite.com/planning/xxx I have already adapted the code to make 301 redirect and indicate to ...
0
votes
0answers
15 views

Why does my staging site show up in Google even though my robots.txt file shouldn't allow it? [duplicate]

I'm having a lot of trouble with something that should be simple. I don't want my development site indexed by Google. According to Google's instructions, and just about everything else I can find says ...
2
votes
2answers
131 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: ...
3
votes
1answer
61 views

Point Domain Names With keyword to main website?

This is Question has been asked several time on Stack Exchange. I have read all those answer, still there are certain some technical specification answered. After lot of research on Internet , I ...
4
votes
3answers
63 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
1
vote
2answers
64 views

Proper content for robots.txt on server maintenance page?

I know that the proper server response header should be 503 Service Unavailable, to let the search bots know that they shouldn't index the pages like Update in progress and Server maintenance. But ...
1
vote
1answer
57 views

Sitemap in different domain and robots.txt crawling

I have the following subdomains: www.example.com api.example.com The subdomain api.example.com is supposed to be invisible to Google. Its robots.txt had a disallow / directive. But I don't want ...
2
votes
1answer
55 views

How much time does robots.txt remain in Google cache?

The old robots.txt alerts me with warning about locked URL. With the new one, the URL passes the tester in Google Webmaster. But when I resend the sitemap I get the same error!
0
votes
2answers
142 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
0
votes
0answers
50 views

Google reports that robots.txt is blocking site - but I don't think it is

I have a site map at http://www.example.com/sitemap_index.xml that has been sumbitted to Google Webmaster Tools, however... Google is reporting that my sitemap is being blocked by robots.txt. My ...
6
votes
1answer
115 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
1
vote
0answers
128 views

Magento Robots.txt and blocking cached images

We have a Magento based store. Images are resized and served cached dynamically. When the image cache in Magento is cleared, there are generated new paths and files. And I get 404 errors for them ...
3
votes
2answers
54 views

Should I block an .htm resource in robots.txt

We use scripts to create static .htm files to show content on our site. Mainly in an effort to reduce database calls. For example, a script that builds a file every 5 minutes that contains a list of ...
2
votes
2answers
82 views

How do robots respond to no user-agent in robots.txt?

I'm just curious because I can't seem to find any mention of it anywhere. What is the standard (or at the very least typical) response to robots.txt without a user-agent parameter? Does it assume the ...