Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
1answer
16 views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
0
votes
2answers
46 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
2
votes
1answer
355 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
0
votes
0answers
8 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
1
vote
1answer
33 views

Include “www” in sitemap URL in robots.txt when site redirects to no-www?

Website is redirected to remove the www so www.example.com is redirecting to example.com. Question is do I need to include the www in the sitemap for the robots.txt?
1
vote
0answers
43 views

Google search console reports URLs blocked by robots.txt even though they shouldn't be

Recently I sent a sitemap to Google Search Console and I got some warnings saying that some urls are blocked by the robots.txt. I checked it and there is no single line that blocks the corresponding ...
2
votes
1answer
29 views

Allowing Google to index images on s3

We host our sites' static content on S3. We also have a very open robots.txt; User-agent: * Allow: * This is because in Webmaster tools I get thousands of warnings Sitemap contains urls which are ...
3
votes
2answers
41 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
4
votes
1answer
66 views

Google shows every page of my WordPress site but I only want the home page indexed

So to start with, I have Google crawl my (WordPress) website with the search console, now I can see my pages in the search engine. But it shows every page separately. Like my /contact, /services and ...
1
vote
0answers
27 views

Is sitemap in sitemap_index getting indexed by Google webmaster?

it sounds a silly question, but I actually could not find a reference to it. I have created a sitemap_index with about 100-ish sitemaps. No errors. I submitted only the sitemap_index to Google ...
1
vote
1answer
22 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: ...
2
votes
1answer
53 views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
0answers
59 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
2
votes
0answers
27 views

How to make nginx override web app route for sitemap and robots?

I would like to make nginx serve sitemap and robots. My problem is that route http://example.com/ is handled by flask, configured to handle request with certain patterns - ...
3
votes
1answer
150 views

Can HTML sitemaps be listed in robots.txt just like XML sitemaps?

I will have two sitemaps for my website. The first sitemap is the .xml version for the search engines. The second sitemap is a sitemap.php on my website for the users. I know I can list the ...
3
votes
1answer
50 views

Robots.txt problem

Two weeks ago I changed the address of my site. I forgot to renew my domain, and for a few days my site was unreachable. This resulted in google indexing my site inproperly. There is a robot.txt error ...
2
votes
2answers
33 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
3
votes
2answers
56 views

Google can't access non-existent robots.txt

I set up a website a few weeks ago and I'm trying to get Google to crawl it. When logging into Google's Search Console (Webmaster Tools) and within: Crawl > Crawl Errors It reports: Google ...
1
vote
2answers
44 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
2answers
75 views

How to block subdomains under my main domain

I want all my subdomains to not be indexed under my domain, that means when I search in Google for example.com to not appear in index test.example.com, but when I search test.example.com to be ...
2
votes
2answers
110 views

Google Search Console: Severe health issues, some important page is blocked by robots.txt

I have just added a new property to the Google search console. I've added it as http://example.com and http://www.example.com. I receive the following error message: Severe health issues are found ...
1
vote
1answer
37 views

The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix?

We have a website: www.example.com which redirects to https://www.example.com. Therefore when you type www.example.com/robots.txt you are redirected to https://www.example.com/robots.txt. Another ...
3
votes
1answer
34 views

cookies and useragent is not enough ? to emulate browser?

I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt Let's take an example, I need to get this page: ...
1
vote
1answer
48 views

Should I block a dynamically generated data results page in robots.txt?

I have a tiny two page site: the parameters page and results page. The results page can either work with URL parameters or hidden POST ones. The output is a giant table of data that contains rankings ...
2
votes
0answers
47 views

Google Webmaster Tools reports URLs blocked by robots.txt on my AVS tube website

I added a tube (with AVS script) in Google Webmaster Tools, but the issue is with the robots.txt file which is blocking everything. Here is my AVS tube robots.txt: User-agent: * Allow: / Disallow: ...
2
votes
1answer
32 views

Google indexed work-in-progress page. how to delete it from Google index?

I developed a page under the following URL: example.org/dev Since the development is finished, I moved the page to the root at example.org/ For some reason, Google indexed the page during ...
6
votes
5answers
589 views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
1
vote
1answer
37 views

Robots.txt disallow all except one directory and access to sitemap

robots.txt containts following instructions: User-agent: * Allow: /public/ Disallow: / Sitemap: http://sitedomain.com/sitemapindex.xml it seems that in this case sitemapindex won't be accessible ...
0
votes
2answers
41 views

Sitemap XML title in search results

I have a website that uses an XML sitemap and a robots.txt file. If I do a Google search of: site: mywebsite.co.uk "sitemap" I see the usual Google response, similar to the following image, ...
8
votes
2answers
816 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
2
votes
2answers
69 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
4
votes
1answer
60 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
5
votes
3answers
67 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
1
vote
1answer
32 views

Remove website pages from search results, but keep them crawled

I would like to remove all pages of my website from the search results, except the home page. Is there a way to do this and keep the pages crawlable? Or is there no way except robots.txt?
3
votes
1answer
59 views

How do I disallow variable parent directories and subdirectory with robots.txt?

I want to disallow robots to index some pages of my site. For example, I want to disallow the following pages: /category/name1/page/1 /category/name1/page/2 /category/etc/page/3 But I want to allow ...
1
vote
1answer
107 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
4
votes
4answers
133 views

How do I disallow a specific query string in robots.txt?

I have the URL http://www.example.com/shopping/books/?b=9 and the following robots.txt file: User-agent: * Disallow: /?b=9 But when I test this in Google Webmaster Tool's robots.txt tester it ...
0
votes
0answers
19 views

Robots.txt Query String - Couldn't Disallow [duplicate]

I have url : http://www.website.com/shopping/books/?b=9 I disallowed /?b=9 from robots.txt as User-agent: * Disallow: /?b=9 But when i test this from Google Webmaster, on Robots.txt Test Tools, ...
2
votes
2answers
130 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
2
votes
1answer
38 views

Using canonical links to eliminate “similar” search results in google

In our site, we have links as such: /{{category-name}}.html - will display all products /category-name/filters/{{some-filter}}.html - will display filtered products The problem we are facing is ...
0
votes
0answers
9 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
1answer
102 views

Using robots.txt to block sessionID URLs

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
2
votes
1answer
72 views

Remove several URL of website from Google

I want to remove multiple URL in my website (developped in ASP MVC) All these URL have the format www.mysite.com/planning/xxx I have already adapted the code to make 301 redirect and indicate to ...
0
votes
0answers
15 views

Why does my staging site show up in Google even though my robots.txt file shouldn't allow it? [duplicate]

I'm having a lot of trouble with something that should be simple. I don't want my development site indexed by Google. According to Google's instructions, and just about everything else I can find says ...
2
votes
2answers
146 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: ...
3
votes
1answer
63 views

Point Domain Names With keyword to main website?

This is Question has been asked several time on Stack Exchange. I have read all those answer, still there are certain some technical specification answered. After lot of research on Internet , I ...
4
votes
3answers
63 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
1
vote
2answers
68 views

Proper content for robots.txt on server maintenance page?

I know that the proper server response header should be 503 Service Unavailable, to let the search bots know that they shouldn't index the pages like Update in progress and Server maintenance. But ...
1
vote
1answer
64 views

Sitemap in different domain and robots.txt crawling

I have the following subdomains: www.example.com api.example.com The subdomain api.example.com is supposed to be invisible to Google. Its robots.txt had a disallow / directive. But I don't want ...
2
votes
1answer
56 views

How much time does robots.txt remain in Google cache?

The old robots.txt alerts me with warning about locked URL. With the new one, the URL passes the tester in Google Webmaster. But when I resend the sitemap I get the same error!