Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

1
vote
1answer
101 views

Problem with Google Mobile Test - Blocking images

I have a situation when I check my Drupal 7 website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 56 resources ...
4
votes
3answers
32 views

Remove large number of useless page from Google index using robots.txt

A large number of auto-generated pages from the search of my site has been indexed by Google due to an error. I'm trying to clean this in order to be sure that only the most high quality content is ...
3
votes
2answers
62 views

Should I block an .htm resource in robots.txt

We use scripts to create static .htm files to show content on our site. Mainly in an effort to reduce database calls. For example, a script that builds a file every 5 minutes that contains a list of ...
0
votes
3answers
33 views

Can backlinks to robots.txt blocked pages boost domain's ranking?

So, imagine, we were blocking some URLs via robots.txt, but apparently these pages were getting backlinks. Later, we decided to redirect these blocked pages to another domain of ours. ~1.5 month ...
2
votes
2answers
46 views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
1
vote
1answer
54 views

Google not indexing site, robots being blocked

I have a small static website which allows customers to purchase a PDF containing some data based from an input from them. The PDF's are a page or two and follow a standard template, and various ...
2
votes
2answers
152 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
3
votes
2answers
2k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
0
votes
0answers
15 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
0
votes
2answers
54 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
3
votes
1answer
369 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
2
votes
1answer
40 views

Include “www” in sitemap URL in robots.txt when site redirects to no-www?

Website is redirected to remove the www so www.example.com is redirecting to example.com. Question is do I need to include the www in the sitemap for the robots.txt?
2
votes
0answers
52 views

Google search console reports URLs blocked by robots.txt even though they shouldn't be

Recently I sent a sitemap to Google Search Console and I got some warnings saying that some urls are blocked by the robots.txt. I checked it and there is no single line that blocks the corresponding ...
1
vote
1answer
50 views

Should I block a dynamically generated data results page in robots.txt?

I have a tiny two page site: the parameters page and results page. The results page can either work with URL parameters or hidden POST ones. The output is a giant table of data that contains rankings ...
2
votes
1answer
33 views

Allowing Google to index images on s3

We host our sites' static content on S3. We also have a very open robots.txt; User-agent: * Allow: * This is because in Webmaster tools I get thousands of warnings Sitemap contains urls which are ...
3
votes
2answers
3k views

Why does my sitemap have a “.gz” extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
3
votes
2answers
46 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
4
votes
1answer
72 views

Google shows every page of my WordPress site but I only want the home page indexed

So to start with, I have Google crawl my (WordPress) website with the search console, now I can see my pages in the search engine. But it shows every page separately. Like my /contact, /services and /...
1
vote
0answers
30 views

Is sitemap in sitemap_index getting indexed by Google webmaster?

it sounds a silly question, but I actually could not find a reference to it. I have created a sitemap_index with about 100-ish sitemaps. No errors. I submitted only the sitemap_index to Google ...
2
votes
1answer
59 views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
1answer
25 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: /...
4
votes
3answers
453 views

Googlebot robots.txt access error on HTTPS when redirected from HTTP

This might be a stupid question, but I never came across this issue before and were not able to find any definite answers to this on the web: Our client migrated to HTTPS a few months ago, using ...
1
vote
0answers
68 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
2
votes
0answers
36 views

How to make nginx override web app route for sitemap and robots?

I would like to make nginx serve sitemap and robots. My problem is that route http://example.com/ is handled by flask, configured to handle request with certain patterns - http://example.com/sitemap....
3
votes
1answer
152 views

Can HTML sitemaps be listed in robots.txt just like XML sitemaps?

I will have two sitemaps for my website. The first sitemap is the .xml version for the search engines. The second sitemap is a sitemap.php on my website for the users. I know I can list the sitemap....
3
votes
1answer
53 views

Robots.txt problem

Two weeks ago I changed the address of my site. I forgot to renew my domain, and for a few days my site was unreachable. This resulted in google indexing my site inproperly. There is a robot.txt error ...
2
votes
2answers
34 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
3
votes
2answers
60 views

Google can't access non-existent robots.txt

I set up a website a few weeks ago and I'm trying to get Google to crawl it. When logging into Google's Search Console (Webmaster Tools) and within: Crawl > Crawl Errors It reports: Google ...
1
vote
3answers
4k views

How to fix robots.txt file error in my GWT?

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file. My blog’s robots.txt file ...
1
vote
2answers
50 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
2
votes
2answers
126 views

Google Search Console: Severe health issues, some important page is blocked by robots.txt

I have just added a new property to the Google search console. I've added it as http://example.com and http://www.example.com. I receive the following error message: Severe health issues are found ...
1
vote
2answers
130 views

Where should the robots.txt file be?

If I have a subdomain: subdir.a.com that redirects to a.com/myfolder/ Should the robots.txt file be in: subdir.a.com/robots.txt or a.com/myfolder/robots.txt
1
vote
2answers
87 views

How to block subdomains under my main domain

I want all my subdomains to not be indexed under my domain, that means when I search in Google for example.com to not appear in index test.example.com, but when I search test.example.com to be indexed....
4
votes
2answers
647 views

Is it possible to exclude Data-URIs (data:image/xxx) through robots.txt?

At the moment Google (I can not find similar informations for Bing, Yahoo!, etc.) does not index Data-URI. But until it is possible I like to know if it is possible to exclude Data-URIs of being ...
1
vote
1answer
43 views

The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix?

We have a website: www.example.com which redirects to https://www.example.com. Therefore when you type www.example.com/robots.txt you are redirected to https://www.example.com/robots.txt. Another ...
3
votes
1answer
39 views

cookies and useragent is not enough ? to emulate browser?

I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt Let's take an example, I need to get this page: http://www.aliexpress.com/...
2
votes
0answers
53 views

Google Webmaster Tools reports URLs blocked by robots.txt on my AVS tube website

I added a tube (with AVS script) in Google Webmaster Tools, but the issue is with the robots.txt file which is blocking everything. Here is my AVS tube robots.txt: User-agent: * Allow: / Disallow: /...
7
votes
5answers
597 views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
2
votes
1answer
33 views

Google indexed work-in-progress page. how to delete it from Google index?

I developed a page under the following URL: example.org/dev Since the development is finished, I moved the page to the root at example.org/ For some reason, Google indexed the page during ...
2
votes
1answer
212 views

Google mobile friendly resources which are blocked by robots.txt

I have a situation when I check my website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 5 resources which are ...
5
votes
5answers
4k views

Disallow robots.txt from being accessed in a browser but still accessible by spiders?

We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
1
vote
1answer
41 views

Robots.txt disallow all except one directory and access to sitemap

robots.txt containts following instructions: User-agent: * Allow: /public/ Disallow: / Sitemap: http://sitedomain.com/sitemapindex.xml it seems that in this case sitemapindex won't be accessible ...
0
votes
2answers
45 views

Sitemap XML title in search results

I have a website that uses an XML sitemap and a robots.txt file. If I do a Google search of: site: mywebsite.co.uk "sitemap" I see the usual Google response, similar to the following image, ...
8
votes
2answers
829 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
3
votes
3answers
2k views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
8
votes
5answers
3k views

Should we modify our Joomla robots.txt after Google's announcement on crawling of CSS and JavaScript?

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html It states: For optimal rendering and indexing, our new ...
2
votes
2answers
87 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
4
votes
1answer
61 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
1
vote
1answer
203 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion). I am building other sites, ...
5
votes
3answers
71 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...