Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
2
votes
1answer
6 views

Will this robots.txt file work?

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for site.com/...
0
votes
1answer
48 views

Google doesn't send me to correct robots.txt directory

So I want to set a robots.txt in my site. Pretty straight forward place in root directory and good. But Google spits out a 404 error telling me the robots.txt is not in the place it expects. Not ...
1
vote
1answer
40 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...
0
votes
0answers
5 views

Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]

Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...
4
votes
2answers
868 views

Google Search Console says “submitted URL blocked by robots.txt” even though there is no robots.txt file

Today I got a email notice from Google that there were search problems with my site. I logged into Google Search Console. It shows that 1 file was being blocked by robots.txt, and has been for ...
5
votes
3answers
13k views

How to resolve Google “Indexed, though blocked by robots.txt”

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
2
votes
0answers
21 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
0
votes
0answers
8 views

Should my sitemap and robots.txt have http or https urls for the given scenario [duplicate]

My website is running on node.js and runs in http on port 3000 I am using NGINX to enable https access so that all pages are served as https I have placed my sitemap.xml and robots.txt inside the ...
1
vote
0answers
11 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
1
vote
1answer
21 views

Robots txt disallow on Blogger

I have a blog in Blogger, so the URL structure is always like this: www.example.com/2019/01/post.html The problem is I don't want google to crawl www.example.com/2019 (it shows all posts of 2019) ...
3
votes
1answer
944 views

Do I need to add subdomain sitemap / robots.txt separately?

Say I have a domain example.com And it has a subdomain sub.example.com And another subdomain internal.example.com I have some links on example.com and sub.example.com for Google to crawl. But I don'...
0
votes
1answer
43 views

Robots.txt, php.ini, .htaccess, etc

This will be my first website, so I am completely new to all of this. I cannot seem to find an answer anywhere, as to whether or not I should disallow configuration files like /php.ini or hidden ...
1
vote
1answer
34 views

Page Resources Couldn't be Loaded” on GSC even after clearing everything on robots.txt

Google Search Console and Mobile-Friendly Test both give me the following two warnings for my Wordpress based website: Content wider than screen Clickable elements too close together The screenshot ...
0
votes
1answer
27 views

Two wordpress with two sitemaps under same domain name

I have two wordpress installations under one domain name, as this: www.exampleWP1.com www.exampleWP1.com/exampleWP2/ I have both of them created in Search Console as independant properties, in Tag ...
1
vote
1answer
22 views

Is Applebot following Crawl-Delay?

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down. Does it follow Crawl-Delay from robots.txt, or not?
2
votes
1answer
106 views

Should one hide RTL encoded URLs in robots.txt or not?

I have a MediaWiki 1.32.0 RTL site (Hebrew) and I desire to hide some of its URLs from search engines like Google and Bing by robots.txt. The robots.txt command Disallow: /מדיה_ויקי:* can have two ...
2
votes
5answers
47 views

How to disallow (xyz.example.com) subdomain URLs in robots.txt

When I'm checking the Google results for our site's site:example.com, we get lots of index results like xyz.example.com xyz.example.com/abe xyz.example.com/abcds How do I disallow this type URL, so ...
1
vote
1answer
23 views

Disallowed “Search” results with robots.txt and Sessions dropped

I've started working on our website and I've found millions of "Search" URL's which I don't think should be getting crawled & indexed (e.g. .../search/?q=brown&prefn1=brand&prefv1=C.P. ...
0
votes
0answers
23 views

Why pages not blocked by robots.txt are not indexing? [duplicate]

In my google search console's coverage section, the diagram tells that there are 114 excluded pages. It says 40 of them is blocked by robots.txt and 22 of them is excluded by 'noindex' tab, but I didn'...
0
votes
1answer
21 views

Google indexing Wordpress in subdirectory disallowed in robots.txt

I have a headless wordpress site just using wordpress API. My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress. I set up robots.txt as follows: User-agent: *...
10
votes
1answer
1k views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
3
votes
1answer
2k views

Should I use “Disallow”, “NoFollow”, or “NoIndex” for paginated listing pages?

I own a web site which has mainly two types of pages: Content pages: they are pretty much static, and link to the previous and next content page (chronologically). The structure is that of a double ...
5
votes
6answers
265 views

How do you prevent web pages from appearing in the Google search index?

I have some web pages that Google has indexed. I have now added a robots.txt file to exclude them, but should I also add a noindex tag? If so, then how do I do so?
2
votes
1answer
864 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://castbird-sourcing.blogspot.com/feeds/posts/default?orderby=...
1
vote
0answers
53 views

How to allow only admin-ajax.php to be crawled? [closed]

My robots.txt looks as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Google Webmaster Tool recently raised a warning saying: Indexed, though blocked by robots.txt I ...
0
votes
0answers
6 views

Sitemap on Google Search Console not helping top 6 links to change [duplicate]

I am currently attempting to alter the way Google will display the top 6 links when you search for our URL on Google's search engine. I have read about creating a sitemap.xml file and created the file ...
2
votes
2answers
39 views

Disallow URLs with an ID from being crawled

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern: www.example.com/node/sport/category/id but not those without the ID: www.example....
6
votes
1answer
3k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
1
vote
1answer
35 views

Main Website with Sub Directory = 2 Robots & Sitemaps respectively?

I have a website example.com and a subdirectory with example.com/blog/. (I can consider it as two websites) The main website example.com is the service based and it is not a WordPress site. So, I ...
0
votes
0answers
50 views

Soft error 404 and submitted URL is blocked by Robots.txt

Soft error 404 and robots txt error:- we have recently submitted sitemap xml to Search console, but we got a warning stating 131 of the page links are softerror 404 and for the home site it says ...
1
vote
1answer
25 views

Magento robots.txt rule also blocking JavaScript from WordPress

I have a Magento site and a WordPress site is used in a directory for the blog. My blog keeps getting hit as not mobile friendly and the reason I believe is that there are certain resources blocked in ...
1
vote
1answer
46 views

Yandex ignores Crawl-delay?

I try to get the Yandex Bot under control and I'm close to block it. After 2 months of robots.txt adjustments, it still keeps hammering my server from multiple IPs. User-Agent: * Crawl-delay: 4 ...
1
vote
1answer
47 views

Robot.Txt File interpretation problem?

I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages: {block name="frontend_robots_txt_user_agent"} User-agent: * {/block} {block name="...
0
votes
1answer
26 views

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory, Is this possible or I must change robots.txt for drafts starting as draft:myDraft?
0
votes
1answer
97 views

Pages uploaded to WordPress as media are causing mobile usability errors in Google Search Console [duplicate]

the new search console is pointing out usability errors on mobile devices on WordPress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...
1
vote
1answer
67 views

Do I need to disallow mediapartners-google in robots.txt for SEO?

I found that most of the websites disallow mediapartners-google in robots.txt. Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
1
vote
0answers
41 views

Google is trying to index the mail.mydomain.example domain and says this is a coverage issue

I have sub domains for my domains. mail.mydomain.example, ftp.mydomain.example, etc. These sub domains are configured with the DNS server for the domains. Without doing anything on my part Google ...
4
votes
1answer
56 views

Robots.txt allow directory root while disallowing all deeper URLs

I would like to allow a path and disallow everything else in the directory. User-Agent: * Allow: /example/ Disallow: /example/* Will this work? If not what would be a better way of doing this?
0
votes
2answers
100 views

Disallowing URL parameter in robots.txt

I have important URLs on my site to which I’d like to append temporary parameters. For example: example.com/tool/input will have the parameter example.com/tool/input?foo input in the URL is ...
1
vote
1answer
28 views

Google desktop crawler showing unreachable - mobile crawler is working

I noticed a sharp drop in Adsense earnings on one of my sites this week and after a bit of digging in the Search Console found errors suggesting there was a problem with my robots.txt. I checked this ...
2
votes
1answer
2k views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity -...
1
vote
3answers
8k views

How to fix robots.txt file error in my GWT?

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file. My blog’s robots.txt file ...
9
votes
3answers
8k views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
1
vote
0answers
23 views

Canoical vs robots disallow for sorted content [duplicate]

I have a web page that displays 5 items (each with a headline, image and short content like in a microblog). Item 1 Item 2 ... Item 5 Based on the query parameter in the URL, I will be sorting the ...
3
votes
1answer
333 views

Sitemap in different domain and robots.txt crawling

I have the following subdomains: www.example.com api.example.com The subdomain api.example.com is supposed to be invisible to Google. Its robots.txt had a Disallow / directive. But I don't want any ...
0
votes
1answer
26 views

Google DMCA on a page that was forbidden in robots.txt

I've received a "Notice of DMCA removal from Google Search" for a page was inaccessible to google as per robots.txt. Why did Google have this page in search if it was disallowed from visiting it at ...
0
votes
2answers
231 views

How to configure the robots.txt file with a multilingual site?

I have a site with Drupal 8 and I activate English and French (Language from the URL path prefix). Here is what my url resembles : https://www.example.com/en/node/add https://www.example.com/fr/node/...
1
vote
1answer
84 views

Should tag and author pages be disallowed in robots.txt?

According to some articles that talk about crawl budget, we should decrease the number pages crawled and provide the only important and valuable pages to search engine crawlers. We have 3,000 ...
1
vote
1answer
55 views

Hacked site created foreign language tags pages, can we block them in robots.txt to remove them from Google?

Our WordPress website was hacked which is now revamped and updated to the WP v5.0.3 but still Tags and 35fc4a-50164960-421910 or 6d285b-31601402-365308 type URLs are indexed in Google. Can we block ...
14
votes
3answers
3k views

What is a minimum valid robots.txt file?

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a ...