Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
1
vote
0answers
18 views

Why is Google indexing our robots.txt file and showing it in search results?

For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below. Our robots.txt file is not linked from anywhere on the site ...
1
vote
2answers
32 views

Does Google TRULY respect noindex and nofollow?

Does google REALLY respect noindex, nofollow and blocking content to being indexed and seen via .htacess or robots.txt file? I need to be sure of this 100%, I need to use duplicate content on some ...
1
vote
1answer
19 views

Effect of Submitted URLs being blocked by robots

I've got an alert on my site for pages being blocked by Robots, and am in the process of fixing it. But I'm curious what the ranking effect is, if there is one at all? And if there is an effect, does ...
1
vote
0answers
41 views

Refer to a MediaWiki website's domain in robots.txt by a variable

I have a website (built with MediaWiki 1.33.0 CMS) which contains a robots.txt file. In that file there is one line containing the literal domain of that site: Sitemap: https://example.com/sitemap/...
4
votes
1answer
47 views

Do the order of “Disallow” and “Sitemap” lines in robots.txt matter?

One can sort robots.txt this way: User-agent: DESIRED_INPUT Sitemap: https://example.com/sitemap-index.xml Disallow: / instead: User-agent: DESIRED_INPUT Disallow: / Sitemap: https://example.com/...
2
votes
1answer
12 views

Override robots.txt exclusion to allow Mediapartners-Google for AdSense

I use robots.txt to block all robots.txt compliant bots from a directory like this: User-agent: * Disallow: /sub/* How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
4
votes
1answer
49 views

Can it be destructive to disallow all MediaWiki:Special pages in robots.txt?

I consider preventing indexation of all of MediaWiki 1.33.0's Special pages; (In Hebrew, "מיוחד" means "special"): Disallow: /מיוחד:* Disallow: /index.php?title=מיוחד:* Disallow: /index.php%3Ftitle%...
2
votes
1answer
44 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
1
vote
1answer
60 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...
0
votes
0answers
9 views

Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]

Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...
1
vote
0answers
11 views

Should my sitemap and robots.txt have http or https urls for the given scenario [duplicate]

My website is running on node.js and runs in http on port 3000 I am using NGINX to enable https access so that all pages are served as https I have placed my sitemap.xml and robots.txt inside the ...
1
vote
0answers
16 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
3
votes
1answer
77 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
1
vote
1answer
24 views

Robots txt disallow on Blogger

I have a blog in Blogger, so the URL structure is always like this: www.example.com/2019/01/post.html The problem is I don't want google to crawl www.example.com/2019 (it shows all posts of 2019) ...
0
votes
1answer
49 views

Robots.txt, php.ini, .htaccess, etc

This will be my first website, so I am completely new to all of this. I cannot seem to find an answer anywhere, as to whether or not I should disallow configuration files like /php.ini or hidden ...
0
votes
1answer
28 views

Two wordpress with two sitemaps under same domain name

I have two wordpress installations under one domain name, as this: www.exampleWP1.com www.exampleWP1.com/exampleWP2/ I have both of them created in Search Console as independant properties, in Tag ...
1
vote
1answer
60 views

Page Resources Couldn't be Loaded” on GSC even after clearing everything on robots.txt

Google Search Console and Mobile-Friendly Test both give me the following two warnings for my Wordpress based website: Content wider than screen Clickable elements too close together The screenshot ...
1
vote
1answer
49 views

Is Applebot following Crawl-Delay?

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down. Does it follow Crawl-Delay from robots.txt, or not?
1
vote
1answer
23 views

Disallowed “Search” results with robots.txt and Sessions dropped

I've started working on our website and I've found millions of "Search" URL's which I don't think should be getting crawled & indexed (e.g. .../search/?q=brown&prefn1=brand&prefv1=C.P. ...
3
votes
1answer
129 views

Should one hide RTL encoded URLs in robots.txt or not?

I have a MediaWiki 1.32.0 RTL site (Hebrew) and I desire to hide some of its URLs from search engines like Google and Bing by robots.txt. The robots.txt command Disallow: /מדיה_ויקי:* can have two ...
2
votes
5answers
65 views

How to disallow (xyz.example.com) subdomain URLs in robots.txt

When I'm checking the Google results for our site's site:example.com, we get lots of index results like xyz.example.com xyz.example.com/abe xyz.example.com/abcds How do I disallow this type URL, so ...
0
votes
0answers
23 views

Why pages not blocked by robots.txt are not indexing? [duplicate]

In my google search console's coverage section, the diagram tells that there are 114 excluded pages. It says 40 of them is blocked by robots.txt and 22 of them is excluded by 'noindex' tab, but I didn'...
0
votes
1answer
32 views

Google indexing Wordpress in subdirectory disallowed in robots.txt

I have a headless wordpress site just using wordpress API. My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress. I set up robots.txt as follows: User-agent: *...
1
vote
0answers
80 views

How to allow only admin-ajax.php to be crawled? [closed]

My robots.txt looks as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Google Webmaster Tool recently raised a warning saying: Indexed, though blocked by robots.txt I ...
0
votes
0answers
7 views

Sitemap on Google Search Console not helping top 6 links to change [duplicate]

I am currently attempting to alter the way Google will display the top 6 links when you search for our URL on Google's search engine. I have read about creating a sitemap.xml file and created the file ...
2
votes
2answers
46 views

Disallow URLs with an ID from being crawled

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern: www.example.com/node/sport/category/id but not those without the ID: www.example....
1
vote
1answer
46 views

Main Website with Sub Directory = 2 Robots & Sitemaps respectively?

I have a website example.com and a subdirectory with example.com/blog/. (I can consider it as two websites) The main website example.com is the service based and it is not a WordPress site. So, I ...
0
votes
0answers
68 views

Soft error 404 and submitted URL is blocked by Robots.txt

Soft error 404 and robots txt error:- we have recently submitted sitemap xml to Search console, but we got a warning stating 131 of the page links are softerror 404 and for the home site it says ...
1
vote
1answer
25 views

Magento robots.txt rule also blocking JavaScript from WordPress

I have a Magento site and a WordPress site is used in a directory for the blog. My blog keeps getting hit as not mobile friendly and the reason I believe is that there are certain resources blocked in ...
1
vote
1answer
55 views

Yandex ignores Crawl-delay?

I try to get the Yandex Bot under control and I'm close to block it. After 2 months of robots.txt adjustments, it still keeps hammering my server from multiple IPs. User-Agent: * Crawl-delay: 4 ...
1
vote
1answer
50 views

Robot.Txt File interpretation problem?

I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages: {block name="frontend_robots_txt_user_agent"} User-agent: * {/block} {block name="...
1
vote
1answer
93 views

Do I need to disallow mediapartners-google in robots.txt for SEO?

I found that most of the websites disallow mediapartners-google in robots.txt. Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
0
votes
1answer
26 views

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory, Is this possible or I must change robots.txt for drafts starting as draft:myDraft?
1
vote
0answers
46 views

Google is trying to index the mail.mydomain.example domain and says this is a coverage issue

I have sub domains for my domains. mail.mydomain.example, ftp.mydomain.example, etc. These sub domains are configured with the DNS server for the domains. Without doing anything on my part Google ...
4
votes
1answer
56 views

Robots.txt allow directory root while disallowing all deeper URLs

I would like to allow a path and disallow everything else in the directory. User-Agent: * Allow: /example/ Disallow: /example/* Will this work? If not what would be a better way of doing this?
0
votes
2answers
118 views

Disallowing URL parameter in robots.txt

I have important URLs on my site to which I’d like to append temporary parameters. For example: example.com/tool/input will have the parameter example.com/tool/input?foo input in the URL is ...
1
vote
1answer
30 views

Google desktop crawler showing unreachable - mobile crawler is working

I noticed a sharp drop in Adsense earnings on one of my sites this week and after a bit of digging in the Search Console found errors suggesting there was a problem with my robots.txt. I checked this ...
1
vote
0answers
23 views

Canoical vs robots disallow for sorted content [duplicate]

I have a web page that displays 5 items (each with a headline, image and short content like in a microblog). Item 1 Item 2 ... Item 5 Based on the query parameter in the URL, I will be sorting the ...
1
vote
1answer
30 views

Google DMCA on a page that was forbidden in robots.txt

I've received a "Notice of DMCA removal from Google Search" for a page was inaccessible to google as per robots.txt. Why did Google have this page in search if it was disallowed from visiting it at ...
0
votes
2answers
327 views

How to configure the robots.txt file with a multilingual site?

I have a site with Drupal 8 and I activate English and French (Language from the URL path prefix). Here is what my url resembles : https://www.example.com/en/node/add https://www.example.com/fr/node/...
1
vote
1answer
108 views

Should tag and author pages be disallowed in robots.txt?

According to some articles that talk about crawl budget, we should decrease the number pages crawled and provide the only important and valuable pages to search engine crawlers. We have 3,000 ...
1
vote
1answer
55 views

Hacked site created foreign language tags pages, can we block them in robots.txt to remove them from Google?

Our WordPress website was hacked which is now revamped and updated to the WP v5.0.3 but still Tags and 35fc4a-50164960-421910 or 6d285b-31601402-365308 type URLs are indexed in Google. Can we block ...
2
votes
1answer
41 views

Prevent dynamically generated thumbnails from being crawled

I have been using Raven Tools, and I want these dynamically generated thumbnails to be not crawled or not be flagged as No TITLE images. I tried adding X-Robots-Tag "noindex, nofollow" on each image ...
6
votes
1answer
786 views

How to tell bots to forget a website and reindex it from scratch

I was wondering if there was anything I could add to robots.txt or sitemap to tell bots to completely forget everything they know about a website and index it from scratch? Context: After replacing a ...
2
votes
1answer
39 views

Can I use robots.txt to allow Google to access some folders, but disallow everything else?

I want to allow Google bots to crawl only my index.php and the content of some folders. Would that work? User-agent: * Allow: /index*.php Allow: /folder1/ Allow: /folder2/ Allow: /folder3/ Allow: /...
1
vote
1answer
50 views

How to prevent pages from being indexed in Google Search by using robots.txt? [duplicate]

In my robots.txt I disallowed some of the "thank you" pages on my site that I don't want to indexed by Google or be in Google search results. But those pages have recently become searchable on Google. ...
1
vote
1answer
147 views

How to add robots.txt to my WordPress site?

How to add robots.txt to my wordpress.com site?
1
vote
1answer
110 views

Google can't access my page due to robots.txt but the testing tool says it's OK

My robots.txt is the typical that allows all bots everywhere User-agent: * Disallow: / When I test it with Google tool everything is OK. But in Google Search Console I get an error that my site can'...
4
votes
2answers
1k views

Google Search Console says “submitted URL blocked by robots.txt” even though there is no robots.txt file

Today I got a email notice from Google that there were search problems with my site. I logged into Google Search Console. It shows that 1 file was being blocked by robots.txt, and has been for ...
0
votes
1answer
101 views

Pages uploaded to WordPress as media are causing mobile usability errors in Google Search Console [duplicate]

the new search console is pointing out usability errors on mobile devices on WordPress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...