Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Low HTML Text Ratio Warning on 1200 Pages of my Website [closed]

My website has News section and every page in News have content around 200 characters which gives out SEO Warning of "Low HTML Text Ratio" and this news pages are my important source of Content around ...
0
votes
0answers
16 views

How to force deindex URLs from Google index ASAP? [duplicate]

Remove pages from Google index Hello, I have bought previously owned domain. Right now it have 150 - 200 URLs indexed by Google that are not related to my current website (powered by wordpress). ...
1
vote
2answers
32 views

Disabling main page url from robots.txt

Example robots.txt: User-agent: * Disallow: / Would it work? I have urls, like https://example.com/en, https://example.com/de I don't want search engines to index https://example.com without ...
1
vote
1answer
17 views

Hide /path from search engines, except for /path/that-i-want-indexed

In our robots.txt we have something like User-agent: * Disallow: /v6.0 Disallow: /v5.8 Disallow: /v5.7 Disallow: /v5.5 Disallow: /v5.1 Disallow: /v5.0 Which properly prevents search engines from ...
1
vote
1answer
58 views

Launching a new website on a subdomain but keeping exsisting website running

We're about to launch a new Drupal 8 website for a client, as not all areas of the current website will be on the new platform, and due to apparent legacy complications, the decision has been made to ...
3
votes
1answer
62 views

Why are bots repeatedly crawling pages of my site that I never created?

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
2
votes
0answers
92 views

In robots.txt - what are the directives from WBCPBlocker

On this site's robots.txt there is a comment preceding many rows of directives blocking SEO tool bots and was wondering if someone could decipher it for me. There's the standard User-agent: * Allow:...
3
votes
1answer
155 views

Prevent Google from indexing images

I have a site where all my images are stored in a specific folder, for example /images/984728176386375837629856e.png but I don't want this page to appear on the SERP. Can I disable the indexing of the ...
2
votes
0answers
15 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
2
votes
1answer
60 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
1
vote
2answers
58 views

Google Search Console and Sitemap set-up for a website subfolder

I'm managing a website https://www.example.com/ and I've just launched a new website https://www.example.com/talkabout (so with a subfolder). I've already set up a GSC account, sitemaps (html and ...
2
votes
1answer
25 views

Can robots.txt be used to prevent bots from seeing lazily loaded content?

Let's say that googlebot is scraping https://example.com/page. example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page /page lazy loads content using /...
2
votes
1answer
45 views

Google's URL inspection says my image URL is blocked by robots.txt — I don't even have one!

I just discovered our image system domain is not being crawled by Google for a long time. The reason is that all the URLs seem to be blocked by robots.txt -- but I don't even have one. Disclaimer: ...
1
vote
0answers
20 views

Why is Google indexing our robots.txt file and showing it in search results?

For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below. Our robots.txt file is not linked from anywhere on the site ...
1
vote
2answers
65 views

Does Google TRULY respect noindex and nofollow?

Does google REALLY respect noindex, nofollow and blocking content to being indexed and seen via .htacess or robots.txt file? I need to be sure of this 100%, I need to use duplicate content on some ...
1
vote
1answer
23 views

Effect of Submitted URLs being blocked by robots

I've got an alert on my site for pages being blocked by Robots, and am in the process of fixing it. But I'm curious what the ranking effect is, if there is one at all? And if there is an effect, does ...
1
vote
0answers
42 views

Refer to a MediaWiki website's domain in robots.txt by a variable

I have a website (built with MediaWiki 1.33.0 CMS) which contains a robots.txt file. In that file there is one line containing the literal domain of that site: Sitemap: https://example.com/sitemap/...
4
votes
1answer
81 views

Do the order of “Disallow” and “Sitemap” lines in robots.txt matter?

One can sort robots.txt this way: User-agent: DESIRED_INPUT Sitemap: https://example.com/sitemap-index.xml Disallow: / instead: User-agent: DESIRED_INPUT Disallow: / Sitemap: https://example.com/...
2
votes
1answer
35 views

Override robots.txt exclusion to allow Mediapartners-Google for AdSense

I use robots.txt to block all robots.txt compliant bots from a directory like this: User-agent: * Disallow: /sub/* How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
4
votes
1answer
53 views

Can it be destructive to disallow all MediaWiki:Special pages in robots.txt?

I consider preventing indexation of all of MediaWiki 1.33.0's Special pages; (In Hebrew, "מיוחד" means "special"): Disallow: /מיוחד:* Disallow: /index.php?title=מיוחד:* Disallow: /index.php%3Ftitle%...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
1
vote
1answer
62 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...
0
votes
0answers
9 views

Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]

Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...
1
vote
0answers
17 views

Should my sitemap and robots.txt have http or https urls for the given scenario [duplicate]

My website is running on node.js and runs in http on port 3000 I am using NGINX to enable https access so that all pages are served as https I have placed my sitemap.xml and robots.txt inside the ...
1
vote
0answers
20 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
3
votes
1answer
78 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
1
vote
1answer
39 views

Robots txt disallow on Blogger

I have a blog in Blogger, so the URL structure is always like this: www.example.com/2019/01/post.html The problem is I don't want google to crawl www.example.com/2019 (it shows all posts of 2019) ...
0
votes
1answer
54 views

Robots.txt, php.ini, .htaccess, etc

This will be my first website, so I am completely new to all of this. I cannot seem to find an answer anywhere, as to whether or not I should disallow configuration files like /php.ini or hidden ...
0
votes
1answer
28 views

Two wordpress with two sitemaps under same domain name

I have two wordpress installations under one domain name, as this: www.exampleWP1.com www.exampleWP1.com/exampleWP2/ I have both of them created in Search Console as independant properties, in Tag ...
1
vote
1answer
105 views

Page Resources Couldn't be Loaded” on GSC even after clearing everything on robots.txt

Google Search Console and Mobile-Friendly Test both give me the following two warnings for my Wordpress based website: Content wider than screen Clickable elements too close together The screenshot ...
1
vote
1answer
98 views

Is Applebot following Crawl-Delay?

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down. Does it follow Crawl-Delay from robots.txt, or not?
1
vote
1answer
24 views

Disallowed “Search” results with robots.txt and Sessions dropped

I've started working on our website and I've found millions of "Search" URL's which I don't think should be getting crawled & indexed (e.g. .../search/?q=brown&prefn1=brand&prefv1=C.P. ...
3
votes
1answer
133 views

Should one hide RTL encoded URLs in robots.txt or not?

I have a MediaWiki 1.32.0 RTL site (Hebrew) and I desire to hide some of its URLs from search engines like Google and Bing by robots.txt. The robots.txt command Disallow: /מדיה_ויקי:* can have two ...
2
votes
5answers
179 views

How to disallow (xyz.example.com) subdomain URLs in robots.txt

When I'm checking the Google results for our site's site:example.com, we get lots of index results like xyz.example.com xyz.example.com/abe xyz.example.com/abcds How do I disallow this type URL, so ...
0
votes
0answers
23 views

Why pages not blocked by robots.txt are not indexing? [duplicate]

In my google search console's coverage section, the diagram tells that there are 114 excluded pages. It says 40 of them is blocked by robots.txt and 22 of them is excluded by 'noindex' tab, but I didn'...
0
votes
1answer
81 views

Google indexing Wordpress in subdirectory disallowed in robots.txt

I have a headless wordpress site just using wordpress API. My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress. I set up robots.txt as follows: User-agent: *...
1
vote
0answers
160 views

How to allow only admin-ajax.php to be crawled? [closed]

My robots.txt looks as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Google Webmaster Tool recently raised a warning saying: Indexed, though blocked by robots.txt I ...
0
votes
0answers
7 views

Sitemap on Google Search Console not helping top 6 links to change [duplicate]

I am currently attempting to alter the way Google will display the top 6 links when you search for our URL on Google's search engine. I have read about creating a sitemap.xml file and created the file ...
2
votes
2answers
63 views

Disallow URLs with an ID from being crawled

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern: www.example.com/node/sport/category/id but not those without the ID: www.example....
1
vote
1answer
70 views

Main Website with Sub Directory = 2 Robots & Sitemaps respectively?

I have a website example.com and a subdirectory with example.com/blog/. (I can consider it as two websites) The main website example.com is the service based and it is not a WordPress site. So, I ...
0
votes
0answers
90 views

Soft error 404 and submitted URL is blocked by Robots.txt

Soft error 404 and robots txt error:- we have recently submitted sitemap xml to Search console, but we got a warning stating 131 of the page links are softerror 404 and for the home site it says ...
1
vote
1answer
28 views

Magento robots.txt rule also blocking JavaScript from WordPress

I have a Magento site and a WordPress site is used in a directory for the blog. My blog keeps getting hit as not mobile friendly and the reason I believe is that there are certain resources blocked in ...
1
vote
1answer
73 views

Yandex ignores Crawl-delay?

I try to get the Yandex Bot under control and I'm close to block it. After 2 months of robots.txt adjustments, it still keeps hammering my server from multiple IPs. User-Agent: * Crawl-delay: 4 ...
1
vote
1answer
54 views

Robot.Txt File interpretation problem?

I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages: {block name="frontend_robots_txt_user_agent"} User-agent: * {/block} {block name="...
1
vote
1answer
141 views

Do I need to disallow mediapartners-google in robots.txt for SEO?

I found that most of the websites disallow mediapartners-google in robots.txt. Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
0
votes
1answer
27 views

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory, Is this possible or I must change robots.txt for drafts starting as draft:myDraft?
1
vote
0answers
50 views

Google is trying to index the mail.mydomain.example domain and says this is a coverage issue

I have sub domains for my domains. mail.mydomain.example, ftp.mydomain.example, etc. These sub domains are configured with the DNS server for the domains. Without doing anything on my part Google ...
4
votes
1answer
58 views

Robots.txt allow directory root while disallowing all deeper URLs

I would like to allow a path and disallow everything else in the directory. User-Agent: * Allow: /example/ Disallow: /example/* Will this work? If not what would be a better way of doing this?
0
votes
2answers
183 views

Disallowing URL parameter in robots.txt

I have important URLs on my site to which I’d like to append temporary parameters. For example: example.com/tool/input will have the parameter example.com/tool/input?foo input in the URL is ...
1
vote
1answer
32 views

Google desktop crawler showing unreachable - mobile crawler is working

I noticed a sharp drop in Adsense earnings on one of my sites this week and after a bit of digging in the Search Console found errors suggesting there was a problem with my robots.txt. I checked this ...