Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
2
votes
0answers
31 views

Ahrefs reports “Fetching robots.txt took too long”

My robots: User-agent: * Disallow: /wp-admin Disallow: /sistema Disallow: /site Disallow: /old Sitemap: http://www.example.com.br/page-sitemap.xml In the google test tool: https://www.google.com/...
1
vote
0answers
15 views

Can I use a wildcard rule to prevent Yandex from indexing all tag and date URLs created by Joomla K2?

I have a site based on joomla and K2 component. I use K2 Tags and Calendar, but i have problem with pages indexing When yandex.ru begin to index it i find out that it index my tags and SEO test sites ...
0
votes
0answers
6 views

301 redirect to robots.txt file to a different domain , may be to a cloud front layer

I would like to know if i can put a 301 redirect on robots.txt file and navigate it to a different location? example site= https://www.example.net/robots.txt redirects to https://differentdomain.net/...
2
votes
1answer
26 views

Google Search Console Warning: “ Indexed, though blocked by robots.txt ” (BUG)

Two days ago I've got an email with a warning from Google Search Console, telling me that 3 URLs from my site where indexed, but were blocked by robots.txt. Since I first deployed my website, I haven'...
2
votes
2answers
118 views

What does crawl-delay: 1 mean if it is interpreted as is?

I have a problem of understanding what crawl-delay: 1 means if it isn't ignored and actually being interpreted as is by a given web crawler. I understood from this answer by Stephen Ostermiller that: ...
0
votes
0answers
9 views

Google does not show all our posts in search results [duplicate]

We have issue with google search console for web site> https://bit.ly/38Xhm97 Our site is written in 2 "languages", Latin and Cyrillic. Sometimes we have problem that our cyrillic posts das not show ...
2
votes
1answer
145 views

How to prevent google index for a specific subpath?

I have my website, say www.example.com, and I want it to be indexed. Anyway I also have one section, www.example.com/section that I don't want to be indexed. I tried adding the following headers to ...
1
vote
2answers
122 views

crawl-delay: x ------— has someone here managed to understand how the math works there?

From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than ...
1
vote
2answers
39 views

Restrict crawling of region/lang combinations other than the provided ones in robots.txt

I want to allow crawling of my website only if the URL starts with accepted region/language combinations which are us/en, gb/en and fr/fr. Other language combinations must be restricted. On the other ...
4
votes
1answer
59 views

Googlebot ignores robots.txt

Two months ago i disallowed some directories from crawling with robots.txt. Since then i monitor logfiles everyday and realize, that Googlebot absolutely ignores robots.txt. Really, it crawls every ...
1
vote
0answers
36 views

Google Search Console reports Soft 404 on an Excel file

since mid of this month, an Excel file that used to be indexed in Google Search is reported as a Soft 404 on Search Console File: https://neoschronos.com/assets/lean-canvas.xls We have multiple ...
0
votes
0answers
17 views

How to force deindex URLs from Google index ASAP? [duplicate]

Remove pages from Google index Hello, I have bought previously owned domain. Right now it have 150 - 200 URLs indexed by Google that are not related to my current website (powered by wordpress). ...
1
vote
2answers
35 views

Disabling main page url from robots.txt

Example robots.txt: User-agent: * Disallow: / Would it work? I have urls, like https://example.com/en, https://example.com/de I don't want search engines to index https://example.com without ...
1
vote
1answer
22 views

Hide /path from search engines, except for /path/that-i-want-indexed

In our robots.txt we have something like User-agent: * Disallow: /v6.0 Disallow: /v5.8 Disallow: /v5.7 Disallow: /v5.5 Disallow: /v5.1 Disallow: /v5.0 Which properly prevents search engines from ...
2
votes
1answer
68 views

Launching a new website on a subdomain but keeping exsisting website running

We're about to launch a new Drupal 8 website for a client, as not all areas of the current website will be on the new platform, and due to apparent legacy complications, the decision has been made to ...
3
votes
1answer
76 views

Why are bots repeatedly crawling pages of my site that I never created?

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
2
votes
0answers
97 views

In robots.txt - what are the directives from WBCPBlocker

On this site's robots.txt there is a comment preceding many rows of directives blocking SEO tool bots and was wondering if someone could decipher it for me. There's the standard User-agent: * Allow:...
3
votes
1answer
211 views

Prevent Google from indexing images

I have a site where all my images are stored in a specific folder, for example /images/984728176386375837629856e.png but I don't want this page to appear on the SERP. Can I disable the indexing of the ...
3
votes
1answer
27 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
2
votes
1answer
64 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
1
vote
2answers
97 views

Google Search Console and Sitemap set-up for a website subfolder

I'm managing a website https://www.example.com/ and I've just launched a new website https://www.example.com/talkabout (so with a subfolder). I've already set up a GSC account, sitemaps (html and ...
2
votes
1answer
25 views

Can robots.txt be used to prevent bots from seeing lazily loaded content?

Let's say that googlebot is scraping https://example.com/page. example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page /page lazy loads content using /...
2
votes
1answer
58 views

Google's URL inspection says my image URL is blocked by robots.txt — I don't even have one!

I just discovered our image system domain is not being crawled by Google for a long time. The reason is that all the URLs seem to be blocked by robots.txt -- but I don't even have one. Disclaimer: ...
1
vote
0answers
21 views

Why is Google indexing our robots.txt file and showing it in search results?

For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below. Our robots.txt file is not linked from anywhere on the site ...
1
vote
2answers
100 views

Does Google TRULY respect noindex and nofollow?

Does google REALLY respect noindex, nofollow and blocking content to being indexed and seen via .htacess or robots.txt file? I need to be sure of this 100%, I need to use duplicate content on some ...
1
vote
1answer
23 views

Effect of Submitted URLs being blocked by robots

I've got an alert on my site for pages being blocked by Robots, and am in the process of fixing it. But I'm curious what the ranking effect is, if there is one at all? And if there is an effect, does ...
1
vote
0answers
42 views

Refer to a MediaWiki website's domain in robots.txt by a variable

I have a website (built with MediaWiki 1.33.0 CMS) which contains a robots.txt file. In that file there is one line containing the literal domain of that site: Sitemap: https://example.com/sitemap/...
4
votes
1answer
102 views

Do the order of “Disallow” and “Sitemap” lines in robots.txt matter?

One can sort robots.txt this way: User-agent: DESIRED_INPUT Sitemap: https://example.com/sitemap-index.xml Disallow: / instead: User-agent: DESIRED_INPUT Disallow: / Sitemap: https://example.com/...
2
votes
1answer
43 views

Override robots.txt exclusion to allow Mediapartners-Google for AdSense

I use robots.txt to block all robots.txt compliant bots from a directory like this: User-agent: * Disallow: /sub/* How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
4
votes
1answer
54 views

Can it be destructive to disallow all MediaWiki:Special pages in robots.txt?

I consider preventing indexation of all of MediaWiki 1.33.0's Special pages; (In Hebrew, "מיוחד" means "special"): Disallow: /מיוחד:* Disallow: /index.php?title=מיוחד:* Disallow: /index.php%3Ftitle%...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
1
vote
1answer
69 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...
0
votes
0answers
9 views

Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]

Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...
1
vote
0answers
20 views

Should my sitemap and robots.txt have http or https urls for the given scenario [duplicate]

My website is running on node.js and runs in http on port 3000 I am using NGINX to enable https access so that all pages are served as https I have placed my sitemap.xml and robots.txt inside the ...
1
vote
0answers
27 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
3
votes
1answer
80 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
1
vote
1answer
53 views

Robots txt disallow on Blogger

I have a blog in Blogger, so the URL structure is always like this: www.example.com/2019/01/post.html The problem is I don't want google to crawl www.example.com/2019 (it shows all posts of 2019) ...
0
votes
1answer
58 views

Robots.txt, php.ini, .htaccess, etc

This will be my first website, so I am completely new to all of this. I cannot seem to find an answer anywhere, as to whether or not I should disallow configuration files like /php.ini or hidden ...
0
votes
1answer
28 views

Two wordpress with two sitemaps under same domain name

I have two wordpress installations under one domain name, as this: www.exampleWP1.com www.exampleWP1.com/exampleWP2/ I have both of them created in Search Console as independant properties, in Tag ...
1
vote
1answer
126 views

Page Resources Couldn't be Loaded” on GSC even after clearing everything on robots.txt

Google Search Console and Mobile-Friendly Test both give me the following two warnings for my Wordpress based website: Content wider than screen Clickable elements too close together The screenshot ...
1
vote
1answer
138 views

Is Applebot following Crawl-Delay?

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down. Does it follow Crawl-Delay from robots.txt, or not?
1
vote
1answer
24 views

Disallowed “Search” results with robots.txt and Sessions dropped

I've started working on our website and I've found millions of "Search" URL's which I don't think should be getting crawled & indexed (e.g. .../search/?q=brown&prefn1=brand&prefv1=C.P. ...
3
votes
1answer
138 views

Should one hide RTL encoded URLs in robots.txt or not?

I have a MediaWiki 1.32.0 RTL site (Hebrew) and I desire to hide some of its URLs from search engines like Google and Bing by robots.txt. The robots.txt command Disallow: /מדיה_ויקי:* can have two ...
2
votes
5answers
290 views

How to disallow (xyz.example.com) subdomain URLs in robots.txt

When I'm checking the Google results for our site's site:example.com, we get lots of index results like xyz.example.com xyz.example.com/abe xyz.example.com/abcds How do I disallow this type URL, so ...
0
votes
0answers
23 views

Why pages not blocked by robots.txt are not indexing? [duplicate]

In my google search console's coverage section, the diagram tells that there are 114 excluded pages. It says 40 of them is blocked by robots.txt and 22 of them is excluded by 'noindex' tab, but I didn'...
0
votes
1answer
111 views

Google indexing Wordpress in subdirectory disallowed in robots.txt

I have a headless wordpress site just using wordpress API. My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress. I set up robots.txt as follows: User-agent: *...
1
vote
0answers
212 views

How to allow only admin-ajax.php to be crawled? [closed]

My robots.txt looks as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Google Webmaster Tool recently raised a warning saying: Indexed, though blocked by robots.txt I ...
0
votes
0answers
9 views

Sitemap on Google Search Console not helping top 6 links to change [duplicate]

I am currently attempting to alter the way Google will display the top 6 links when you search for our URL on Google's search engine. I have read about creating a sitemap.xml file and created the file ...
2
votes
2answers
93 views

Disallow URLs with an ID from being crawled

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern: www.example.com/node/sport/category/id but not those without the ID: www.example....
1
vote
1answer
81 views

Main Website with Sub Directory = 2 Robots & Sitemaps respectively?

I have a website example.com and a subdirectory with example.com/blog/. (I can consider it as two websites) The main website example.com is the service based and it is not a WordPress site. So, I ...

1
2 3 4 5
13