Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
1
vote
2answers
38 views

Restrict crawling of region/lang combinations other than the provided ones in robots.txt

I want to allow crawling of my website only if the URL starts with accepted region/language combinations which are us/en, gb/en and fr/fr. Other language combinations must be restricted. On the other ...
0
votes
0answers
9 views

Google does not show all our posts in search results [duplicate]

We have issue with google search console for web site> https://bit.ly/38Xhm97 Our site is written in 2 "languages", Latin and Cyrillic. Sometimes we have problem that our cyrillic posts das not show ...
2
votes
1answer
138 views

How to prevent google index for a specific subpath?

I have my website, say www.example.com, and I want it to be indexed. Anyway I also have one section, www.example.com/section that I don't want to be indexed. I tried adding the following headers to ...
1
vote
1answer
70 views

crawl-delay: x ------— has someone here managed to understand how the math works there?

From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than ...
1
vote
3answers
8k views

How to fix robots.txt file error in my GWT?

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file. My blog’s robots.txt file ...
3
votes
1answer
23 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
23
votes
2answers
15k views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
6
votes
5answers
21k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
4
votes
1answer
46 views

Googlebot ignores robots.txt

Two months ago i disallowed some directories from crawling with robots.txt. Since then i monitor logfiles everyday and realize, that Googlebot absolutely ignores robots.txt. Really, it crawls every ...
4
votes
1answer
2k views

Should I use “Disallow”, “NoFollow”, or “NoIndex” for paginated listing pages?

I own a web site which has mainly two types of pages: Content pages: they are pretty much static, and link to the previous and next content page (chronologically). The structure is that of a double ...
2
votes
1answer
67 views

Launching a new website on a subdomain but keeping exsisting website running

We're about to launch a new Drupal 8 website for a client, as not all areas of the current website will be on the new platform, and due to apparent legacy complications, the decision has been made to ...
9
votes
10answers
24k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
1
vote
0answers
27 views

Google Search Console reports Soft 404 on an Excel file

since mid of this month, an Excel file that used to be indexed in Google Search is reported as a Soft 404 on Search Console File: https://neoschronos.com/assets/lean-canvas.xls We have multiple ...
1
vote
2answers
34 views

Disabling main page url from robots.txt

Example robots.txt: User-agent: * Disallow: / Would it work? I have urls, like https://example.com/en, https://example.com/de I don't want search engines to index https://example.com without ...
0
votes
0answers
16 views

How to force deindex URLs from Google index ASAP? [duplicate]

Remove pages from Google index Hello, I have bought previously owned domain. Right now it have 150 - 200 URLs indexed by Google that are not related to my current website (powered by wordpress). ...
1
vote
1answer
20 views

Hide /path from search engines, except for /path/that-i-want-indexed

In our robots.txt we have something like User-agent: * Disallow: /v6.0 Disallow: /v5.8 Disallow: /v5.7 Disallow: /v5.5 Disallow: /v5.1 Disallow: /v5.0 Which properly prevents search engines from ...
3
votes
1answer
71 views

Why are bots repeatedly crawling pages of my site that I never created?

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
4
votes
1answer
2k views

How to have 2 sitemaps - 1 for main site the other for wordpress in subfolder

Background My site runs on a dedicated server and and has Wordpress in a subfolder. I have my mainsite which is not wordpress and that has a sitemap - for example example.com/sitemap.xml ... that ...
2
votes
0answers
95 views

In robots.txt - what are the directives from WBCPBlocker

On this site's robots.txt there is a comment preceding many rows of directives blocking SEO tool bots and was wondering if someone could decipher it for me. There's the standard User-agent: * Allow:...
3
votes
1answer
165 views

Prevent Google from indexing images

I have a site where all my images are stored in a specific folder, for example /images/984728176386375837629856e.png but I don't want this page to appear on the SERP. Can I disable the indexing of the ...
2
votes
1answer
62 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
1
vote
2answers
82 views

Google Search Console and Sitemap set-up for a website subfolder

I'm managing a website https://www.example.com/ and I've just launched a new website https://www.example.com/talkabout (so with a subfolder). I've already set up a GSC account, sitemaps (html and ...
2
votes
1answer
25 views

Can robots.txt be used to prevent bots from seeing lazily loaded content?

Let's say that googlebot is scraping https://example.com/page. example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page /page lazy loads content using /...
2
votes
1answer
50 views

Google's URL inspection says my image URL is blocked by robots.txt — I don't even have one!

I just discovered our image system domain is not being crawled by Google for a long time. The reason is that all the URLs seem to be blocked by robots.txt -- but I don't even have one. Disclaimer: ...
5
votes
1answer
2k views

Should RSS feeds be disallowed in robots.txt?

I have RSS feeds on my site. I've decided to follow Stack Exchange and disallow my RSS feeds in the robots.txt. I don't want search engine to display the RSS feed page to people, that's not really a ...
16
votes
3answers
9k views

Can robots.txt be in a server's sub-directory?

I have a sub-directory that I would like to hide from the search engine web crawlers. One way to do this is to use a robots.txt in the root directory of the server (the standard way). However, anyone ...
1
vote
0answers
20 views

Why is Google indexing our robots.txt file and showing it in search results?

For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below. Our robots.txt file is not linked from anywhere on the site ...
1
vote
2answers
85 views

Does Google TRULY respect noindex and nofollow?

Does google REALLY respect noindex, nofollow and blocking content to being indexed and seen via .htacess or robots.txt file? I need to be sure of this 100%, I need to use duplicate content on some ...
4
votes
1answer
53 views

Can it be destructive to disallow all MediaWiki:Special pages in robots.txt?

I consider preventing indexation of all of MediaWiki 1.33.0's Special pages; (In Hebrew, "מיוחד" means "special"): Disallow: /מיוחד:* Disallow: /index.php?title=מיוחד:* Disallow: /index.php%3Ftitle%...
3
votes
2answers
359 views

Block third domain from being indexed

We have a web application that handles many websites that have test and production environment. This application has only one folder and it's hosted with IIS. We already have noindex nofollow, index ...
1
vote
0answers
24 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
1
vote
1answer
23 views

Effect of Submitted URLs being blocked by robots

I've got an alert on my site for pages being blocked by Robots, and am in the process of fixing it. But I'm curious what the ranking effect is, if there is one at all? And if there is an effect, does ...
1
vote
0answers
42 views

Refer to a MediaWiki website's domain in robots.txt by a variable

I have a website (built with MediaWiki 1.33.0 CMS) which contains a robots.txt file. In that file there is one line containing the literal domain of that site: Sitemap: https://example.com/sitemap/...
4
votes
1answer
94 views

Do the order of “Disallow” and “Sitemap” lines in robots.txt matter?

One can sort robots.txt this way: User-agent: DESIRED_INPUT Sitemap: https://example.com/sitemap-index.xml Disallow: / instead: User-agent: DESIRED_INPUT Disallow: / Sitemap: https://example.com/...
3
votes
1answer
78 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
2
votes
1answer
39 views

Override robots.txt exclusion to allow Mediapartners-Google for AdSense

I use robots.txt to block all robots.txt compliant bots from a directory like this: User-agent: * Disallow: /sub/* How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
1
vote
1answer
123 views

Is Applebot following Crawl-Delay?

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down. Does it follow Crawl-Delay from robots.txt, or not?
3
votes
1answer
104 views

After changing domain name Google's index says pages are blocked by robots.txt despite Allow: /

Two weeks ago I changed the address of my site. I forgot to renew my domain, and for a few days my site was unreachable. This resulted in google indexing my site improperly. There is a robot.txt error ...
1
vote
2answers
160 views

Allow Google to crawl the domain root (home page) while disallowing all other pages in robots.txt

I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
2
votes
1answer
3k views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
4
votes
2answers
3k views

Allow a folder and disallow all sub folders in robots.txt

I would like to allow folder /news/ and disallow all the sub folders under /news/ e.g. /news/abc/, /news/123/. How can I do that please? I think Disallow: /news/ will block everything in it, ...
1
vote
2answers
489 views

How to block subdomains under my main domain

I want all my subdomains to not be indexed under my domain, that means when I search in Google for example.com to not appear in index test.example.com, but when I search test.example.com to be indexed....
0
votes
1answer
53 views

Google doesn't send me to correct robots.txt directory

So I want to set a robots.txt in my site. Pretty straight forward place in root directory and good. But Google spits out a 404 error telling me the robots.txt is not in the place it expects. Not ...
1
vote
1answer
65 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...
0
votes
0answers
9 views

Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]

Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...
4
votes
2answers
1k views

Google Search Console says “submitted URL blocked by robots.txt” even though there is no robots.txt file

Today I got a email notice from Google that there were search problems with my site. I logged into Google Search Console. It shows that 1 file was being blocked by robots.txt, and has been for ...
6
votes
3answers
17k views

How to resolve Google “Indexed, though blocked by robots.txt”

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
1
vote
0answers
19 views

Should my sitemap and robots.txt have http or https urls for the given scenario [duplicate]

My website is running on node.js and runs in http on port 3000 I am using NGINX to enable https access so that all pages are served as https I have placed my sitemap.xml and robots.txt inside the ...
1
vote
1answer
45 views

Robots txt disallow on Blogger

I have a blog in Blogger, so the URL structure is always like this: www.example.com/2019/01/post.html The problem is I don't want google to crawl www.example.com/2019 (it shows all posts of 2019) ...

1 2 3 4 5 13