Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
2
votes
1answer
41 views

What is difference between robots.txt, sitemap, robots meta tag, robots header tag?

So I am trying to learn SEO and I am honestly confused and have following 8 questions. Do I tell a bot not to visit a certain link through X-Robots-Tag or through robot meta tag or robots.txt? Is it ...
2
votes
2answers
137 views

Disallow root but not 4 subdirectories for robots.txt

I have a project and I would like to disallow everything starting with root. From what I understand I think I can do so by doing this Disallow: / Disallow: /* However I would like to allow 4 ...
0
votes
1answer
28 views

Can a subdirectory override the parent domain's robots.txt?

Context: sith.edu is mapping the subdirectory sith.edu/rogueone via reverse proxy (/rogueone is using a different CMS and is hosted elsewhere). My question: If sith.edu adds disallow /rogueone to ...
-1
votes
1answer
13 views

Why is Azure pinging my site?

I have noticed more and more, traffic from IPs assigned to Microsoft Azure/Microsoft Corporation hitting my WordPress site many times a day. It's direct traffic to just the home and privacy policy ...
1
vote
2answers
46 views

Why are Bing and SEMRushBot ignoring crawl-delay in my robots.txt file?

We host a large number of sites that have a large number of catalog pages. We would like to slow down several bots because the traffic is quite excessive from those bots. Specifically, we get quite a ...
0
votes
0answers
35 views

robots.txt query string disallow except for one case

I can't find solution for my task. For example, I have 2 pages: / and /some-category/. Both of them have GET parameters: page and several filters. I need the result: /, /?page=x, /some-category/ to be ...
0
votes
0answers
17 views

Is this Robots.txt ok for blogger/blogspot?

User-agent: * Allow: /ads.txt Disallow: /ads Disallow: /search User-agent: Mediapartners-Google Disallow: Allow: / Sitemap: https://example.com/atom.xml?redirect=false&start-index=1&max-...
1
vote
0answers
17 views

Disallowing a handler in robots.txt while adding its dynamic URLs to the XML sitemap

I'm using ASP.NET webforms, and I have a page lets call it Subjects.aspx, I don't want crawlers to crawl that page but I want them to crawl the dynamic URLs that are powered by it. For example /...
1
vote
1answer
38 views

Google is indexing URLs with parameters that are disallowed in robots.txt despite canonical URLs without parameters listed in the sitemap

All of my webpages are showing ?mode=grid & ?mode=list in the google coverage. But submitted sitemap shows normal URLs. For example: example.com/page/?mode=grid example.com/page/?mode=list ...
0
votes
1answer
38 views

Googlebot blocked by robots.txt

I have been checking the errors of my website in the google console , and I fount it face an error Googlebot blocked by robots.txt and it shows a link https://static.doubleclick.net/instream/...
2
votes
1answer
30 views

Robots.txt for a multilanguage site where root is redirected

I have a site which offers two languages, English and Spanish. When the user navigates to the home page, let's say www.example.com the page redirects you to either /es if your browser language is ...
0
votes
0answers
36 views

Google Search console reports home page blocked by robots.txt and search results only show the domain name even though it isn't blocked

For the last few days I haven't seen the title and meta description in Google search results. When I inspected URL in search console, I got this error message: URL is on Google, but has issues. ...
1
vote
1answer
43 views

Google image crawler won't respect my robots.txt entry to not crawl images

I was looking for a way to prevent reverse image searching (namely I didn't want for people who had a copy of one of my images to upload it to google and discover where it originated from). I created ...
1
vote
0answers
21 views

Google say URL is blocked by robots.txt but it isn't

My Google search console says that my submitted URL is blocked by robots.txt, but I checked header, tags and robots.txt it isn't. My URL was on #1 for my target keyword and daily get 1000 clicks ...
4
votes
1answer
156 views

Disallow directory at root only, and not elsewhere using robots.txt

I need to exclude anything that contains /page/ but only at root: Disallowed: example.com/page/1 Allowed: example.com/category/page/1 How to safely write this in robots.txt?
1
vote
1answer
19 views

Disavow tool property drop down menu has nothing in it even a month after verifying my site in Google Search Console

About a month ago, I launched a site and after launching, I installed the search console on it. After a month, the disavow menu has not been activated yet (as you can see in picture) and this is my ...
2
votes
1answer
27 views

favicon.ico 500 error when fetching robots.txt

I have an error 500 on my example.com/robots.txt page. Could this cause any SEO / crawling issues? It's just the favicon , but you never know.
0
votes
0answers
16 views

Permanently redirecting (`301`) robots.txt from root (`/`) to static files directory (`/static/file/robots.txt`)

After reading up on this Google Webmasters article on creating a robots.txt file, I noticed that they mentioned that the robots.txt file should always be at the root directory (/). However, I've been ...
2
votes
1answer
206 views

Ahrefs reports “Fetching robots.txt took too long”

My robots: User-agent: * Disallow: /wp-admin Disallow: /sistema Disallow: /site Disallow: /old Sitemap: http://www.example.com.br/page-sitemap.xml In the google test tool: https://www.google.com/...
1
vote
0answers
18 views

Can I use a wildcard rule to prevent Yandex from indexing all tag and date URLs created by Joomla K2?

I have a site based on joomla and K2 component. I use K2 Tags and Calendar, but i have problem with pages indexing When yandex.ru begin to index it i find out that it index my tags and SEO test sites ...
3
votes
1answer
35 views

Can robots.txt redirect to a different domain?

I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer. example: https://www.mydomain.example/...
2
votes
1answer
57 views

Google Search Console Warning: “ Indexed, though blocked by robots.txt ” (BUG)

Two days ago I've got an email with a warning from Google Search Console, telling me that 3 URLs from my site where indexed, but were blocked by robots.txt. Since I first deployed my website, I haven'...
2
votes
2answers
179 views

What does crawl-delay: 1 mean if it is interpreted as is?

I have a problem of understanding what crawl-delay: 1 means if it isn't ignored and actually being interpreted as is by a given web crawler. I understood from this answer by Stephen Ostermiller that: ...
0
votes
0answers
9 views

Google does not show all our posts in search results [duplicate]

We have issue with google search console for web site> https://bit.ly/38Xhm97 Our site is written in 2 "languages", Latin and Cyrillic. Sometimes we have problem that our cyrillic posts das not show ...
2
votes
1answer
163 views

How to prevent google index for a specific subpath?

I have my website, say www.example.com, and I want it to be indexed. Anyway I also have one section, www.example.com/section that I don't want to be indexed. I tried adding the following headers to ...
1
vote
2answers
124 views

crawl-delay: x ------— has someone here managed to understand how the math works there?

From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than ...
1
vote
2answers
39 views

Restrict crawling of region/lang combinations other than the provided ones in robots.txt

I want to allow crawling of my website only if the URL starts with accepted region/language combinations which are us/en, gb/en and fr/fr. Other language combinations must be restricted. On the other ...
4
votes
1answer
84 views

Googlebot ignores robots.txt

Two months ago i disallowed some directories from crawling with robots.txt. Since then i monitor logfiles everyday and realize, that Googlebot absolutely ignores robots.txt. Really, it crawls every ...
1
vote
0answers
48 views

Google Search Console reports Soft 404 on an Excel file

since mid of this month, an Excel file that used to be indexed in Google Search is reported as a Soft 404 on Search Console File: https://neoschronos.com/assets/lean-canvas.xls We have multiple ...
0
votes
0answers
17 views

How to force deindex URLs from Google index ASAP? [duplicate]

Remove pages from Google index Hello, I have bought previously owned domain. Right now it have 150 - 200 URLs indexed by Google that are not related to my current website (powered by wordpress). ...
1
vote
2answers
39 views

Disabling main page url from robots.txt

Example robots.txt: User-agent: * Disallow: / Would it work? I have urls, like https://example.com/en, https://example.com/de I don't want search engines to index https://example.com without ...
1
vote
1answer
34 views

Hide /path from search engines, except for /path/that-i-want-indexed

In our robots.txt we have something like User-agent: * Disallow: /v6.0 Disallow: /v5.8 Disallow: /v5.7 Disallow: /v5.5 Disallow: /v5.1 Disallow: /v5.0 Which properly prevents search engines from ...
2
votes
1answer
72 views

Launching a new website on a subdomain but keeping exsisting website running

We're about to launch a new Drupal 8 website for a client, as not all areas of the current website will be on the new platform, and due to apparent legacy complications, the decision has been made to ...
3
votes
1answer
93 views

Why are bots repeatedly crawling pages of my site that I never created?

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
2
votes
0answers
110 views

In robots.txt - what are the directives from WBCPBlocker

On this site's robots.txt there is a comment preceding many rows of directives blocking SEO tool bots and was wondering if someone could decipher it for me. There's the standard User-agent: * Allow:...
3
votes
1answer
386 views

Prevent Google from indexing images

I have a site where all my images are stored in a specific folder, for example /images/984728176386375837629856e.png but I don't want this page to appear on the SERP. Can I disable the indexing of the ...
3
votes
1answer
29 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
2
votes
1answer
88 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
1
vote
2answers
165 views

Google Search Console and Sitemap set-up for a website subfolder

I'm managing a website https://www.example.com/ and I've just launched a new website https://www.example.com/talkabout (so with a subfolder). I've already set up a GSC account, sitemaps (html and ...
3
votes
1answer
26 views

Can robots.txt be used to prevent bots from seeing lazily loaded content?

Let's say that googlebot is scraping https://example.com/page. example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page /page lazy loads content using /...
3
votes
1answer
83 views

Google's URL inspection says my image URL is blocked by robots.txt — I don't even have one!

I just discovered our image system domain is not being crawled by Google for a long time. The reason is that all the URLs seem to be blocked by robots.txt -- but I don't even have one. Disclaimer: ...
1
vote
0answers
21 views

Why is Google indexing our robots.txt file and showing it in search results?

For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below. Our robots.txt file is not linked from anywhere on the site ...
1
vote
2answers
153 views

Does Google TRULY respect noindex and nofollow?

Does google REALLY respect noindex, nofollow and blocking content to being indexed and seen via .htacess or robots.txt file? I need to be sure of this 100%, I need to use duplicate content on some ...
1
vote
1answer
23 views

Effect of Submitted URLs being blocked by robots

I've got an alert on my site for pages being blocked by Robots, and am in the process of fixing it. But I'm curious what the ranking effect is, if there is one at all? And if there is an effect, does ...
1
vote
0answers
42 views

Refer to a MediaWiki website's domain in robots.txt by a variable

I have a website (built with MediaWiki 1.33.0 CMS) which contains a robots.txt file. In that file there is one line containing the literal domain of that site: Sitemap: https://example.com/sitemap/...
4
votes
1answer
134 views

Do the order of “Disallow” and “Sitemap” lines in robots.txt matter?

One can sort robots.txt this way: User-agent: DESIRED_INPUT Sitemap: https://example.com/sitemap-index.xml Disallow: / instead: User-agent: DESIRED_INPUT Disallow: / Sitemap: https://example.com/...
2
votes
1answer
49 views

Override robots.txt exclusion to allow Mediapartners-Google for AdSense

I use robots.txt to block all robots.txt compliant bots from a directory like this: User-agent: * Disallow: /sub/* How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
4
votes
1answer
55 views

Can it be destructive to disallow all MediaWiki:Special pages in robots.txt?

I consider preventing indexation of all of MediaWiki 1.33.0's Special pages; (In Hebrew, "מיוחד" means "special"): Disallow: /מיוחד:* Disallow: /index.php?title=מיוחד:* Disallow: /index.php%3Ftitle%...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
1
vote
1answer
71 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...

1
2 3 4 5
14