Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
1 vote
1 answer
19 views

Getting Robots.txt Not Found 404 Error on Google Search Console

I have created robots.txt in my root directory and when I navigate to my website address like https://example.ca/robots.txt I am able to see the rules User-agent:* Disallow: /captcha/ Disallow: /...
user avatar
  • 115
1 vote
0 answers
11 views

Googlebot is crawling unpublished pages from Adobe AEM and getting 404 errors

My company is currently using Adobe AEM as a CMS for our company website. We are currently having issues with 4xx errors appearing specifically 404 as unpublished pages are still being crawled. What ...
user avatar
  • 11
2 votes
2 answers
24 views

Disallow top level of directory, but not subdirectories in robots.txt

I have a directory that I don't want Google to index at the top level. I currently do the following: Disallow: /profiles/ This stops Google from indexing https://example.com/profiles/, but it also ...
user avatar
3 votes
1 answer
117 views

Redirect Errors in Google Search Console when trying to index site

I've built a small site for a local trader which is not a complicated site in any way. In fact it is 100% static. I have a few years experience and in that time have never encountered an error with ...
user avatar
1 vote
1 answer
22 views

Only allow crawling of pages in sitemap

My idea is, to use robots.txt to disallow everyting and put all allowed pages into the sitemaps. User-agent: * Disallow: / Sitemap: https://www.example.com/sitemap.xml Is this working as expected?
user avatar
  • 11
3 votes
3 answers
42 views

Why would a human read robots.txt

As a part of our new website I've written code in a page filter to do some basic bot / crawler detection by looking at the user agent. That way when a bot is detected we serve up some generic user ...
user avatar
0 votes
1 answer
43 views

My website's image does not work with twitter cards (unclear issues with robots.txt)

Problem I am trying to properly set up metadata for a twitter summary card on my website. I have everything in place, but the card image is not showing. I use Grav, but I guess it does not matter. ...
user avatar
  • 188
3 votes
1 answer
73 views

How to Allow bots only on main page

I'm trying to create a robots.txt file that allows bots to access ONLY the home page and no other page. Below is the content of the robots.txt file based on the research I have done. Is this correct? ...
user avatar
8 votes
1 answer
698 views

Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)

Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says the /fish rule matches any path that starts with /fish (...) it doesn'...
user avatar
  • 183
4 votes
1 answer
67 views

Is there a way to allow ad bots to crawl a website that has "noindex,nofollow" on it?

Due to duplicated content, there is a "noindex,nofollow" on the website, but in order to run ppc ads, there needs to be a way for the ads bots to crawl the website Can this be done without ...
user avatar
2 votes
2 answers
42 views

Should I exclude the assets folder from search engine crawlers in angular

I'm working with angular and had added the assets folder as element to exclude from the search engine crawlers, into my robots.txt file. But I now get the following error while testing the site health ...
user avatar
0 votes
0 answers
9 views

Sitemap in robots.txt is not appearing in Google Search Console, no URLs detected in sitemap [duplicate]

At the bottom of robots.txt I have Sitemap: https://domain2.com/sitemap.xml. In the Google Search Console for my property at domain1 I see a false HTML sitemap with errors, no trace of my actual ...
user avatar
  • 141
1 vote
0 answers
22 views

Disallow crawling of all links to a domain, except only specific URLs with parameters

I have a domain like example.com under which everything is blocked and links to an SPA. Like example.com, and we've disabled everything like so: User-agent: * Disallow: / Now this SPA uses parameters ...
user avatar
1 vote
1 answer
19 views

Block crawling of URLs with "tag" using robots.txt

I want to block robots from crawling URLs that contain the word "tag" on my website. Here are some example URLs: https://www.example.com/six-senses-fort-barwara-a-glimpse-into-the-...
user avatar
1 vote
1 answer
20 views

Question regarding Sitemap value in a robots.txt file, and the lines of a txt sitemap

I have two related questions. To start, in a robots.txt file can you just put the following for sitemap: Sitemap: www.example.com Or do you have to have: Sitemap: http://www.example.com or Sitemap: ...
user avatar
  • 11
2 votes
1 answer
60 views

How to block all .html pages in robot.txt?

I have a new website that has a different product URL from the old website. URL Old Website https://www.example.com/product/MA-593838554.html https://www.example.com/product/MR-559927541.html etc ...
user avatar
  • 43
4 votes
1 answer
908 views

Do I still need to ping Google after changing a sitemap nowadays?

Months ago I've implemented SEO in a website and it was recommended that every time that the sitemaps changed I should ping this URL http://www.google.com/ping?sitemap=http://mywebsite.com/sitemap.xml....
user avatar
4 votes
1 answer
156 views

Should I block the '_next' directory in a site using Next.js with robots.txt?

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx ...
user avatar
  • 141
3 votes
0 answers
17 views

Prevent crawling of mobile site will impact the actual's site ranking or indexing?

We have a responsive mobile-first indexing site in place. Unfortunately, the m.example.com - mobile version is also there and it ranks for certain keywords. We don't want to have such a separate m....
user avatar
  • 31
2 votes
1 answer
71 views

Avoid unnecessary crawling of ad landing URLs by disallowing query string with robots.txt

Will the following configuration allow me to run an Ads campaign for products with the ?string parameter in the URL? User-agent: * Disallow: /*?string User-agent: AdsBot-Google Allow: / User-agent: ...
user avatar
3 votes
1 answer
169 views

Is there a way to have Google index a page but not cache it?

I see that many popular sites have pages indexed in Google, but when I try to access Google's cache the page isn't there. How can I go about doing the same thing with my site? I read that you have to ...
user avatar
5 votes
1 answer
609 views

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

It seems common wisdom that using X-Robots-Tag/meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.: Using both X-Robots-Tag and meta robots on a URL is ...
user avatar
  • 153
1 vote
1 answer
89 views

If I disable indexing from the CDN server robots.txt file will this create an SEO issue?

I have 1 own website and 1 CDN server. I have CSS, JS and images on my CDN server. All the CSS, JS and images on my own website come from the CDN server. If I block my CDN server with a robots.txt ...
user avatar
  • 654
2 votes
2 answers
186 views

What generates the preview data for Google search engine?

Well when I search our website through Google site:allsports.app. I notice first that it correctly limited itself to what is (was) defined in the sitemap. I also notice that for most pages it does not ...
user avatar
  • 131
1 vote
1 answer
21 views

Robots.txt not blocking yet GSC URL Inspection Tool says it is

Recently I realized Disallow /was set on my staging sites. I didn't realize until I got a warning in Search Console... Mistakes happen I've removed the directive and cleared all caches at least 10 ...
user avatar
  • 4,490
4 votes
1 answer
303 views

Ahrefs reached max crawl depth, could it be caused by robots.txt changes?

Three weeks ago I add this in my robots.txt to disallow pagination so it will not appear in the Google index: User-Agent: * Disallow: /blog/1 Disallow: /blog/2 Disallow: /blog/3 Disallow: /blog/4 ...
user avatar
4 votes
1 answer
184 views

Do SEO analysis bots have any advantage in terms of website traffic?

I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as AhrefsBot/7.0; +http://ahrefs.com/robot/ SemrushBot/...
user avatar
  • 442
1 vote
1 answer
55 views

Should I add links to the Sitemap that are disallowed in robots.txt?

My robots.txt contains the line Disallow: /search.php. Shall I nevertheless include http://example.com/search.php into my sitemap?
user avatar
  • 113
2 votes
1 answer
89 views

What is a robots.txt file and how can I use it

I want to use a robots.txt file for my website, which is currently displaying a 404 error when requesting it (e.g., https://www.example.com/robots.txt). How do I make one? There's no option to create ...
user avatar
1 vote
1 answer
27 views

What is the affect of changing robots.txt file access permission?

I want to change access permission of robots.txt file to 403. Just my static ip can access directly not any one. This position is true? why some people do that? if it is true, What is the affect of ...
user avatar
  • 11
2 votes
1 answer
25 views

How do rule length and use of wildcards determine whether allow or disallow takes precedence in Robots.txt?

I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt Could someone elaborate on the difference between these two: http://example.com/page.htm allow: /page ...
user avatar
  • 41
1 vote
1 answer
86 views

Do I need to allow bots on /* if I disallow /specific-directory?

If I want all bots to not crawl /specific-directory, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate? # All Bots User-...
user avatar
1 vote
1 answer
89 views

Is it true to generalize that the second or later forward slash always acts like a wildcard in robots.txt files?

Is it true to generalize that (in 2021) the second or later forward slash acts like a wildcard global (asterisk) in robots.txt files? In another approach, it true to say that /MY_NAMESPACE:/ could ...
user avatar
0 votes
0 answers
70 views

Block some subdomains using robots.txt -Angular website with IIS [duplicate]

We have a testing environment with the surprising sub-domain name tests and I would like to block it to search engines. How can I let Google scan my site https://example.com and block https://tests....
user avatar
  • 101
2 votes
2 answers
61 views

Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

I've written my own SiteSearch Script in PHP. The SiteSearch parses two GET parameters from the queryString: search // the search-phrase filters (optional) // indicating which parts of the site to ...
user avatar
0 votes
0 answers
28 views

Google Sitemap XML isn't showing About Page, Contact Page, Etc, only

My google sitemap has been uploaded to Google Search Console using example.com/sitemap_index.xml When I click to the page in google search console 'sitemaps read:' I can only see the following: ...
user avatar
  • 1
2 votes
0 answers
42 views

Is “cbsearch.site” a web robot, and if yes, how do I block it?

I saw invalid traffic in my Google AdSense account. I read some articles from the web, which all made it clear that invalid traffic could be caused by web "robots". So I tried to read on how ...
user avatar
0 votes
1 answer
730 views

Website not properly indexed by Google search console says "'noindex' detected in 'robots' meta tag" while this is not the case

TLTR I have a website (SPA) where SEO is very important. It was very poorly indexed and search console tells me it is due to 'noindex' in the robots meta, while I cannot see how Google comes to that ...
user avatar
  • 1
2 votes
1 answer
27 views

Disallowing path with parameter

I found a loads of examples where in robots.txt were disallowed urls with similar approaches: Disallow: /*?dir=*&order=*&p=* Where every path with specified parameters will be blocked. What if ...
user avatar
  • 207
2 votes
1 answer
114 views

Do I need to submit sitemap.xml to google if I already have it in robots.txt and hosted on S3?

I have a static site hosted on S3 bucket which will have a robots.txt at root directory level. This robots.txt file will have reference to my sitemap.xml file which is hosted on another S3 bucket. It ...
user avatar
  • 23
3 votes
1 answer
48 views

Do I get back all the search engine rankings I had after having blocked a domain using robots.txt?

I blocked Google from crawling my domain using a robots.txt rule User-Agent: * Disallow: / by mistake. Just recently I noticed, two months later, and I removed the robots.txt file. The website was ...
user avatar
  • 31
2 votes
1 answer
47 views

Is Allow: in robots txt an exclusion to an exclusion?

Say I have a MediaWiki website and all the Special: namespace webpages are excluded with robots.txt Disallow: Special: but there are a few specific Special: webpages that I do want to include; Allow:...
user avatar
  • 27
1 vote
1 answer
34 views

Block short URL in robots.txt

We have example.com/article-page-one/ This page has a short URL in Wordpress like this : example.com/?p=1 In this page we put short URL for users. What will happen if we block short URL? The robots....
user avatar
1 vote
1 answer
301 views

Cache-Control for ads.txt, robots.txt and sitemap.xml files that are generated dynamically

I have some "static files" that I build on the fly, because they are generated dynamically. ads.txt robots.txt sitemap.xml What is the ideal Cache-Control for those files? PS: My server is ...
user avatar
2 votes
0 answers
29 views

Could allowing our images to be indexed hurt our web search rankings?

We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
user avatar
  • 161
2 votes
0 answers
146 views

Restrict Google from indexing subdomain from Azure CDN services

How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
user avatar
3 votes
1 answer
78 views

What will happen if I have two htaccess/robots.txt files in two directories?

I think there is (or I did something wrong) on my site: I installed a pre-made WordPress theme and normally they are installed as 'quick-install-package' with duplicator. When I installed the package, ...
user avatar
  • 61
0 votes
0 answers
23 views

how to specify all values for a parameter in robots.txt and sitemap.xml?

I have a php file that extracts a post id from a url's parameter to show a specific post content. for example it uses the postid parameter from the following url to show the post content with id of 21:...
user avatar
2 votes
1 answer
67 views

Block URLs that begin and end in common prefixes and suffixes using robots.txt

I have the urls like: /category1/type-name-more And I need to disallow all urls of this type. Disallow: /*-more$ is not OK as it may block all the urls ending in -more. So is the variant Disallow: /...
user avatar
  • 21
6 votes
4 answers
3k views

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

In my web application I generate links in the following format: https://example.com/^token^ Each link according to my specs is sent via email and should not allow robots to scrape it. In order to ...
user avatar

1
2 3 4 5
15