Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

Block crawling of URLs with "tag" using robots.txt

I want to block robots from crawling URLs that contain the word "tag" on my website. Here are some example URLs: https://www.example.com/six-senses-fort-barwara-a-glimpse-into-the-...
1
vote
1answer
18 views

Question regarding Sitemap value in a robots.txt file, and the lines of a txt sitemap

I have two related questions. To start, in a robots.txt file can you just put the following for sitemap: Sitemap: www.example.com Or do you have to have: Sitemap: http://www.example.com or Sitemap: ...
2
votes
1answer
50 views

How to block all .html pages in robot.txt?

I have a new website that has a different product URL from the old website. URL Old Website https://www.example.com/product/MA-593838554.html https://www.example.com/product/MR-559927541.html etc ...
4
votes
1answer
884 views

Do I still need to ping Google after changing a sitemap nowadays?

Months ago I've implemented SEO in a website and it was recommended that every time that the sitemaps changed I should ping this URL http://www.google.com/ping?sitemap=http://mywebsite.com/sitemap.xml....
3
votes
0answers
37 views

Should I block the '_next' directory in a site using Next.js with robots.txt?

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx ...
3
votes
0answers
17 views

Prevent crawling of mobile site will impact the actual's site ranking or indexing?

We have a responsive mobile-first indexing site in place. Unfortunately, the m.example.com - mobile version is also there and it ranks for certain keywords. We don't want to have such a separate m....
2
votes
1answer
61 views

Avoid unnecessary crawling of ad landing URLs by disallowing query string with robots.txt

Will the following configuration allow me to run an Ads campaign for products with the ?string parameter in the URL? User-agent: * Disallow: /*?string User-agent: AdsBot-Google Allow: / User-agent: ...
3
votes
1answer
162 views

Is there a way to have Google index a page but not cache it?

I see that many popular sites have pages indexed in Google, but when I try to access Google's cache the page isn't there. How can I go about doing the same thing with my site? I read that you have to ...
5
votes
1answer
587 views

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

It seems common wisdom that using X-Robots-Tag/meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.: Using both X-Robots-Tag and meta robots on a URL is ...
1
vote
1answer
30 views

If I disable indexing from the CDN server robots.txt file will this create an SEO issue?

I have 1 own website and 1 CDN server. I have CSS, JS and images on my CDN server. All the CSS, JS and images on my own website come from the CDN server. If I block my CDN server with a robots.txt ...
2
votes
2answers
173 views

What generates the preview data for Google search engine?

Well when I search our website through Google site:allsports.app. I notice first that it correctly limited itself to what is (was) defined in the sitemap. I also notice that for most pages it does not ...
1
vote
1answer
19 views

Robots.txt not blocking yet GSC URL Inspection Tool says it is

Recently I realized Disallow /was set on my staging sites. I didn't realize until I got a warning in Search Console... Mistakes happen I've removed the directive and cleared all caches at least 10 ...
3
votes
1answer
144 views

Ahrefs reached max crawl depth, could it be caused by robots.txt changes?

Three weeks ago I add this in my robots.txt to disallow pagination so it will not appear in the Google index: User-Agent: * Disallow: /blog/1 Disallow: /blog/2 Disallow: /blog/3 Disallow: /blog/4 ...
4
votes
1answer
139 views

Do SEO analysis bots have any advantage in terms of website traffic?

I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as AhrefsBot/7.0; +http://ahrefs.com/robot/ SemrushBot/...
1
vote
1answer
29 views

Should I add links to the Sitemap that are disallowed in robots.txt?

My robots.txt contains the line Disallow: /search.php. Shall I nevertheless include http://example.com/search.php into my sitemap?
2
votes
1answer
76 views

What is a robots.txt file and how can I use it

I want to use a robots.txt file for my website, which is currently displaying a 404 error when requesting it (e.g., https://www.example.com/robots.txt). How do I make one? There's no option to create ...
1
vote
1answer
25 views

What is the affect of changing robots.txt file access permission?

I want to change access permission of robots.txt file to 403. Just my static ip can access directly not any one. This position is true? why some people do that? if it is true, What is the affect of ...
2
votes
1answer
25 views

How do rule length and use of wildcards determine whether allow or disallow takes precedence in Robots.txt?

I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt Could someone elaborate on the difference between these two: http://example.com/page.htm allow: /page ...
1
vote
1answer
86 views

Do I need to allow bots on /* if I disallow /specific-directory?

If I want all bots to not crawl /specific-directory, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate? # All Bots User-...
1
vote
1answer
75 views

Is it true to generalize that the second or later forward slash always acts like a wildcard in robots.txt files?

Is it true to generalize that (in 2021) the second or later forward slash acts like a wildcard global (asterisk) in robots.txt files? In another approach, it true to say that /MY_NAMESPACE:/ could ...
0
votes
0answers
34 views

Block some subdomains using robots.txt -Angular website with IIS [duplicate]

We have a testing environment with the surprising sub-domain name tests and I would like to block it to search engines. How can I let Google scan my site https://example.com and block https://tests....
2
votes
2answers
61 views

Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

I've written my own SiteSearch Script in PHP. The SiteSearch parses two GET parameters from the queryString: search // the search-phrase filters (optional) // indicating which parts of the site to ...
0
votes
0answers
25 views

Google Sitemap XML isn't showing About Page, Contact Page, Etc, only

My google sitemap has been uploaded to Google Search Console using example.com/sitemap_index.xml When I click to the page in google search console 'sitemaps read:' I can only see the following: ...
0
votes
0answers
17 views

Should API urls be disallowed in robots.text on SPAs

I have a single page app that I want google to index. However, I can see from the search console that Google also crawls /api/* urls. Search console does not tell wether google bot calls them directly ...
1
vote
0answers
40 views

Disallow dynamic URL paths ending in a common suffix using robots.txt

I want to block Google from crawling the following URLs: https://www.example.com/tonga/posts https://www.example.com/india/posts https://www.example.com/usa/posts https://www.example.com/uae/posts ...
2
votes
0answers
39 views

Is “cbsearch.site” a web robot, and if yes, how do I block it?

I saw invalid traffic in my Google AdSense account. I read some articles from the web, which all made it clear that invalid traffic could be caused by web "robots". So I tried to read on how ...
0
votes
1answer
603 views

Website not properly indexed by Google search console says "'noindex' detected in 'robots' meta tag" while this is not the case

TLTR I have a website (SPA) where SEO is very important. It was very poorly indexed and search console tells me it is due to 'noindex' in the robots meta, while I cannot see how Google comes to that ...
2
votes
1answer
26 views

Disallowing path with parameter

I found a loads of examples where in robots.txt were disallowed urls with similar approaches: Disallow: /*?dir=*&order=*&p=* Where every path with specified parameters will be blocked. What if ...
2
votes
1answer
86 views

Do I need to submit sitemap.xml to google if I already have it in robots.txt and hosted on S3?

I have a static site hosted on S3 bucket which will have a robots.txt at root directory level. This robots.txt file will have reference to my sitemap.xml file which is hosted on another S3 bucket. It ...
3
votes
1answer
47 views

Do I get back all the search engine rankings I had after having blocked a domain using robots.txt?

I blocked Google from crawling my domain using a robots.txt rule User-Agent: * Disallow: / by mistake. Just recently I noticed, two months later, and I removed the robots.txt file. The website was ...
2
votes
1answer
44 views

Is Allow: in robots txt an exclusion to an exclusion?

Say I have a MediaWiki website and all the Special: namespace webpages are excluded with robots.txt Disallow: Special: but there are a few specific Special: webpages that I do want to include; Allow:...
1
vote
1answer
32 views

Block short URL in robots.txt

We have example.com/article-page-one/ This page has a short URL in Wordpress like this : example.com/?p=1 In this page we put short URL for users. What will happen if we block short URL? The robots....
1
vote
1answer
183 views

Cache-Control for ads.txt, robots.txt and sitemap.xml files that are generated dynamically

I have some "static files" that I build on the fly, because they are generated dynamically. ads.txt robots.txt sitemap.xml What is the ideal Cache-Control for those files? PS: My server is ...
0
votes
0answers
12 views

what is the difference between "allow: /" and "disallow:" in robots.txt? [duplicate]

I know how robots.txt works but I am confused at one point, the point is what is the difference between these two, I know that they have the same function but which one is more valid than the other, ...
0
votes
0answers
15 views

Remove Google suggestions and links from search results [duplicate]

How to remove / delete google suggestions and related keywords links. As some results showing my personal data into some sites , need to remove those pages or links from search results showed by ...
2
votes
0answers
29 views

Could allowing our images to be indexed hurt our web search rankings?

We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
2
votes
0answers
112 views

Restrict Google from indexing subdomain from Azure CDN services

How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
3
votes
1answer
71 views

What will happen if I have two htaccess/robots.txt files in two directories?

I think there is (or I did something wrong) on my site: I installed a pre-made WordPress theme and normally they are installed as 'quick-install-package' with duplicator. When I installed the package, ...
0
votes
0answers
18 views

how to specify all values for a parameter in robots.txt and sitemap.xml?

I have a php file that extracts a post id from a url's parameter to show a specific post content. for example it uses the postid parameter from the following url to show the post content with id of 21:...
2
votes
1answer
37 views

Block URLs that begin and end in common prefixes and suffixes using robots.txt

I have the urls like: /category1/type-name-more And I need to disallow all urls of this type. Disallow: /*-more$ is not OK as it may block all the urls ending in -more. So is the variant Disallow: /...
6
votes
4answers
3k views

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

In my web application I generate links in the following format: https://example.com/^token^ Each link according to my specs is sent via email and should not allow robots to scrape it. In order to ...
2
votes
1answer
25 views

In robots.txt, should I use a wildcard at the end of a disallow directive?

I want to disallow a specific folder and all of its files and subdirectories but I don't know the difference between Disallow: /somedir/ and Disallow: /somedir/*. which one of these lines should I use?...
0
votes
1answer
43 views

Is it possible to tell web crawlers to agree to a site's TOS before indexing?

Is there a way with robots.txt or page headers to inform the managers of web crawlers they must agree to a terms of service prior to indexing any crawled pages?
1
vote
1answer
35 views

Order of items in a very basic robots.txt changing the command scope?

I had a very simple robots.txt file setup for a site I maintain. After a spike of traffic that the ISP put down to crawlers they suggested I add a crawl delay directive which is fair enough. So I ...
3
votes
1answer
127 views

Link rel="nofollow" vs robots.txt disallow for "/search" URLs?

There's a /search path on my web app that I don't want robots to crawl or index. What would be the best approach for this scenario? 1: Block them from accessing by adding a Disallow for /search on ...
1
vote
1answer
46 views

Robots.txt disallow url only if it has parameters?

Take for example /about?param1=blah /contact?param1=bleh&param2=bluh But allow /about /contact Is that possible? Would disallowing /about?* work?
4
votes
1answer
313 views

How do I "noindex" a text (.txt) file?

I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as ...
0
votes
1answer
23 views

Run beta versions of sites SEO safely [duplicate]

We have domain.com and want to test a new site template on beta.domain.com. What is the correct way to run a beta test without affecting SEO of the main domain.com? In the topic SEO safe way to run ...
0
votes
0answers
55 views

Google search says No information is available for my GitHub page

Today I search my GitHub page in Google Chrome and Firefox (Through Google search) and for the first time, it says no information is available (it says 2019- GitHub under that it says no information ...
1
vote
1answer
50 views

Google Search Console coverage reports "Submitted URL marked 'noindex'" for a 404 page not in the sitemap without a noindex tag

Error: "Submitted URL marked 'noindex'" error in the Google Search Console coverage report Situation URL is not submitted in our sitemap Google is not indexing the URL There is no noindex ...

1
2 3 4 5
15