Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

How to disallow URLs containing fragment identifier or a hash symbol?

Can we disallow URLs contain #!/xxxx in robots.txt? For Example: /peach-polyester#!/inventory/linen-rentals/
user2475988's user avatar
1 vote
0 answers
45 views

What is the meaning of `Disallow: /? in a robots.txt file?

What is the meaning of `Disallow: /? in a robots.txt file? User-agent: * Disallow: /?*
Larry's user avatar
  • 11
1 vote
2 answers
166 views

Why my robots.txt is not honored by Google?

This is my robots.txt. User-agent: * Disallow: /property/ Sitemap: https://example.com/sitemap.xml As you can see, I have disallowed the /property/ path. Yet I see some of the links on my website ...
Saeed Neamati's user avatar
2 votes
1 answer
186 views

Shoud a sitemap include favicon.ico and robots.txt?

Should a sitemap include favicon.ico and robots.txt? They are in mine (and do exist) and I'm worried removing them will cause some problem.
ysth's user avatar
  • 121
1 vote
1 answer
26 views

What is the Query to add in robots.txt for blocking URLs has some special characters?

I was going through the Google Search Console pages issues and one of the issue reason is "Alternate page with proper canonical tag" and this has some URLs which look like below example.com/...
Rahul Chalana's user avatar
1 vote
1 answer
17 views

How to properly block indexing my admin subdomain?

I have a website hosted on https://admin.example.com. However, the public website displayed is on this sub-directory : https://www.example.com/website/ For technical reasons, we cannot install it or ...
B4b4j1's user avatar
  • 131
0 votes
1 answer
37 views

How to properly set up google search indexing for site which requires authentication

I was not sure of the best way to describe this in the title but my scenario is the following: Every page on our site requires authentication and will redirect unauthenticated users to a login page ...
Mason's user avatar
  • 101
0 votes
1 answer
18 views

I have added this in my Bloggerspot blog's robot txt file, is it in the correct format?

User-agent:* Disallow: Allow:/ Sitemap:https://example.com/?m=1
Anum ishtiaq's user avatar
0 votes
0 answers
40 views

Why is my WordPress website not getting indexed on Google Search Console despite using Yoast SEO and manually creating a sitemap and robots.txt?

I have added my website in my Google Search Console but it's not getting indexed. It's a WordPress website and I'm using Yoast SEO but since it didn't work out. I had to create Sitemap manually and ...
pkguy's user avatar
  • 1
3 votes
2 answers
392 views

Is it possible to exclude just OpenAI - ChatGPT from scraping my website?

I would like to not have OpenAI, ChatGPT or the potentially associated Bingbot scrape my website. The latter seems to have a defined bot name. What about the first two? They seem to be using Common ...
Ivan Balepin's user avatar
1 vote
1 answer
18 views

What do backslashes in WP generated robots.txt rules mean?

I am trying to implement a library to handle robots.txt and I am a bit confused about the meaning of some rules that I believe are generated by WordPress: Disallow: /?s=* Disallow: /*/?s=*\ Disallow: /...
alturkovic's user avatar
0 votes
1 answer
51 views

Disallow all directories including a given word in robots.txt

I would like to disallow all directories including a given phrase, i.e. www.example.com/directory-1-aaa/ www.example.com/directory-1-bbb/ www.example.com/directory-1-ccc/ www.example.com/directory-1-...
Wojciech Dziedzic's user avatar
0 votes
0 answers
18 views

Reproduce the API links appearing in Google search results

We have the following problem. The links to our API (that doesn't require any authentication) have suddenly started showing up in Google and Bing search results. As far as we know, these links have ...
ZeeG's user avatar
  • 1
2 votes
2 answers
93 views

Trying to block images from google image indexing but not working

I am trying to block images from google image search, but for some reason they are not working. I have a Wordpress self-hosted blog and I have full access to the backend. HTML of mydomain.com looks ...
cyanite's user avatar
  • 21
2 votes
1 answer
144 views

How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?

I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
Clipso's user avatar
  • 21
1 vote
1 answer
72 views

Why is my server 302-redirecting Lighthouse away from my robots.txt file, but not when I visit it with a web browser?

This is my robots.txt file: User-agent: * Disallow: /user/ Disallow: /user/* Disallow: /home/* Disallow: /adminpanel/* Disallow: /Valorant_Guess/* Disallow: /Leagueoflegends_Guess/* Disallow: /...
szocel's user avatar
  • 41
2 votes
1 answer
127 views

Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?

I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1? My goal is to allow crawlers not to run over ...
Milos Cuculovic's user avatar
2 votes
1 answer
1k views

What does User-agent: Mediapartners-Google mean?

I was going through the google support website and there they said: "To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
Muhammad Ahmad's user avatar
0 votes
0 answers
56 views

Alternate Canonical Tags error on Google Search Console

I recently started my blogging site and after a few days on Google Search Console I got an error saying alternate page with proper canonical tag. Now I searched the internet and found the fix. A line ...
Muhammad Ahmad's user avatar
1 vote
1 answer
47 views

WordPress doesn't identify the subfolder

I have a subfolder with a separate WordPress running on it. When I click on a link which should lead to the destination in the new website (the subfolder) it reads everything except the mentioned ...
Majid Marashi's user avatar
3 votes
2 answers
81 views

Correct way to define all in robots.txt

Is the correct way of defining allow all or disallow all in robots to use Allow: /? I've seen a variation on this using a * rather than a /. Allow: * Is the * method valid? Thanks
onebitrocket's user avatar
1 vote
1 answer
43 views

Is blocking indexing and crawling at the same time bad?

have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
Sarian Morio's user avatar
2 votes
1 answer
183 views

robots.txt to disallow folder, but allow sub-folder

I'm fairly new to robots.txt but have the basics. My first problem is that I'd like to disallow all the translated pages, except in the help file. So I have this: Disallow: /de/ But I want, at the ...
arathra's user avatar
  • 23
0 votes
1 answer
40 views

Allow and dissallow directory with # in robots.txt [duplicate]

I'd like to block and unblock following URL formats: Allow:https://example.com/buyer/#/search/catalog/Lubricating-Oils/Automotive-Lubricants Dissallow:https://example.com/buyer/#/search/product-...
suhanshukla's user avatar
0 votes
0 answers
42 views

How to stop crawling subdomains used for API and backend?

I have a subdomain (that is core.domain.com) which is used for API and backend and it doesn't have folder to upload a separate robots.txt to stop crawling the mentioned subdomain.
Reza Ataei's user avatar
2 votes
1 answer
54 views

How to decrease unwanted traffic after malware attack?

After malware attack on my website I'm seeing more than 20000 get requests every day which are pointing to non existent places on my server. Those addresses were valid during the attack and were ...
hanecki's user avatar
  • 21
2 votes
1 answer
170 views

How to disallow URLs in Robots.txt that DO NOT end with a certain characters

My website has 2 types of URLs URLs that end with .html -> example.com/example.html URLs that do not end with .html -> example.com/example The .html version is the version I want to be ...
Eddie Segal's user avatar
3 votes
2 answers
423 views

How to configure robots.txt to disallow any URL/file/directory with a certain word in it?

I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt, but I have a more detailed need than that. Google search console keeps trying to discover/crawl certain ...
ravl13's user avatar
  • 45
2 votes
0 answers
187 views

What is correct way to stop query indexing using Robots.txt?

How to stop indexing search result using Robots.txt? hello. Recently, I'm having an issue with indexed pages in Google Search Console growing 10x. These pages are mostly search query (/?s=) and ...
cheonmu's user avatar
  • 21
2 votes
1 answer
356 views

Why is Google indexing my robots.txt file?

Google Search is indexing my robots.txt file. I know this because when I search site:example.com on Google, my robots.txt shows in the list of results. I don't want my robots.txt to show in Google. ...
Mustafa's user avatar
  • 123
1 vote
1 answer
411 views

Getting Robots.txt Not Found 404 Error on Google Search Console

I have created robots.txt in my root directory and when I navigate to my website address like https://example.ca/robots.txt I am able to see the rules User-agent:* Disallow: /captcha/ Disallow: /...
Behseini's user avatar
  • 115
2 votes
1 answer
144 views

Googlebot is crawling unpublished pages from Adobe AEM and getting 404 errors

My company is currently using Adobe AEM as a CMS for our company website. We are currently having issues with 4xx errors appearing specifically 404 as unpublished pages are still being crawled. What ...
Zachc40's user avatar
  • 21
2 votes
2 answers
46 views

Disallow top level of directory, but not subdirectories in robots.txt

I have a directory that I don't want Google to index at the top level. I currently do the following: Disallow: /profiles/ This stops Google from indexing https://example.com/profiles/, but it also ...
Keyslinger's user avatar
3 votes
1 answer
724 views

Redirect Errors in Google Search Console when trying to index site

I've built a small site for a local trader which is not a complicated site in any way. In fact it is 100% static. I have a few years experience and in that time have never encountered an error with ...
Cutey from Cute Code's user avatar
2 votes
1 answer
309 views

Only allow crawling of pages in sitemap

My idea is, to use robots.txt to disallow everyting and put all allowed pages into the sitemaps. User-agent: * Disallow: / Sitemap: https://www.example.com/sitemap.xml Is this working as expected?
Georgio's user avatar
  • 21
3 votes
3 answers
56 views

Why would a human read robots.txt

As a part of our new website I've written code in a page filter to do some basic bot / crawler detection by looking at the user agent. That way when a bot is detected we serve up some generic user ...
user avatar
0 votes
1 answer
528 views

My website's image does not work with twitter cards (unclear issues with robots.txt)

Problem I am trying to properly set up metadata for a twitter summary card on my website. I have everything in place, but the card image is not showing. I use Grav, but I guess it does not matter. ...
Marcel's user avatar
  • 188
3 votes
1 answer
92 views

How to Allow bots only on main page

I'm trying to create a robots.txt file that allows bots to access ONLY the home page and no other page. Below is the content of the robots.txt file based on the research I have done. Is this correct? ...
The Oracle's user avatar
8 votes
1 answer
717 views

Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)

Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says the /fish rule matches any path that starts with /fish (...) it doesn'...
Déjà vu's user avatar
  • 183
4 votes
1 answer
272 views

Is there a way to allow ad bots to crawl a website that has "noindex,nofollow" on it?

Due to duplicated content, there is a "noindex,nofollow" on the website, but in order to run ppc ads, there needs to be a way for the ads bots to crawl the website Can this be done without ...
user1898662's user avatar
2 votes
2 answers
472 views

Should I exclude the assets folder from search engine crawlers in angular

I'm working with angular and had added the assets folder as element to exclude from the search engine crawlers, into my robots.txt file. But I now get the following error while testing the site health ...
Raphaël Balet's user avatar
2 votes
0 answers
39 views

Disallow crawling of all links to a domain, except only specific URLs with parameters

I have a domain like example.com under which everything is blocked and links to an SPA. Like example.com, and we've disabled everything like so: User-agent: * Disallow: / Now this SPA uses parameters ...
Florian Monfort's user avatar
1 vote
1 answer
28 views

Block crawling of URLs with "tag" using robots.txt

I want to block robots from crawling URLs that contain the word "tag" on my website. Here are some example URLs: https://www.example.com/six-senses-fort-barwara-a-glimpse-into-the-...
user avatar
1 vote
1 answer
24 views

Question regarding Sitemap value in a robots.txt file, and the lines of a txt sitemap

I have two related questions. To start, in a robots.txt file can you just put the following for sitemap: Sitemap: www.example.com Or do you have to have: Sitemap: http://www.example.com or Sitemap: ...
Trevor's user avatar
  • 11
2 votes
1 answer
274 views

How to block all .html pages in robot.txt?

I have a new website that has a different product URL from the old website. URL Old Website https://www.example.com/product/MA-593838554.html https://www.example.com/product/MR-559927541.html etc ...
fikfe's user avatar
  • 63
4 votes
1 answer
1k views

Do I still need to ping Google after changing a sitemap nowadays?

Months ago I've implemented SEO in a website and it was recommended that every time that the sitemaps changed I should ping this URL http://www.google.com/ping?sitemap=http://mywebsite.com/sitemap.xml....
Artur Haddad's user avatar
7 votes
1 answer
2k views

Should I block the '_next' directory in a site using Next.js with robots.txt?

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx ...
checklist's user avatar
  • 171
4 votes
0 answers
19 views

Prevent crawling of mobile site will impact the actual's site ranking or indexing?

We have a responsive mobile-first indexing site in place. Unfortunately, the m.example.com - mobile version is also there and it ranks for certain keywords. We don't want to have such a separate m....
Ranjini's user avatar
  • 41
3 votes
1 answer
142 views

Avoid unnecessary crawling of ad landing URLs by disallowing query string with robots.txt

Will the following configuration allow me to run an Ads campaign for products with the ?string parameter in the URL? User-agent: * Disallow: /*?string User-agent: AdsBot-Google Allow: / User-agent: ...
TheGrimmy61's user avatar
3 votes
1 answer
383 views

Is there a way to have Google index a page but not cache it?

I see that many popular sites have pages indexed in Google, but when I try to access Google's cache the page isn't there. How can I go about doing the same thing with my site? I read that you have to ...
One Piece Power's user avatar

1
2 3 4 5
15