All Questions
Tagged with web-crawlers robots.txt
201 questions
0
votes
0
answers
12
views
Testing sitemap, rich results, etc. on my dev slot
How can I set things up so that I can access my dev slot with the Google developer console, but in a way that Google will not index the page.
I can't use robots.txt because we push our app to ...
- 139
2
votes
1
answer
60
views
Is there a crawl-delay limit?
Does anyone know if there is a max limit to Crawl-Delay time? I read people set to 10 to keep bots down to 8,600 crawls a day, but I would like to see the limit even more.
I set mine to 30, which ...
0
votes
0
answers
42
views
Do two word bots need escaping with %20 or are spaces Ok in robots.txt
I want to tell Baidu not to crawl. One of their crawlers is Baidu Spider.
Should that be:
User-agent: Baidu%20Spider
or
User-agent: Baidu Spider
- 47
0
votes
0
answers
25
views
Issues with Googlebot crawling and negative number pages
I am dealing with very heavy load from Googlebot, I have hundreds of sites running on multiple IPs all on the same server and googlebot is constantly crawling these sites causing a lot of availability ...
- 101
1
vote
2
answers
3k
views
How do I stop Amazon from crawling a website?
I would like to prevent Amazon from scraping product data on my website. So I found this document: https://developer.amazon.com/amazonbot
And this example:
User-agent: Amazonbot # Amazon's ...
Rhothgar
1
vote
0
answers
560
views
Should Feed/RSS Be Blocked from Robots.txt?
Should Feed/RSS be blocked via robots.txt?
Disallow: /*feed*
Disallow: /*rss*
If yes, is this the correct way to apply?
If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
- 41
1
vote
1
answer
49
views
What is the Query to add in robots.txt for blocking URLs has some special characters?
I was going through the Google Search Console pages issues and one of the issue reason is "Alternate page with proper canonical tag" and this has some URLs which look like below
example.com/...
1
vote
1
answer
48
views
What do backslashes in WP generated robots.txt rules mean?
I am trying to implement a library to handle robots.txt and I am a bit confused about the meaning of some rules that I believe are generated by WordPress:
Disallow: /?s=*
Disallow: /*/?s=*\
Disallow: /...
- 113
2
votes
2
answers
438
views
Trying to block images from google image indexing but not working
I am trying to block images from google image search, but for some reason they are not working. I have a Wordpress self-hosted blog and I have full access to the backend. HTML of mydomain.com looks ...
- 21
2
votes
1
answer
966
views
How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?
I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
- 21
2
votes
1
answer
397
views
Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?
I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over ...
- 123
3
votes
2
answers
5k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
- 102
1
vote
1
answer
94
views
Is blocking indexing and crawling at the same time bad?
have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
0
votes
1
answer
67
views
Allow and dissallow directory with # in robots.txt [duplicate]
I'd like to block and unblock following URL formats:
Allow:https://example.com/buyer/#/search/catalog/Lubricating-Oils/Automotive-Lubricants
Dissallow:https://example.com/buyer/#/search/product-...
3
votes
2
answers
1k
views
How to configure robots.txt to disallow any URL/file/directory with a certain word in it?
I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt, but I have a more detailed need than that.
Google search console keeps trying to discover/crawl certain ...
- 45
2
votes
0
answers
433
views
What is correct way to stop query indexing using Robots.txt?
How to stop indexing search result using Robots.txt?
hello.
Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.
These pages are mostly search query (/?s=) and ...
- 21
2
votes
1
answer
871
views
Only allow crawling of pages in sitemap
My idea is, to use robots.txt to disallow everyting and put all allowed pages into the sitemaps.
User-agent: *
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
Is this working as expected?
- 21
8
votes
1
answer
791
views
Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)
Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says
the /fish rule matches any path that starts with /fish (...) it doesn'...
- 183
4
votes
1
answer
420
views
Is there a way to allow ad bots to crawl a website that has "noindex,nofollow" on it?
Due to duplicated content, there is a "noindex,nofollow" on the website, but in order to run ppc ads, there needs to be a way for the ads bots to crawl the website
Can this be done without ...
- 41
2
votes
2
answers
1k
views
Should I exclude the assets folder from search engine crawlers in angular
I'm working with angular and had added the assets folder as element to exclude from the search engine crawlers, into my robots.txt file.
But I now get the following error while testing the site health ...
- 163
2
votes
0
answers
61
views
Disallow crawling of all links to a domain, except only specific URLs with parameters
I have a domain like example.com under which everything is blocked and links to an SPA.
Like example.com, and we've disabled everything like so:
User-agent: *
Disallow: /
Now this SPA uses parameters ...
- 121
5
votes
1
answer
861
views
Why does it cause issues to use robots.txt and a meta robots tag at the same time?
It seems common wisdom that using X-Robots-Tag/meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.:
Using both X-Robots-Tag and meta robots on a URL is ...
- 153
2
votes
1
answer
228
views
Robots.txt not blocking yet GSC URL Inspection Tool says it is
Recently I realized Disallow /was set on my staging sites.
I didn't realize until I got a warning in Search Console...
Mistakes happen
I've removed the directive and cleared all caches at least 10 ...
- 6,559
4
votes
2
answers
340
views
Do SEO analysis bots have any advantage in terms of website traffic?
I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as
AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/...
- 452
1
vote
1
answer
397
views
Should I add links to the Sitemap that are disallowed in robots.txt?
My robots.txt contains the line Disallow: /search.php.
Shall I nevertheless include http://example.com/search.php into my sitemap?
- 113
3
votes
1
answer
149
views
What is a robots.txt file and how can I use it
I want to use a robots.txt file for my website, which is currently displaying a 404 error when requesting it (e.g., https://www.example.com/robots.txt). How do I make one? There's no option to create ...
2
votes
1
answer
28
views
How do rule length and use of wildcards determine whether allow or disallow takes precedence in Robots.txt?
I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt
Could someone elaborate on the difference between these two:
http://example.com/page.htm
allow: /page
...
- 41
1
vote
1
answer
94
views
Do I need to allow bots on /* if I disallow /specific-directory?
If I want all bots to not crawl /specific-directory, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate?
# All Bots
User-...
user118386
2
votes
2
answers
83
views
Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?
I've written my own SiteSearch Script in PHP.
The SiteSearch parses two GET parameters from the queryString:
search // the search-phrase
filters (optional) // indicating which parts of the site to ...
- 2,365
2
votes
1
answer
231
views
Do I need to submit sitemap.xml to google if I already have it in robots.txt and hosted on S3?
I have a static site hosted on S3 bucket which will have a robots.txt at root directory level.
This robots.txt file will have reference to my sitemap.xml file which is hosted on another S3 bucket. It ...
- 23
2
votes
0
answers
35
views
Could allowing our images to be indexed hurt our web search rankings?
We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
- 161
2
votes
0
answers
463
views
Restrict Google from indexing subdomain from Azure CDN services
How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
6
votes
4
answers
4k
views
How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?
In my web application I generate links in the following format:
https://example.com/^token^
Each link according to my specs is sent via email and should not allow robots to scrape it. In order to ...
0
votes
1
answer
44
views
Is it possible to tell web crawlers to agree to a site's TOS before indexing?
Is there a way with robots.txt or page headers to inform the managers of web crawlers they must agree to a terms of service prior to indexing any crawled pages?
- 551
4
votes
1
answer
1k
views
Link rel="nofollow" vs robots.txt disallow for "/search" URLs?
There's a /search path on my web app that I don't want robots to crawl or index.
What would be the best approach for this scenario?
1: Block them from accessing by adding a Disallow for /search on ...
- 988
4
votes
1
answer
590
views
How do I "noindex" a text (.txt) file?
I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as ...
- 573
0
votes
0
answers
93
views
Google search says No information is available for my GitHub page
Today I search my GitHub page in Google Chrome and Firefox (Through Google search) and for the first time, it says no information is available (it says 2019- GitHub under that it says no information ...
- 101
6
votes
1
answer
1k
views
What is difference between robots.txt, sitemap, robots meta tag, robots header tag?
So I am trying to learn SEO and I am honestly confused and have following 8 questions.
Do I tell a bot not to visit a certain link through X-Robots-Tag or through robot meta tag or robots.txt?
Is it ...
- 185
2
votes
2
answers
1k
views
Disallow root but not 4 subdirectories for robots.txt
I have a project and I would like to disallow everything starting with root.
From what I understand I think I can do so by doing this
Disallow: /
Disallow: /*
However I would like to allow 4 ...
- 185
-1
votes
1
answer
31
views
Why is Azure pinging my site?
I have noticed more and more, traffic from IPs assigned to Microsoft Azure/Microsoft Corporation hitting my WordPress site many times a day. It's direct traffic to just the home and privacy policy ...
4
votes
2
answers
3k
views
Why are Bing and SEMRushBot ignoring crawl-delay in my robots.txt file?
We host a large number of sites that have a large number of catalog pages. We would like to slow down several bots because the traffic is quite excessive from those bots. Specifically, we get quite a ...
- 141
1
vote
0
answers
44
views
Disallowing a handler in robots.txt while adding its dynamic URLs to the XML sitemap
I'm using ASP.NET webforms, and I have a page lets call it Subjects.aspx, I don't want crawlers to crawl that page but I want them to crawl the dynamic URLs that are powered by it. For example /...
- 123
2
votes
1
answer
473
views
Robots.txt for a multilanguage site where root is redirected
I have a site which offers two languages, English and Spanish. When the user navigates to the home page, let's say www.example.com the page redirects you to either /es if your browser language is ...
2
votes
1
answer
237
views
Google image crawler won't respect my robots.txt entry to not crawl images
I was looking for a way to prevent reverse image searching (namely I didn't want for people who had a copy of one of my images to upload it to google and discover where it originated from). I created ...
- 123
2
votes
1
answer
2k
views
Ahrefs reports "Fetching robots.txt took too long"
My robots:
User-agent: *
Disallow: /wp-admin
Disallow: /sistema
Disallow: /site
Disallow: /old
Sitemap: http://www.example.com.br/page-sitemap.xml
In the google test tool: https://www.google.com/...
- 121
5
votes
1
answer
3k
views
Can robots.txt redirect to a different domain?
I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer.
example: https://www.mydomain.example/...
- 51
2
votes
2
answers
4k
views
What does crawl-delay: 1 mean if it is interpreted as is?
I have a problem of understanding what crawl-delay: 1 means if it isn't ignored and actually being interpreted as is by a given web crawler.
I understood from this answer by Stephen Ostermiller that:
...
user58733
1
vote
2
answers
376
views
crawl-delay: x --------- has someone here managed to understand how the math works there?
From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than ...
user58733
1
vote
1
answer
95
views
Hide /path from search engines, except for /path/that-i-want-indexed
In our robots.txt we have something like
User-agent: *
Disallow: /v6.0
Disallow: /v5.8
Disallow: /v5.7
Disallow: /v5.5
Disallow: /v5.1
Disallow: /v5.0
Which properly prevents search engines from ...
- 111
3
votes
1
answer
343
views
Why are bots repeatedly crawling pages of my site that I never created?
In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
- 165