All Questions
Tagged with web-crawlers robots.txt
14 questions with no upvoted or accepted answers
2
votes
0
answers
433
views
What is correct way to stop query indexing using Robots.txt?
How to stop indexing search result using Robots.txt?
hello.
Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.
These pages are mostly search query (/?s=) and ...
- 21
2
votes
0
answers
61
views
Disallow crawling of all links to a domain, except only specific URLs with parameters
I have a domain like example.com under which everything is blocked and links to an SPA.
Like example.com, and we've disabled everything like so:
User-agent: *
Disallow: /
Now this SPA uses parameters ...
- 121
2
votes
0
answers
35
views
Could allowing our images to be indexed hurt our web search rankings?
We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
- 161
2
votes
0
answers
463
views
Restrict Google from indexing subdomain from Azure CDN services
How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
2
votes
0
answers
418
views
Set bot crawl delay 10 seconds EXCEPT Googlebot?
Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace?
I like being indexed by all these bots but with Yandex and Baidu it is ...
- 131
1
vote
0
answers
560
views
Should Feed/RSS Be Blocked from Robots.txt?
Should Feed/RSS be blocked via robots.txt?
Disallow: /*feed*
Disallow: /*rss*
If yes, is this the correct way to apply?
If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
- 41
1
vote
0
answers
44
views
Disallowing a handler in robots.txt while adding its dynamic URLs to the XML sitemap
I'm using ASP.NET webforms, and I have a page lets call it Subjects.aspx, I don't want crawlers to crawl that page but I want them to crawl the dynamic URLs that are powered by it. For example /...
- 123
1
vote
0
answers
80
views
Limit search engines to indexing just one domain when multiple domains share the same web server and document root
I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
- 11
1
vote
0
answers
371
views
Stop getting traffic from Googlebot on high server load
In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing.
So what I thought of; is depending on the ...
- 111
0
votes
0
answers
12
views
Testing sitemap, rich results, etc. on my dev slot
How can I set things up so that I can access my dev slot with the Google developer console, but in a way that Google will not index the page.
I can't use robots.txt because we push our app to ...
- 139
0
votes
0
answers
42
views
Do two word bots need escaping with %20 or are spaces Ok in robots.txt
I want to tell Baidu not to crawl. One of their crawlers is Baidu Spider.
Should that be:
User-agent: Baidu%20Spider
or
User-agent: Baidu Spider
- 47
0
votes
0
answers
25
views
Issues with Googlebot crawling and negative number pages
I am dealing with very heavy load from Googlebot, I have hundreds of sites running on multiple IPs all on the same server and googlebot is constantly crawling these sites causing a lot of availability ...
- 101
0
votes
0
answers
93
views
Google search says No information is available for my GitHub page
Today I search my GitHub page in Google Chrome and Firefox (Through Google search) and for the first time, it says no information is available (it says 2019- GitHub under that it says no information ...
- 101
-1
votes
1
answer
31
views
Why is Azure pinging my site?
I have noticed more and more, traffic from IPs assigned to Microsoft Azure/Microsoft Corporation hitting my WordPress site many times a day. It's direct traffic to just the home and privacy policy ...
