All Questions
4 questions with no upvoted or accepted answers
2
votes
0
answers
433
views
What is correct way to stop query indexing using Robots.txt?
How to stop indexing search result using Robots.txt?
hello.
Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.
These pages are mostly search query (/?s=) and ...
- 21
2
votes
0
answers
61
views
Disallow crawling of all links to a domain, except only specific URLs with parameters
I have a domain like example.com under which everything is blocked and links to an SPA.
Like example.com, and we've disabled everything like so:
User-agent: *
Disallow: /
Now this SPA uses parameters ...
- 121
1
vote
0
answers
371
views
Stop getting traffic from Googlebot on high server load
In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing.
So what I thought of; is depending on the ...
- 111
0
votes
0
answers
25
views
Issues with Googlebot crawling and negative number pages
I am dealing with very heavy load from Googlebot, I have hundreds of sites running on multiple IPs all on the same server and googlebot is constantly crawling these sites causing a lot of availability ...
- 101