All Questions
15 questions
4
votes
2
answers
340
views
Do SEO analysis bots have any advantage in terms of website traffic?
I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as
AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/...
- 452
4
votes
1
answer
336
views
Using robots.txt to deny access to MediaWiki special pages using substring matching
I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
- 165
2
votes
2
answers
225
views
Search engines that index your site despite a disallow in robots.txt
Recently, WordPress changed their options from "Block search engines" to "Discourage search engines"
So that leaves me wondering, that rewording must be that some search engines disregard your ...
- 131
2
votes
2
answers
216
views
Prevent crawler that doesn't honoring robots.txt
I have some problem, when I try to write robots.txt for my site ...
I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
- 123
2
votes
1
answer
35
views
Should article preview pages be crawled and indexed by search engines?
I have a page called "all articles" that loads previews of articles using AJAX. Because of that, the content won't be visible to search engines, so the page would be considered "poor content".
Since ...
- 217
2
votes
0
answers
35
views
Could allowing our images to be indexed hurt our web search rankings?
We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
- 161
1
vote
4
answers
3k
views
In WordPress, should I "Allow search engines to index this site" during development?
I am going to create a web page, and I will do it online.
Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
- 504
1
vote
1
answer
193
views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
- 251
1
vote
1
answer
94
views
Is blocking indexing and crawling at the same time bad?
have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
1
vote
2
answers
446
views
What is better for SEO robots.txt file or dynamically generated robots.txt URL?
I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
- 669
1
vote
3
answers
219
views
How do I remove a LOT of indexed pages from Google?
A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
- 11
1
vote
0
answers
80
views
Limit search engines to indexing just one domain when multiple domains share the same web server and document root
I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
- 11
1
vote
0
answers
593
views
How to prevent CDN content URLs being indexed by Google [duplicate]
Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too).
So, even when a URL is disallowed in robots....
1
vote
1
answer
85
views
Why would I use a robots.txt file? [duplicate]
From what I understand after reading Google's Controlling Crawling and Indexing:
The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
- 111
-2
votes
1
answer
711
views
How to block everything from being indexed except sitemap.xml
I want to block everything and index sitemap.xml file alone. So I do it as shown below:
User-agent: *
Disallow: /
Allow: /sitemap.xml
Sitemap: https://example.net/sitemap.xml
However, I am not sure ...
- 101