All Questions
Tagged with web-crawlers search-engine-indexing
11 questions
15
votes
3
answers
3k
views
How does Google find a domain with no links to it?
I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else.
I just discovered tonight that the page is already indexed in ...
- 21.6k
10
votes
8
answers
9k
views
How long did it take for your new website to be indexed by search engines [duplicate]
Possible Duplicate:
Why isn't my website in Google search results?
I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
5
votes
5
answers
262
views
What measures can be taken to make sure Google is aware of the existence of a newly created page?
Consider a website with a large number of pages. New pages are published regularly.
When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
- 159
5
votes
2
answers
8k
views
Can Angular project be indexed in Google without Angular Universal?
I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
- 151
4
votes
2
answers
340
views
Do SEO analysis bots have any advantage in terms of website traffic?
I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as
AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/...
- 452
3
votes
5
answers
5k
views
Google crawl speed -- how fast can it go?
I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
- 141
3
votes
3
answers
132
views
How long before Google indexes a new (to me) domain that previously had spam and virus problems?
I registered a new ".com" domain, but when I added it to the Google Search Console I saw that there were problems with viruses and spam problems. I submitted a request for reconsideration. Google ...
- 131
2
votes
1
answer
236
views
What is a difference in the order of "follow" and "index" in a robots meta tag?
What is the difference between
<meta name="robots" content="follow, index">
and
<meta name="robots" content="index, follow">
Would either create ...
- 21
2
votes
2
answers
2k
views
Is it possible for Google to discover a website that is not linked to anywhere? [duplicate]
My colleague claims that he has seen cases where websites were "discovered" by Google without having a link to them anywhere on the web.
For him, this means that the only way to not being indexed (...
- 227
2
votes
2
answers
225
views
Search engines that index your site despite a disallow in robots.txt
Recently, WordPress changed their options from "Block search engines" to "Discourage search engines"
So that leaves me wondering, that rewording must be that some search engines disregard your ...
- 131
1
vote
1
answer
193
views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
- 251