All Questions
Tagged with web-crawlers search-engine-indexing
76 questions
0
votes
1
answer
36
views
Do indexer bots set navigator.webdriver property to true?
The webdriver read-only property of the navigator interface indicates whether the user agent is controlled by automation.
Source
I couldn't find anywhere if it includes indexing crawlers such as ...
- 175
2
votes
1
answer
1k
views
Google Indexing Tons of Non-Existent URLs
A client that I am working for has come to me with a problem. Their Google Adwords account has been disabled because there is a bunch of traffic to crazy non-existent pages on their website. Here are ...
- 23
4
votes
2
answers
340
views
Do SEO analysis bots have any advantage in terms of website traffic?
I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as
AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/...
- 452
1
vote
1
answer
59
views
Is disallowing all useragents sufficient to keep a site from being crawled by search engines?
We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
- 113
-1
votes
3
answers
3k
views
How to remove bulk urls from google index
I have URLs indexed by Google and I want to remove them from indexing. I found that I can use bulk remove option by providing a CSV file including all indexed URLs to Google Search Console. I have ...
- 1
1
vote
1
answer
94
views
Is blocking indexing and crawling at the same time bad?
have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
2
votes
2
answers
396
views
Will bots crawl my page generated by javascript
I use simple javascript to generate pages based on thousands of products stored in a JSON object.
The page URLs will be like /productList?keyword=XXX&page=x. The keyowrd is what uses input through ...
- 505
5
votes
2
answers
11k
views
Ampersand (&) in actual URL and sitemap
I have a sitemap (which is submitted to Google for indexing) in which I am adding URLs which have an ampersand (&) in it. Since in sitemap & has to be escaped I replaced & with & ...
- 153
6
votes
2
answers
1k
views
How to keep crawler robots from indexing my page when there is a query string present in the URL?
I have a /products page.
In that page, users may use filters to select the products they want to see.
Those filters will work using query strings like: /products?brands=X,Y&size=1,2 and so on.
I ...
- 988
5
votes
3
answers
3k
views
Does Bing execute Javascript when crawling and indexing web pages (like Google)?
Some content on some web pages is only displayed with Javascript when a button is clicked (for example). We know that Google executes Javascript and is (very likely) to find this content, but what ...
- 4,625
-2
votes
1
answer
710
views
How to block everything from being indexed except sitemap.xml
I want to block everything and index sitemap.xml file alone. So I do it as shown below:
User-agent: *
Disallow: /
Allow: /sitemap.xml
Sitemap: https://example.net/sitemap.xml
However, I am not sure ...
- 101
1
vote
1
answer
59
views
Should we Put Coupon Pages into sitemap because they do have a faster expiry rate?
Is it a good practice to keep coupons/ voucher pages into a sitemap. because they do have a faster expiry rate.
Case:
We are updating our sitemap once in 24 hours so, will it be a good practice to ...
- 470
3
votes
1
answer
6k
views
URL will be indexed only if certain conditions are met
I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
- 397
5
votes
2
answers
8k
views
Can Angular project be indexed in Google without Angular Universal?
I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
- 151
1
vote
4
answers
3k
views
In WordPress, should I "Allow search engines to index this site" during development?
I am going to create a web page, and I will do it online.
Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
- 504
2
votes
1
answer
236
views
What is a difference in the order of "follow" and "index" in a robots meta tag?
What is the difference between
<meta name="robots" content="follow, index">
and
<meta name="robots" content="index, follow">
Would either create ...
- 21
3
votes
2
answers
3k
views
Preventing links with query parameters from being indexed by Google
I manage a mobile version of a desktop site. The mobile site has an mdot subdomain and is served from a different server. As part of recommended best practices by Google, I've included a link on the ...
- 170
1
vote
1
answer
417
views
Can search engine fetch/index website that using ipv6 only?
I see some host providers offer cheap server but only support ipv6. I want to rent it, but I am still considering the SEO impact.
Can google and other search engine crawl and index it? If yes, will it ...
- 1,136
6
votes
2
answers
2k
views
Delivering a different version of website to Google robots (SSR)
I am working on an enterprise website project as an SEO manager.
They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render the ...
- 443
2
votes
1
answer
614
views
Can I access the old version of Google Search Console to see crawling and indexing?
I remember google webmaster console had a crawl and indexing, But in new version, I couldn't find crawling or indexing. Everything is different.
Can I access to old style of google console?
- 269
0
votes
1
answer
269
views
Google not indexing product pages of my website [duplicate]
After many months and many changes to sitemaps/pages trying to solve the problem, Google still is not indexing the product pages of this site (link removed).
Here's the sitemap i'm using (link removed)...
- 3
3
votes
5
answers
5k
views
Google crawl speed -- how fast can it go?
I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
- 141
3
votes
1
answer
1k
views
Is it good for SEO to change an article's slug when changing its title?
I have the following URL view:
example.com/article/{slug}-{id}
If I change the title of the article, then the slug will also change so that there is no disagreement.
Please note that the article ...
- 157
2
votes
1
answer
306
views
If a web crawler get a 404 error, will it not index that url?
Say I have a page page.example.com/page. In my (python) server, you only get served html if you are logged in, which you can do at page.example.com/login. If you aren't logged in, the server responds ...
- 123
1
vote
1
answer
167
views
Why would Google Search Console report many times more indexed URLs than a Screaming Frog crawl finds?
I've just crawled my website in Screaming Frog and I have around 6K Internal URLs and 4K external URLs. Then I looked in GSC and saw that Google indexed 50K pages.
How is that possible?
1
vote
1
answer
187
views
Is it normal that URL's are getting crawled in Screaming frog Still they have a no-follow and no-index tag on them
I'm getting a lot of URL's in screaming frog where the URLs are no-follow no-index but, they are still getting crawled in screaming frog. Is it fine or should I change anything in this?
- 470
2
votes
0
answers
35
views
Could allowing our images to be indexed hurt our web search rankings?
We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
- 161
1
vote
2
answers
344
views
Deleted tags & categories from my blog still come up in Google search results
I am a newbie in blogging. I've deleted some of the unwanted tags and categories from my wordpress blog but they kinda still come put in the search results when I search my website on Google.
How do ...
- 133
2
votes
2
answers
418
views
How many levels of sub-pages do Google bots crawl to when fetching and indexing linked pages in Google Search Console?
Let's say I submit a sitemap page that contains all the links I want Google to crawl, but instead of submitting a sitemap, I fetch the page and linked pages and let Google index them. How many levels ...
- 231
0
votes
0
answers
212
views
Invalid traffic coming from Sogou
I have not indexed my website in Sogou. I have only very less visitors. But i am getting traffic from Sogou daily. The search keyword is "site:mywebsitename" and is from China. Just visiting main page....
- 154
15
votes
3
answers
3k
views
How does Google find a domain with no links to it?
I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else.
I just discovered tonight that the page is already indexed in ...
- 21.6k
1
vote
2
answers
1k
views
How can I stop Google from indexing "pretty links" external redirects from my WordPress site?
When I search site:[example].com in Google for my blog, the majority of the pages that are being indexed are "pretty links" to external sites.
I created these links using the Pretty Links plugin so ...
- 11
3
votes
3
answers
132
views
How long before Google indexes a new (to me) domain that previously had spam and virus problems?
I registered a new ".com" domain, but when I added it to the Google Search Console I saw that there were problems with viruses and spam problems. I submitted a request for reconsideration. Google ...
- 131
2
votes
1
answer
69
views
Do search engines crawl links with query strings?
If I were to have a url like https://example.com/books, that contained several links to urls such as:
https://example.com/books?id=foo
https://example.com/books?id=bar
https://example.com/books?id=...
- 227
1
vote
0
answers
593
views
How to prevent CDN content URLs being indexed by Google [duplicate]
Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too).
So, even when a URL is disallowed in robots....
10
votes
8
answers
9k
views
How long did it take for your new website to be indexed by search engines [duplicate]
Possible Duplicate:
Why isn't my website in Google search results?
I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
1
vote
0
answers
80
views
Limit search engines to indexing just one domain when multiple domains share the same web server and document root
I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
- 11
3
votes
3
answers
4k
views
How can I force Google to re-crawl a link?
Google Webmaster Tools is showing thousands of links from a North Korean travel agency, but when I go to the link, they don't exist! How can I force Google to re-index that link so it drops off my ...
- 512
4
votes
1
answer
2k
views
Website not fetched by Bingbot: Unexpected Error
My website has been online for a couple years. It's on the first result for Google but it doesn't seem to be indexed by Bing. I tried to fetch it with bing in their webmaster tools and it always ...
- 43
3
votes
7
answers
1k
views
Prevent robots from indexing specific word
I need to prevent robots from indexing specific word on my webpage.
Here is what I did:
Let's say I need to prevent JOHN GALLOW from being indexed. For that purpose, I wrote this pararaph: Over the ...
- 133
0
votes
2
answers
110
views
How to say, "Google, index my web site only during specific times?"
Is it possible tell Google to only index my site between certain hours?
If it is, how can I manage it?
- 11
4
votes
1
answer
3k
views
How to make sure an individual page from a website is not indexed by search engines?
I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed.
Can it be reliably ...
- 463
2
votes
0
answers
374
views
Google is adding "Archive" to the title of tag and category pages and not using the meta description
I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them?
How to change them? Actually it looks so awkward!
I have configured my ...
- 63
1
vote
2
answers
519
views
Should I submit both mobile and desktop domain names for indexing on non-responsive sites or is that bad for SEO?
I have a desktop version of my site using yourwebsite.com and a mobile version using m.yourwebsite.com.
Should I treat them as one site and only submit the desktop version along with its sitemap and ...
-1
votes
1
answer
72
views
Maximum time to cache HTTPS URL after HTTP to HTTPS migration?
We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
- 77
1
vote
1
answer
705
views
Cookie-based A/B testing on same URL - SEO impact?
we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much.
...
- 11
0
votes
0
answers
17
views
In a sitemap, should I update the lastmod tag of a url based on the text content or html content? [duplicate]
Imagine I have this blogging / ecommerce website with 1000 posts / products. And I've built a sitemap for it, which is dynamically generated. Basically it's a list with a bunch of <url> and <...
- 988
4
votes
1
answer
336
views
Using robots.txt to deny access to MediaWiki special pages using substring matching
I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
- 165
1
vote
1
answer
547
views
Trick to index infinite scroll content on Google?
I have a Question & Answer website, sort of like stackoverflow.
When you open it, it displays 20 latest questions.
When scroll to bottom of page, and it fetches 20 more, and so on, infinitely.
...
- 111
0
votes
1
answer
505
views
Crawl errors for pdf download links in my website
We are building a local business classified website which has about 6000 unique listingsunder huge number of categories and sub-categories.
For each listing there is a details page in which we have ...
- 142