All Questions
31 questions
1
vote
1
answer
59
views
Is disallowing all useragents sufficient to keep a site from being crawled by search engines?
We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
- 113
2
votes
2
answers
396
views
Will bots crawl my page generated by javascript
I use simple javascript to generate pages based on thousands of products stored in a JSON object.
The page URLs will be like /productList?keyword=XXX&page=x. The keyowrd is what uses input through ...
- 505
4
votes
2
answers
340
views
Do SEO analysis bots have any advantage in terms of website traffic?
I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as
AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/...
- 452
1
vote
1
answer
417
views
Can search engine fetch/index website that using ipv6 only?
I see some host providers offer cheap server but only support ipv6. I want to rent it, but I am still considering the SEO impact.
Can google and other search engine crawl and index it? If yes, will it ...
- 1,136
6
votes
2
answers
1k
views
How to keep crawler robots from indexing my page when there is a query string present in the URL?
I have a /products page.
In that page, users may use filters to select the products they want to see.
Those filters will work using query strings like: /products?brands=X,Y&size=1,2 and so on.
I ...
- 988
1
vote
1
answer
167
views
Why would Google Search Console report many times more indexed URLs than a Screaming Frog crawl finds?
I've just crawled my website in Screaming Frog and I have around 6K Internal URLs and 4K external URLs. Then I looked in GSC and saw that Google indexed 50K pages.
How is that possible?
0
votes
0
answers
17
views
In a sitemap, should I update the lastmod tag of a url based on the text content or html content? [duplicate]
Imagine I have this blogging / ecommerce website with 1000 posts / products. And I've built a sitemap for it, which is dynamically generated. Basically it's a list with a bunch of <url> and <...
- 988
1
vote
0
answers
80
views
Limit search engines to indexing just one domain when multiple domains share the same web server and document root
I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
- 11
-1
votes
3
answers
3k
views
How to remove bulk urls from google index
I have URLs indexed by Google and I want to remove them from indexing. I found that I can use bulk remove option by providing a CSV file including all indexed URLs to Google Search Console. I have ...
- 1
2
votes
1
answer
69
views
Do search engines crawl links with query strings?
If I were to have a url like https://example.com/books, that contained several links to urls such as:
https://example.com/books?id=foo
https://example.com/books?id=bar
https://example.com/books?id=...
- 227
3
votes
1
answer
1k
views
Is it good for SEO to change an article's slug when changing its title?
I have the following URL view:
example.com/article/{slug}-{id}
If I change the title of the article, then the slug will also change so that there is no disagreement.
Please note that the article ...
- 157
2
votes
2
answers
418
views
How many levels of sub-pages do Google bots crawl to when fetching and indexing linked pages in Google Search Console?
Let's say I submit a sitemap page that contains all the links I want Google to crawl, but instead of submitting a sitemap, I fetch the page and linked pages and let Google index them. How many levels ...
- 231
5
votes
2
answers
8k
views
Can Angular project be indexed in Google without Angular Universal?
I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
- 151
6
votes
2
answers
2k
views
Delivering a different version of website to Google robots (SSR)
I am working on an enterprise website project as an SEO manager.
They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render the ...
- 443
-1
votes
1
answer
72
views
Maximum time to cache HTTPS URL after HTTP to HTTPS migration?
We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
- 77
1
vote
0
answers
593
views
How to prevent CDN content URLs being indexed by Google [duplicate]
Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too).
So, even when a URL is disallowed in robots....
1
vote
1
answer
38
views
Re-Indexing Home Page - HTML/DOM Change
(Site on WordPress)
Currently, our company homepage is using images instead of divs to display our main products/solutions. As the SEO, I wanted to remove these images (which consist of mostly text) ...
- 419
2
votes
1
answer
35
views
Should article preview pages be crawled and indexed by search engines?
I have a page called "all articles" that loads previews of articles using AJAX. Because of that, the content won't be visible to search engines, so the page would be considered "poor content".
Since ...
- 217
2
votes
0
answers
112
views
Which one is more SEO friendly Dynamic pages or static pages for blogging website
I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
- 21
1
vote
2
answers
519
views
Should I submit both mobile and desktop domain names for indexing on non-responsive sites or is that bad for SEO?
I have a desktop version of my site using yourwebsite.com and a mobile version using m.yourwebsite.com.
Should I treat them as one site and only submit the desktop version along with its sitemap and ...
1
vote
1
answer
705
views
Cookie-based A/B testing on same URL - SEO impact?
we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much.
...
- 11
0
votes
1
answer
505
views
Crawl errors for pdf download links in my website
We are building a local business classified website which has about 6000 unique listingsunder huge number of categories and sub-categories.
For each listing there is a details page in which we have ...
- 142
1
vote
1
answer
74
views
Notify search engine crawlers about user generated content so that our rankings are not biased by keywords used by users
Is there a meta tag or a way to notify crawlers that a page is generated by the user?
To clarify the issue, We have a website already promoted by SEO and we want to add dynamic user generated content ...
- 113
2
votes
2
answers
225
views
Search engines that index your site despite a disallow in robots.txt
Recently, WordPress changed their options from "Block search engines" to "Discourage search engines"
So that leaves me wondering, that rewording must be that some search engines disregard your ...
- 131
0
votes
2
answers
265
views
Opening forum for search engine bots and SEO
We develop and sell components for other software developers through our website. The main products are ActiveX and WinFroms.NET grid controls. Some years ago we established a forum dedicated to our ...
- 423
2
votes
3
answers
125
views
Using Ajax #! for Google but site is not being crawled any more
Our pages used to be formulated using the ? string but since we changed to AJAX loading we now use #!, for example:
Old link: /post.php?idPost=5326
New link: /post.php#!idPost=5326
The snapshot you ...
- 23
4
votes
2
answers
548
views
Google Webmaster Tools reports fake 404 errors
I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...
- 381
1
vote
0
answers
24
views
Does google crawl css scripts to check of hidden content [duplicate]
Possible Duplicate:
hidden texts and page rank
Does googlebot crawl CSS scripts to determine if content is hidden or not? If so, is it advisable to hide or not to hide content to improve on the ...
5
votes
5
answers
262
views
What measures can be taken to make sure Google is aware of the existence of a newly created page?
Consider a website with a large number of pages. New pages are published regularly.
When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
- 159
2
votes
1
answer
294
views
Should I strategically use NoFollow on internal links to make our indexing crawl more efficient?
We have a well designed internal linking system that makes certain no page is more than four pages deep, and the indexing process has gone well for us.
However, in addition to the hierarchical page ...
- 1,328
0
votes
1
answer
66
views
Possible reasons RSS generated content won't index
I built a digg-like clone at fantasybookmark.com. Google crawled the site, but only indexed about 21 pages out of the 2,000 or so (and growing) pages on the site. We use rss feeds to find and ...
Bob Cavezza