All Questions
Tagged with crawling or web-crawlers
73 questions with no upvoted or accepted answers
4
votes
0
answers
257
views
Using display: none VS a SPA approach, when it comes to SEO and performance
Say I have a component like,
// first approach (SPA)
const Component = () => {
const [show, toggle] = React.useReducer(show => !show, false)
return (
<div>
{show && &...
- 43
3
votes
1
answer
443
views
Why Googlebot is crawling only one page?
I am trying to understand why Google bot is crawling only one page only on my website.
At the Google console I registered a small XML sitemap with 9 URLs. Only one URL I marked as a high priority (0.8)...
- 227
3
votes
0
answers
730
views
Prevent search engines from crawling and indexing all domains and subdomains set up under an Apache server
My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (...
- 195
3
votes
0
answers
91
views
Do HTML elements with the boolean attribute hidden get picked up by search engines?
I recently learned of the existence of the hidden attribute. It basically hides elements that are not relevant in the current state of the webpage (e.g., a user that is not logged in):
<p hidden>...
- 133
3
votes
0
answers
67
views
Should a page with adsense be viewed at least once for adsense crawler to crawl?
I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
- 2,343
3
votes
0
answers
293
views
Does Googlebot cache JavaScript files between two pages if they expire in one year?
My question is about javascript websites.
Suppose GoogleBot has crawled example.com/index.html?id=1
Does it request all the javascript files again when crawling example.com/index.html?id=2?
index....
- 243
3
votes
0
answers
1k
views
Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?
Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e.
REQUEST_PATH : /foo/bar.json
HTTP_HOST ...
- 165
2
votes
0
answers
26
views
How to manage internal links using short alternate URLs when the canonical URLs are complex to generate?
I am working on an existing e-commerce website where the canonical URL for product pages are long and complex to generate, hyphenated, and ending by the product id: https://example.com/brand-model-and-...
- 357
2
votes
0
answers
43
views
One day after putting my site online and registering it with Google I got spam mail. How?
I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
2
votes
0
answers
30
views
Googlebot increased the number of requests for a certain URL even though it has no links
As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
- 51
2
votes
0
answers
433
views
What is correct way to stop query indexing using Robots.txt?
How to stop indexing search result using Robots.txt?
hello.
Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.
These pages are mostly search query (/?s=) and ...
- 21
2
votes
0
answers
61
views
Disallow crawling of all links to a domain, except only specific URLs with parameters
I have a domain like example.com under which everything is blocked and links to an SPA.
Like example.com, and we've disabled everything like so:
User-agent: *
Disallow: /
Now this SPA uses parameters ...
- 121
2
votes
0
answers
281
views
How can I let Google crawl my site for Programmable Search Engine (custom site search), but not have it show in public search results?
The answer for my question might be inherent in the question itself- that if I want Google to crawl my site for a custom site search, such as through Programmable Search Engine, that it is those same ...
- 121
2
votes
0
answers
320
views
Does Googlebot respect the cache-control header?
Does Googlebot respect the cache-control header? By setting an appropriate cache-control header, can one improve the crawl efficiency?
- 2,343
2
votes
0
answers
35
views
Could allowing our images to be indexed hurt our web search rankings?
We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
- 161
2
votes
0
answers
463
views
Restrict Google from indexing subdomain from Azure CDN services
How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
2
votes
0
answers
190
views
When setting up a campaign, Facebook is crawling my website using an Iphone user agent
Whenever I'm I'm setting up a campaing on Facebook, I get two visits from Facebook's auto crawlers.
The first one I can identify as a robot and serve the SSR (HTML) version to it.
The user agent is:
...
- 988
2
votes
0
answers
50
views
Can many single page hits from a variety user agents from the same VPN with the same screen resolution be hurting my search rankings?
I am trying to clarify what is happening with the traffic on my site. Lately I am being flooded with many strange hits, mostly from US and they seem to only open 1 page or visit for few seconds.
The ...
- 203
2
votes
0
answers
751
views
What is user agent ;__CT_JOB_ID__:?
I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format:
;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e;
Each visitor has a unique IP. So ...
- 141
2
votes
0
answers
318
views
Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages
I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
- 369
2
votes
0
answers
568
views
What % can be regarded as "normal" "not viewed traffic" reported in AWStats?
Recently statistics for my site is about 50% "not viewed traffic". This seems much as many robots are blocked. What % of "not viewed traffic" can be regarded as "normal"?
Is my site under attack when ...
- 71
2
votes
0
answers
374
views
Google is adding "Archive" to the title of tag and category pages and not using the meta description
I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them?
How to change them? Actually it looks so awkward!
I have configured my ...
- 63
2
votes
0
answers
418
views
Set bot crawl delay 10 seconds EXCEPT Googlebot?
Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace?
I like being indexed by all these bots but with Yandex and Baidu it is ...
- 131
2
votes
0
answers
112
views
Which one is more SEO friendly Dynamic pages or static pages for blogging website
I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
- 21
2
votes
0
answers
126
views
Browser's auto request detection with X-Purpose
3 to 4 months ago, I found out that safari/chrome/firefox all have feature that automatically send request with X-Purpose header (with preview value for Safari) to get live thumbnail or top sites ...
- 121
2
votes
0
answers
164
views
different host path for web and android for app indexing
As per google's recommendation we use following meta tag for android application indexing:
android-app://<package_name>/<scheme>/<host_path>
Example :
<link rel="alternate" ...
- 1,714
2
votes
0
answers
765
views
Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output
I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
- 153
2
votes
1
answer
475
views
Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?
Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState?
I have found contradictory meanings. Can you support or refute one of ...
- 119
1
vote
2
answers
25
views
Domain change, 301 Redirects, Site Map & Indexing
I changed my old domain name to a shorter domain name. I set up 301 redirects and uploaded the XML site map of the new website to Google Search Console. Google indexed the new domain, but I still see ...
- 11
1
vote
0
answers
560
views
Should Feed/RSS Be Blocked from Robots.txt?
Should Feed/RSS be blocked via robots.txt?
Disallow: /*feed*
Disallow: /*rss*
If yes, is this the correct way to apply?
If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
- 41
1
vote
0
answers
183
views
Connecting to MySQL database hosted on CPanel
I have followed all of the usual steps in setting up a pipeline but cannot seem to get the database itself to connect. I am confident in the correct database name, username passowrd etc but am ...
- 11
1
vote
0
answers
26
views
Sitemapindex containing nearly identical sitemaps? - is that a problem
I am currently in the process of migrating content from one domain to another domain,
In this case I have some difficulty on how I should map the sitemap.
I currently have a sitemapindex file, which ...
- 111
1
vote
0
answers
290
views
Google Search Console detects issue with noindex pages excluded from sitemap, why? How to resolve?
Google emails me this:
Message type: [WNC-10030322]
Search Console
Page indexing issues detected in [DOMAIN]
To the owner of [DOMAIN]:
Search Console has identified that your site is affected by 1 ...
- 113
1
vote
1
answer
84
views
How do password manager apps detect input tags?
How? Does it scrape the website for adjacent input tags, one with text and one with password? No matter which website I visit, it accurately prompts a dropdown (Touch ID on Safari) on the username ...
- 11
1
vote
0
answers
56
views
Exposing content from client-side headless CMS API to web crawlers
This is something I've been struggling to wrap my head around for the past few days. Tried to keep the rambling to a minimum but concise questions are at the bottom.
As I understand it, utilizing JS ...
1
vote
0
answers
528
views
Do Baidu and/or Yandex publish the current IP ranges used by their site indexing bots?
I need to identify bots operated by search engine operators to index sites.
I've looked at an existing question on this matter and the answers given there are too simplistic.
The User-Agent header is ...
- 125
1
vote
0
answers
48
views
Quickest way to remove many dynamic pages with duplicate content due to a spider trap
I have an angular JS site that is reliant on google rendering the pages.
I created a spider trap and have about 9000 pages in the index which most are duplicate content.
Each page that is currently ...
- 51
1
vote
0
answers
26
views
Can you specify the max-snippet length in a X-robots-tag header?
The specs by Google does not address this issue. So, I would like to know what is the x-robots-tag equivalent for:
<meta name="googlebot" content="max-snippet:20"/>
Is it ...
- 11
1
vote
0
answers
44
views
Disallowing a handler in robots.txt while adding its dynamic URLs to the XML sitemap
I'm using ASP.NET webforms, and I have a page lets call it Subjects.aspx, I don't want crawlers to crawl that page but I want them to crawl the dynamic URLs that are powered by it. For example /...
- 123
1
vote
0
answers
78
views
How can I distinguish users from robots to use server side rendering with all bots?
How can I reliably distinguish users from robots?
We have an ecommerce shop that goes through some heavy traffic peaks at some times. We would like to move some/all of the load during these peaks to ...
- 111
1
vote
0
answers
16
views
Google Search Console errors (nofollow?)
I have a web site where some internal links take you to another pages of the same site, passing some parameters.
Eg.:
<a href="/BookMgm/PreparePayment?quantity=1&itemId=123">Book</a>
...
- 111
1
vote
0
answers
38
views
Web page crawling by Google for SEO
On Our website we have an angular component which have multiple tabs and loads all the data (of all the tabs) at once.
As per our new requirement to increase the 1st load speed, we will be loading ...
- 11
1
vote
0
answers
80
views
Limit search engines to indexing just one domain when multiple domains share the same web server and document root
I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
- 11
1
vote
0
answers
10
views
Should we apply "noreferrer" on "Read More" or "View All"?
I experienced years back a search engine crawler issue that Google started showing "Read More" as title of the page in it's search results. Because that page was linked through "read more" link on new ...
1
vote
0
answers
23
views
How to deal with great number of URLs that will not be in anchor and should be known only for friendly crawlers?
I have a hypothetical page, that contains pages like:
example.com/see/cat
example.com/see/mouse
example.com/see/<basically anything>
I can have thousands of such pages. The number of pages ...
- 131
1
vote
0
answers
81
views
When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?
I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl.
At the ...
- 49
1
vote
0
answers
19
views
301 redirects to a similar domain
My company has two similar but not identical sites. Similar page hierarchy and content but different brands. We want to shut one down and 301 the pages to what we feel is relevant on the better site.
...
- 11
1
vote
0
answers
26
views
Facing a challenge with the "Fetch As Google" feature in my webmaster
I am facing a challenge with the "Fetch As Google" feature in my webmaster
Steps to reproduce bug:-
1. Goto webmaster , click on "Crawl" -> "Fetch As Google"
2. Select "Crawl this URL and its direct ...
- 11
1
vote
0
answers
371
views
Stop getting traffic from Googlebot on high server load
In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing.
So what I thought of; is depending on the ...
- 111
1
vote
0
answers
43
views
Product carousel on my site is showing up blank in Google's fetch and render for mobile
Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
- 11