Skip to main content

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

70 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4 votes
0 answers
233 views

Using display: none VS a SPA approach, when it comes to SEO and performance

Say I have a component like, // first approach (SPA) const Component = () => { const [show, toggle] = React.useReducer(show => !show, false) return ( <div> {show && &...
legoMyEgo's user avatar
3 votes
1 answer
376 views

Why Googlebot is crawling only one page?

I am trying to understand why Google bot is crawling only one page only on my website. At the Google console I registered a small XML sitemap with 9 URLs. Only one URL I marked as a high priority (0.8)...
bakunet's user avatar
  • 227
3 votes
0 answers
630 views

Prevent search engines from crawling and indexing all domains and subdomains set up under an Apache server

My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (...
FiddlingAway's user avatar
3 votes
0 answers
84 views

Do HTML elements with the boolean attribute hidden get picked up by search engines?

I recently learned of the existence of the hidden attribute. It basically hides elements that are not relevant in the current state of the webpage (e.g., a user that is not logged in): <p hidden>...
Juan Marco's user avatar
3 votes
0 answers
64 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
Kannan's user avatar
  • 2,343
3 votes
0 answers
283 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
Siamak's user avatar
  • 243
3 votes
0 answers
1k views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
riffraff's user avatar
  • 165
2 votes
0 answers
25 views

How to manage internal links using short alternate URLs when the canonical URLs are complex to generate?

I am working on an existing e-commerce website where the canonical URL for product pages are long and complex to generate, hyphenated, and ending by the product id: https://example.com/brand-model-and-...
OuzoPower's user avatar
  • 357
2 votes
0 answers
42 views

One day after putting my site online and registering it with Google I got spam mail. How?

I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
Frank Conijn - Support Ukraine's user avatar
2 votes
0 answers
30 views

Googlebot increased the number of requests for a certain URL even though it has no links

As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
Lalikman's user avatar
2 votes
0 answers
336 views

What is correct way to stop query indexing using Robots.txt?

How to stop indexing search result using Robots.txt? hello. Recently, I'm having an issue with indexed pages in Google Search Console growing 10x. These pages are mostly search query (/?s=) and ...
cheonmu's user avatar
  • 21
2 votes
0 answers
42 views

Disallow crawling of all links to a domain, except only specific URLs with parameters

I have a domain like example.com under which everything is blocked and links to an SPA. Like example.com, and we've disabled everything like so: User-agent: * Disallow: / Now this SPA uses parameters ...
Florian Monfort's user avatar
2 votes
0 answers
262 views

How can I let Google crawl my site for Programmable Search Engine (custom site search), but not have it show in public search results?

The answer for my question might be inherent in the question itself- that if I want Google to crawl my site for a custom site search, such as through Programmable Search Engine, that it is those same ...
Bobby_Cheh's user avatar
2 votes
0 answers
282 views

Does Googlebot respect the cache-control header?

Does Googlebot respect the cache-control header? By setting an appropriate cache-control header, can one improve the crawl efficiency?
Kannan's user avatar
  • 2,343
2 votes
0 answers
31 views

Could allowing our images to be indexed hurt our web search rankings?

We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
Mindastic's user avatar
  • 161
2 votes
0 answers
416 views

Restrict Google from indexing subdomain from Azure CDN services

How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
ankur lakhanpal's user avatar
2 votes
0 answers
120 views

When setting up a campaign, Facebook is crawling my website using an Iphone user agent

Whenever I'm I'm setting up a campaing on Facebook, I get two visits from Facebook's auto crawlers. The first one I can identify as a robot and serve the SSR (HTML) version to it. The user agent is: ...
cbdeveloper's user avatar
2 votes
0 answers
48 views

Can many single page hits from a variety user agents from the same VPN with the same screen resolution be hurting my search rankings?

I am trying to clarify what is happening with the traffic on my site. Lately I am being flooded with many strange hits, mostly from US and they seem to only open 1 page or visit for few seconds. The ...
adrianTNT's user avatar
  • 203
2 votes
0 answers
750 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
A User's user avatar
  • 141
2 votes
0 answers
277 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
Vecta's user avatar
  • 369
2 votes
0 answers
538 views

What % can be regarded as "normal" "not viewed traffic" reported in AWStats?

Recently statistics for my site is about 50% "not viewed traffic". This seems much as many robots are blocked. What % of "not viewed traffic" can be regarded as "normal"? Is my site under attack when ...
snayob's user avatar
  • 71
2 votes
0 answers
352 views

Google is adding "Archive" to the title of tag and category pages and not using the meta description

I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them? How to change them? Actually it looks so awkward! I have configured my ...
Guruprasad's user avatar
2 votes
0 answers
391 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
slindsey3000's user avatar
2 votes
0 answers
85 views

Which one is more SEO friendly Dynamic pages or static pages for blogging website

I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
Aayush Goyal's user avatar
2 votes
0 answers
121 views

Browser's auto request detection with X-Purpose

3 to 4 months ago, I found out that safari/chrome/firefox all have feature that automatically send request with X-Purpose header (with preview value for Safari) to get live thumbnail or top sites ...
vutran's user avatar
  • 121
2 votes
0 answers
160 views

different host path for web and android for app indexing

As per google's recommendation we use following meta tag for android application indexing: android-app://<package_name>/<scheme>/<host_path> Example : <link rel="alternate" ...
abhinsit's user avatar
  • 1,714
2 votes
0 answers
762 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
Karl Cassar's user avatar
2 votes
1 answer
469 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
VB_'s user avatar
  • 119
1 vote
0 answers
257 views

Should Feed/RSS Be Blocked from Robots.txt?

Should Feed/RSS be blocked via robots.txt? Disallow: /*feed* Disallow: /*rss* If yes, is this the correct way to apply? If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
gfo95's user avatar
  • 41
1 vote
0 answers
128 views

Connecting to MySQL database hosted on CPanel

I have followed all of the usual steps in setting up a pipeline but cannot seem to get the database itself to connect. I am confident in the correct database name, username passowrd etc but am ...
Anton Gomes's user avatar
1 vote
0 answers
25 views

Sitemapindex containing nearly identical sitemaps? - is that a problem

I am currently in the process of migrating content from one domain to another domain, In this case I have some difficulty on how I should map the sitemap. I currently have a sitemapindex file, which ...
Carlton Banks's user avatar
1 vote
0 answers
214 views

Google Search Console detects issue with noindex pages excluded from sitemap, why? How to resolve?

Google emails me this: Message type: [WNC-10030322] Search Console Page indexing issues detected in [DOMAIN] To the owner of [DOMAIN]: Search Console has identified that your site is affected by 1 ...
2540625's user avatar
  • 141
1 vote
1 answer
71 views

How do password manager apps detect input tags?

How? Does it scrape the website for adjacent input tags, one with text and one with password? No matter which website I visit, it accurately prompts a dropdown (Touch ID on Safari) on the username ...
Ninad's user avatar
  • 11
1 vote
0 answers
46 views

Exposing content from client-side headless CMS API to web crawlers

This is something I've been struggling to wrap my head around for the past few days. Tried to keep the rambling to a minimum but concise questions are at the bottom. As I understand it, utilizing JS ...
PrestigiousAd9972's user avatar
1 vote
0 answers
407 views

Do Baidu and/or Yandex publish the current IP ranges used by their site indexing bots?

I need to identify bots operated by search engine operators to index sites. I've looked at an existing question on this matter and the answers given there are too simplistic. The User-Agent header is ...
reinierpost's user avatar
1 vote
0 answers
42 views

Quickest way to remove many dynamic pages with duplicate content due to a spider trap

I have an angular JS site that is reliant on google rendering the pages. I created a spider trap and have about 9000 pages in the index which most are duplicate content. Each page that is currently ...
Malooii's user avatar
  • 51
1 vote
0 answers
24 views

Can you specify the max-snippet length in a X-robots-tag header?

The specs by Google does not address this issue. So, I would like to know what is the x-robots-tag equivalent for: <meta name="googlebot" content="max-snippet:20"/> Is it ...
Jesske's user avatar
  • 11
1 vote
0 answers
39 views

Disallowing a handler in robots.txt while adding its dynamic URLs to the XML sitemap

I'm using ASP.NET webforms, and I have a page lets call it Subjects.aspx, I don't want crawlers to crawl that page but I want them to crawl the dynamic URLs that are powered by it. For example /...
Ali Kleit's user avatar
  • 123
1 vote
0 answers
71 views

How can I distinguish users from robots to use server side rendering with all bots?

How can I reliably distinguish users from robots? We have an ecommerce shop that goes through some heavy traffic peaks at some times. We would like to move some/all of the load during these peaks to ...
Qwerty's user avatar
  • 111
1 vote
0 answers
15 views

Google Search Console errors (nofollow?)

I have a web site where some internal links take you to another pages of the same site, passing some parameters. Eg.: <a href="/BookMgm/PreparePayment?quantity=1&itemId=123">Book</a> ...
Gianpiero's user avatar
  • 111
1 vote
0 answers
38 views

Web page crawling by Google for SEO

On Our website we have an angular component which have multiple tabs and loads all the data (of all the tabs) at once. As per our new requirement to increase the 1st load speed, we will be loading ...
Saket Kumar's user avatar
1 vote
0 answers
67 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
codaamok's user avatar
1 vote
0 answers
10 views

Should we apply "noreferrer" on "Read More" or "View All"?

I experienced years back a search engine crawler issue that Google started showing "Read More" as title of the page in it's search results. Because that page was linked through "read more" link on new ...
Puneetpal singh's user avatar
1 vote
0 answers
20 views

How to deal with great number of URLs that will not be in anchor and should be known only for friendly crawlers?

I have a hypothetical page, that contains pages like: example.com/see/cat example.com/see/mouse example.com/see/<basically anything> I can have thousands of such pages. The number of pages ...
matousc's user avatar
  • 131
1 vote
0 answers
74 views

When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?

I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl. At the ...
Becky's user avatar
  • 49
1 vote
0 answers
19 views

301 redirects to a similar domain

My company has two similar but not identical sites. Similar page hierarchy and content but different brands. We want to shut one down and 301 the pages to what we feel is relevant on the better site. ...
Frank's user avatar
  • 11
1 vote
0 answers
25 views

Facing a challenge with the "Fetch As Google" feature in my webmaster

I am facing a challenge with the "Fetch As Google" feature in my webmaster Steps to reproduce bug:- 1. Goto webmaster , click on "Crawl" -> "Fetch As Google" 2. Select "Crawl this URL and its direct ...
Kavita Ahuja's user avatar
1 vote
0 answers
350 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
Wazery's user avatar
  • 111
1 vote
0 answers
41 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
Nish's user avatar
  • 11
1 vote
0 answers
40 views

How do spambots find and submit to email opt-in the end of a funnel?

I have an opt-in page that's being nuked right now with spambots. A simple webpage that I use to allow subscribers into my email list. The only way to get to that page was through one of my tracking ...
av17's user avatar
  • 11