Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

47 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8
votes
1answer
987 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
3
votes
0answers
44 views

Using display: none VS a SPA approach, when it comes to SEO and performance

Say I have a component like, // first approach (SPA) const Component = () => { const [show, toggle] = React.useReducer(show => !show, false) return ( <div> {show && &...
2
votes
0answers
44 views

Can many single page hits from a variety user agents from the same VPN with the same screen resolution be hurting my search rankings?

I am trying to clarify what is happening with the traffic on my site. Lately I am being flooded with many strange hits, mostly from US and they seem to only open 1 page or visit for few seconds. The ...
2
votes
0answers
727 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
2
votes
0answers
50 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
2
votes
0answers
188 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
2
votes
0answers
151 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
2
votes
0answers
283 views

What % can be regarded as “normal” “not viewed traffic” reported in AWStats?

Recently statistics for my site is about 50% "not viewed traffic". This seems much as many robots are blocked. What % of "not viewed traffic" can be regarded as "normal"? Is my site under attack when ...
2
votes
0answers
260 views

Google is adding “Archive” to the title of tag and category pages and not using the meta description

I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them? How to change them? Actually it looks so awkward! I have configured my ...
2
votes
0answers
239 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
2
votes
0answers
81 views

Which one is more SEO friendly Dynamic pages or static pages for blogging website

I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
2
votes
0answers
75 views

Browser's auto request detection with X-Purpose

3 to 4 months ago, I found out that safari/chrome/firefox all have feature that automatically send request with X-Purpose header (with preview value for Safari) to get live thumbnail or top sites ...
2
votes
0answers
154 views

different host path for web and android for app indexing

As per google's recommendation we use following meta tag for android application indexing: android-app://<package_name>/<scheme>/<host_path> Example : <link rel="alternate" ...
2
votes
0answers
756 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
2
votes
0answers
973 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
2
votes
1answer
460 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
0answers
17 views

Disallowing a handler in robots.txt while adding its dynamic URLs to the XML sitemap

I'm using ASP.NET webforms, and I have a page lets call it Subjects.aspx, I don't want crawlers to crawl that page but I want them to crawl the dynamic URLs that are powered by it. For example /...
1
vote
0answers
19 views

When setting up a campaign, Facebook is crawling my website using an Iphone user agent

Whenever I'm I'm setting up a campaing on Facebook, I get two visits from Facebook's auto crawlers. The first one I can identify as a robot and serve the SSR (HTML) version to it. The user agent is: ...
1
vote
0answers
48 views

How can I distinguish users from robots to use server side rendering with all bots?

How can I reliably distinguish users from robots? We have an ecommerce shop that goes through some heavy traffic peaks at some times. We would like to move some/all of the load during these peaks to ...
1
vote
0answers
23 views

Can I Prevent Adsense Loading For GoogleBot?

I have been working on lots of performance updates to my site and I have gotten pretty much every page type to load in under 2 seconds. Most single post pages are coming in at around 1.5 seconds.When ...
1
vote
0answers
11 views

Google Search Console errors (nofollow?)

I have a web site where some internal links take you to another pages of the same site, passing some parameters. Eg.: <a href="/BookMgm/PreparePayment?quantity=1&itemId=123">Book</a> ...
1
vote
0answers
28 views

Web page crawling by Google for SEO

On Our website we have an angular component which have multiple tabs and loads all the data (of all the tabs) at once. As per our new requirement to increase the 1st load speed, we will be loading ...
1
vote
0answers
36 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
1
vote
0answers
6 views

Should we apply “noreferrer” on “Read More” or “View All”?

I experienced years back a search engine crawler issue that Google started showing "Read More" as title of the page in it's search results. Because that page was linked through "read more" link on new ...
1
vote
0answers
18 views

How to deal with great number of URLs that will not be in anchor and should be known only for friendly crawlers?

I have a hypothetical page, that contains pages like: example.com/see/cat example.com/see/mouse example.com/see/<basically anything> I can have thousands of such pages. The number of pages ...
1
vote
0answers
53 views

When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?

I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl. At the ...
1
vote
0answers
15 views

301 redirects to a similar domain

My company has two similar but not identical sites. Similar page hierarchy and content but different brands. We want to shut one down and 301 the pages to what we feel is relevant on the better site. ...
1
vote
0answers
23 views

Facing a challenge with the “Fetch As Google” feature in my webmaster

I am facing a challenge with the "Fetch As Google" feature in my webmaster Steps to reproduce bug:- 1. Goto webmaster , click on "Crawl" -> "Fetch As Google" 2. Select "Crawl this URL and its direct ...
1
vote
0answers
206 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
37 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
1
vote
0answers
35 views

How do spambots find and submit to email opt-in the end of a funnel?

I have an opt-in page that's being nuked right now with spambots. A simple webpage that I use to allow subscribers into my email list. The only way to get to that page was through one of my tracking ...
1
vote
0answers
56 views

Dealing with Google spidering JavaScript incorrectly

Google spiders JavaScript now to look for possible URLs. This is mostly a good thing, but occasionally it gets it wrong. Is there any way to hide possible URLs in JavaScript from googlebot? For ...
1
vote
0answers
49 views

Google doesn't index my pages managed by session

My website has an architecture like this: em1 sub1 sub2 ... em2 sub1 sub2 ... em3 ... ... All elements "em" has the same "sub" associated. The difference only resides on the content in the "...
1
vote
0answers
60 views

Sitemap with an App using Codeigniter and jQuery Mobile?

We've got dynamic content that we want the google bot to search. Our app is running codeingiter and jQuery mobile. So, our site is basically 6 pages that are updated using XHR calls. The data is ...
1
vote
0answers
165 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
1
vote
0answers
38 views

Google not returning an international version of the page

I run a website that duplicates some content across international editions. These are differentiated by the country codes e.g. /uk/folder/article1/ /au/folder/article1/ The UK version is considered ...
1
vote
0answers
139 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) /Guitars/Children%C3%C6%92%E2%E2%82%AC%C5%A1%...
0
votes
0answers
29 views

Google search says No information is available for my GitHub page

Today I search my GitHub page in Google Chrome and Firefox (Through Google search) and for the first time, it says no information is available (it says 2019- GitHub under that it says no information ...
0
votes
0answers
15 views

Why Google Cache Showing 404 on my React Base Webpages?

My web page is client-side rendered with React. On hitting the cache, it is showing 404 instead of a rendered page. Example Screenshot When I inspect the Page URL on Search Console, Its parsing and ...
0
votes
0answers
51 views

Is it SEO problematic to create footnotes with frontend JavaScript?

I have an RTL (Hebrew) MediaWiki 1.34.0 website in which I am having a serious problem creating footnotes for articles with the core Extension:Cite because its de facto lack of support in Hebrew, as ...
0
votes
0answers
9 views

How can I created tracking redirects that are neither crawled nor indexed?

I have tracking redirects on my site. I use server-side tracking on them. They are used for things like: Advertisements Social media sharing The URLs are of the forms: /ad-click?id=12345 /...
0
votes
0answers
82 views

Using prerender.io for SEO on my AngularJS app - Dynamic and static web pages

I have a website that was built with AngularJS. I want the website to be crawled by search engines (with the dynamic content). I've tried to use Google services to do the crawl (from a sitemap.xml ...
0
votes
0answers
139 views

Invalid traffic coming from Sogou

I have not indexed my website in Sogou. I have only very less visitors. But i am getting traffic from Sogou daily. The search keyword is "site:mywebsitename" and is from China. Just visiting main page....
0
votes
0answers
61 views

Google is marking as duplicate my unique URLs powered by Angular Universal

I have a problem in the search console, and google is marking my unique urls as duplicate The application is build with Angular Universal and is rendered in the server Note: This site is only ...
0
votes
0answers
93 views

Baidu requesting malformed URLs - my fault or theirs?

I often get lines like this in my error log: 2015/11/11 12:15:58 [error] 10927#0: *167859 open() "/var/www/sites/mysite/http/https:/mysub.domain.com/filename.jpg" failed (2: No such file or directory)...
0
votes
0answers
25 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...
-1
votes
1answer
13 views

Why is Azure pinging my site?

I have noticed more and more, traffic from IPs assigned to Microsoft Azure/Microsoft Corporation hitting my WordPress site many times a day. It's direct traffic to just the home and privacy policy ...