Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
1
vote
1answer
22 views

What is “ineedthispage”? Where is it coming from? It is killing the server

I am seeing the following in my "access_log": ip.address.output.here - - [12/Dec/2019:15:17:07 +0000] "GET /pathto/article/15104133/?ineedthispage=yes&ineedthispage=yes&ineedthispage=yes&...
4
votes
0answers
146 views

isAccessibleForFree and paywalled content delivered to Googlebots

I've got a site with some paywalled articles. We want to ensure that Googlebots can index the paywalled content, but we don't want to be accused of cloaking. I've got two types of articles: Premium ...
0
votes
0answers
24 views

How does Googlebot crawl pages having cookiepro integration

Google bot is appending the link present in cookiepro banner on my main product-details page and hence giving 404. On the webapp the link is going to the proper URL, but I can see a lot of URLS in my ...
1
vote
0answers
19 views

What means the “average Bytes” metric in logfiles?

I'm currently studying the logfiles - namely those from Googlebot access. I realized, that the average Bytes metric is in the majority of cases (urls) 10-20% of the page size. Many questions were ...
2
votes
1answer
53 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
3
votes
1answer
32 views

What headers used in request by google bot?

What headers used in request by google bot? Must be user agent, but what else? I'm interested in cache control headers, does bot try to get fresh page, but not from cache? The related question: Do ...
2
votes
0answers
15 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
2
votes
1answer
36 views

Is a child element set to visible while its parent set to hidden bad for SEO?

To modify the style of a component within a framework, I need to set the visibility of the parent container to hidden and the visibility of its child that contains the content to visible. Is that bad ...
1
vote
2answers
167 views

What are the SEO implications of an established website shifting from server-side rendering to client side?

I run a decently established website (20k monthly users, already indexed by Googlebot, etc) that has been built on Django a year and a half ago. The vast majority of traffic on my site comes from the ...
1
vote
0answers
109 views

Googlebot intermittently fails to download ressources from AWS CloudFront CDN

I have a weird issue and after a few weeks of investigation I think I may have found the cause : Cloudfront & Googlebot. Context : I launched a French website that compares and combines all ...
2
votes
2answers
259 views

Can I request a site removal from Google and then re-index a new site on that domain?

. Basically I have replaced my site completely but upon searching Google, it returned some old pages that didn't exist anymore, so I have used the Google index -> Remove URLs from Google Webmaster ...
2
votes
1answer
35 views

Can a JS redirect to a not found page work in place of a real 404?

I need assistance with some issues on a Big Commerce site. To preface, it is not my site and the IT team has a complex technology stack being utilized to build the site. The issue is that adding an ...
0
votes
0answers
12 views

In a sitemap, should I update the lastmod tag of a url based on the text content or html content? [duplicate]

Imagine I have this blogging / ecommerce website with 1000 posts / products. And I've built a sitemap for it, which is dynamically generated. Basically it's a list with a bunch of <url> and <...
1
vote
1answer
24 views

If I make changes in the template I use for a product page, do I need to update all <lastmod> tags for all product pages in my sitemap.xml?

I'm currently building a sitemap for my web app. Note: my app is a single page app built with React and Firebase. Basically I wan to index blogPost and product pages. So I intend to generate my ...
3
votes
1answer
77 views

Flag my website as a single page application?

Is it possible to flag my website as a SPA? The reason I want to do this is because on Google Search Console I see my links indexed without canonical links, but if I try for a live version the ...
1
vote
1answer
24 views

Google live test can't load AngularJS XHR resources due to CORS policy

I implemented my website using AngularJs, It works correctly on any browsers and locations I've tested. Although they claimed that Google Search Engine sees exactly what modern browsers display, my ...
1
vote
0answers
15 views

Can I Prevent Adsense Loading For GoogleBot?

I have been working on lots of performance updates to my site and I have gotten pretty much every page type to load in under 2 seconds. Most single post pages are coming in at around 1.5 seconds.When ...
3
votes
1answer
56 views

Would Google accept a text sitemap if the URL had a .php extension?

Google accepts a sitemap as .txt file with a list of URLs separated each by a newline. Would Google also allow the same text file format with a .php? Ie.mysitemap.php instead of mysitemap.txt?
2
votes
1answer
309 views

Crawled page's screenshot is blank in the Google Search Console

I set up a SPA page through Angular, and achieved the effect of Server Side Render through Rendertron, In the facebook debug tool, I could get my dynamically generated Meta data, but the preview of ...
2
votes
2answers
38 views

Will lengthening our URLs change how Googlebot crawls our site?

Currently we are using URL fetched from main title of article (55 characters). We are planning to increase title of article up to 100 characters which tends to increase length of URL to 100+. Will ...
2
votes
1answer
17 views

Page cannot be reached in Google Rich snippet test

When i test my webpage in google rich test, it say page cannot be reached. My website is on shopify so i guess there should be no issue from server side. I need your help please... I cannot figure out ...
1
vote
0answers
25 views

Can alternate URLs for hreflinks be in different XML sitemap files when using <xml:link rel=“alternate”

Regarding the XML sitemaps (that use alternative languages), in this page it is specified the following: Common Mistakes Here are the most common mistakes with hreflang usage: Missing ...
2
votes
1answer
23 views

Can robots.txt be used to prevent bots from seeing lazily loaded content?

Let's say that googlebot is scraping https://example.com/page. example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page /page lazy loads content using /...
1
vote
0answers
23 views

Is there a way to use js templating with wordpress without impacting seo or ux?

I have one JSON file that contains the text content for my hundreds of Wordpress posts. I am using templating, mustache.js, to insert the text content and render the posts. This saves me a lot of time,...
1
vote
1answer
32 views

Google Crawl Stats: What's included in “time spent downloading a page” metric?

Background I'm the owner of a quite large site (millions of pages). I'm trying to optimize the site for "time spent downloading a page"-metric in google search console. Currently my stats in Search ...
7
votes
1answer
854 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
2
votes
1answer
19 views

Is it OK for SEO to remove 10 of 16 translations of our site in subdirectories?

Our website has 16 different languages, each in a subdirectory. hreflang is not implemented. We are now trying to redesign the website and we want 6 languages to be active. The rest of the languages ...
7
votes
2answers
4k views

Googlebot crawls my page too often

I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot). But I have 1 article that have ...
1
vote
1answer
20 views

How does a live test in Google search console affect ranking if it doesn't pick up all the images and other content?

In google search console I conducted live test URL for my site's home page. However, the screenshot of the live test does not look like my home page. There are not many images and text. My home page ...
15
votes
3answers
9k views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
1
vote
0answers
38 views

404 errors as a result of …/false urls

I have been reviewing 404 error logs on a WooCommerce site I have been working on (logs held within the website server) and there are a lot of requests to product pages which have /false on the end of ...
3
votes
2answers
56 views

Will I get penalized by Google for cloaking if I redirect mobile users to a sub domain?

I have a website https://example.com which is responsive but it came a time where it displays too much text. So I decided to create a progressive web application version and would like to deploy it ...
3
votes
0answers
1k views

URL will be indexed only if certain conditions are met

I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
2
votes
2answers
26 views

How does google treat h1-3's that are styled to be smaller than a lower class of heading?

For example: You have you h1 <h1>Epic SEO Keywords</h1> then right below <h2 class="Massive text">Enjoy the future of you, visualise your dreams</h2> Will Google render ...
2
votes
1answer
211 views

Google AdWords ads are disapproved because “destination not working”

We have been having an issue setting up adwords for one of our clients and despite multiple phone calls to Google we cannot get an answer out of them as to why our ads are being disapproved. ...
3
votes
1answer
92 views

How to change Google crawl rate in new Google search console?

Google has now completely removed the old search console (https://searchengineland.com/the-old-google-search-console-is-no-longer-available-321650) where you used to be able to request a custom crawl ...
3
votes
2answers
161 views

Can Google crawl dynamic structured data added in a window.onload() event?

I inject some JSON-LD into a page dynamically via Javascript, when I test the page using Google's Structured Data Testing Tool, the expected output appears and I can see the Product Element is ...
1
vote
1answer
47 views

Shrinking site's size by removing pages from sitemap and from internal links

We are an eCommerce site with about 50,000 pages. We decided to focus on about 2,000 so google will crawl our site better. We planning to remove 48,000 pages from the sitemaps and from the site links, ...
1
vote
1answer
338 views

Security Issue - Harmful content

I am trying to solve an issue with my website already for some time but without any luck. Google has detected somewhere in my website harmful content (Malware). But the problem is, I could not find ...
2
votes
0answers
23 views

URL crawlable in old Google search console, but blocked in new

Google stopped crawling images for our site, when we moved images to cdn subdomain. Old search console states image is crawlable, but new one states it is blocked by robots.txt (for same link). our ...
3
votes
1answer
51 views

Can I hide ads for Googlebot so it loads my site faster?

I have ads which load sometimes slower, I want to hide those ads for Googlebot. I plan to hide that ad code for googlebot only and show them only to users. Do I risk a Google penalty if I hide ads, ...
2
votes
1answer
3k views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
1
vote
0answers
43 views

Crawl anomaly caused by googlebot crawling secure_link mp4 files with md5 and expires parameters

Googlebot continues to crawl mp4 files in the format www.example.com/video/12345.mp4?md5=2029zanlkdain312on&expires=1523327783 which has obviously expired (only valid timestamp+30minutes, after ...
3
votes
2answers
155 views

Googlebot keeps on crawling stale/nonexistent(410) resources and then shows a crawl anomaly followed by de-indexing pages

We have been experiencing constant crawl anomalies caused by googlebot continued crawling of 'rnd.js?asdfasdfasfs3423' (hash is random on each pageload). It has been 3 months since we removed rnd.js ...
0
votes
0answers
46 views

Referencement issue with irrelevant suffix

I have a problem concerning the "coverage" on Google Search Console. The website of the company I'm working for has weird errors on this page. Google is telling that it had issues referencing pages ...
3
votes
0answers
54 views

Pantheon Dev Site Trapped in Google Index (with injected robots.txt)

We have a bizarre scenario I'd like opinions on. A site at example.com is being built on pantheon, using their example.pantheon.io staging / dev URL*. Pantheon's development platform injects a ...
2
votes
0answers
376 views

Disqus comments cause “clickable elements too close together” in Google Search Console mobile usabality report

I just got an email from Google that they found some problems on my site: clickable elements too close together Content wider than screen I fetched problematic URLs as Googlebot and noticed that it ...
1
vote
1answer
61 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...
11
votes
1answer
9k views

link rel=“alternate”, multiple languages and canonical urls

Context: We've got a website which is available in multiple language versions. However, the content is the same. Each translated version of the site is available on a distinct subdomain. On some ...
0
votes
0answers
9 views

Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]

Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...