Questions tagged [googlebot]
Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.
734
questions
1
vote
1answer
22 views
What is “ineedthispage”? Where is it coming from? It is killing the server
I am seeing the following in my "access_log":
ip.address.output.here - - [12/Dec/2019:15:17:07 +0000] "GET /pathto/article/15104133/?ineedthispage=yes&ineedthispage=yes&ineedthispage=yes&...
4
votes
0answers
146 views
isAccessibleForFree and paywalled content delivered to Googlebots
I've got a site with some paywalled articles. We want to ensure that Googlebots can index the paywalled content, but we don't want to be accused of cloaking.
I've got two types of articles:
Premium ...
0
votes
0answers
24 views
How does Googlebot crawl pages having cookiepro integration
Google bot is appending the link present in cookiepro banner on my main product-details page and hence giving 404.
On the webapp the link is going to the proper URL, but I can see a lot of URLS in my ...
1
vote
0answers
19 views
What means the “average Bytes” metric in logfiles?
I'm currently studying the logfiles - namely those from Googlebot access. I realized, that the average Bytes metric is in the majority of cases (urls) 10-20% of the page size. Many questions were ...
2
votes
1answer
53 views
Allow all bots except Googlebot which should only be able to crawl the home page
I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else.
I'd prefer to do this in robots.txt. How can I ...
3
votes
1answer
32 views
What headers used in request by google bot?
What headers used in request by google bot? Must be user agent, but what else? I'm interested in cache control headers, does bot try to get fresh page, but not from cache?
The related question: Do ...
2
votes
0answers
15 views
Googlebot-Image incorrectly crossing domain URLs
My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com).
We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
2
votes
1answer
36 views
Is a child element set to visible while its parent set to hidden bad for SEO?
To modify the style of a component within a framework, I need to set the visibility of the parent container to hidden and the visibility of its child that contains the content to visible.
Is that bad ...
1
vote
2answers
167 views
What are the SEO implications of an established website shifting from server-side rendering to client side?
I run a decently established website (20k monthly users, already indexed by Googlebot, etc) that has been built on Django a year and a half ago. The vast majority of traffic on my site comes from the ...
1
vote
0answers
109 views
Googlebot intermittently fails to download ressources from AWS CloudFront CDN
I have a weird issue and after a few weeks of investigation I think I may have found the cause : Cloudfront & Googlebot.
Context :
I launched a French website that compares and combines all ...
2
votes
2answers
259 views
Can I request a site removal from Google and then re-index a new site on that domain?
.
Basically I have replaced my site completely but upon searching Google, it returned some old pages that didn't exist anymore, so I have used the Google index -> Remove URLs from Google Webmaster ...
2
votes
1answer
35 views
Can a JS redirect to a not found page work in place of a real 404?
I need assistance with some issues on a Big Commerce site.
To preface, it is not my site and the IT team has a complex technology stack being utilized to build the site.
The issue is that adding an ...
0
votes
0answers
12 views
In a sitemap, should I update the lastmod tag of a url based on the text content or html content? [duplicate]
Imagine I have this blogging / ecommerce website with 1000 posts / products. And I've built a sitemap for it, which is dynamically generated. Basically it's a list with a bunch of <url> and <...
1
vote
1answer
24 views
If I make changes in the template I use for a product page, do I need to update all <lastmod> tags for all product pages in my sitemap.xml?
I'm currently building a sitemap for my web app.
Note: my app is a single page app built with React and Firebase.
Basically I wan to index blogPost and product pages. So I intend to generate my ...
3
votes
1answer
77 views
Flag my website as a single page application?
Is it possible to flag my website as a SPA?
The reason I want to do this is because on Google Search Console I see my links indexed without canonical links, but if I try for a live version the ...
1
vote
1answer
24 views
Google live test can't load AngularJS XHR resources due to CORS policy
I implemented my website using AngularJs, It works correctly on any browsers and locations I've tested.
Although they claimed that Google Search Engine sees exactly what modern browsers display, my ...
1
vote
0answers
15 views
Can I Prevent Adsense Loading For GoogleBot?
I have been working on lots of performance updates to my site and I have gotten pretty much every page type to load in under 2 seconds. Most single post pages are coming in at around 1.5 seconds.When ...
3
votes
1answer
56 views
Would Google accept a text sitemap if the URL had a .php extension?
Google accepts a sitemap as .txt file with a list of URLs separated each by a newline.
Would Google also allow the same text file format with a .php? Ie.mysitemap.php instead of mysitemap.txt?
2
votes
1answer
309 views
Crawled page's screenshot is blank in the Google Search Console
I set up a SPA page through Angular, and achieved the effect of Server Side Render through Rendertron, In the facebook debug tool, I could get my dynamically generated Meta data, but the preview of ...
2
votes
2answers
38 views
Will lengthening our URLs change how Googlebot crawls our site?
Currently we are using URL fetched from main title of article (55 characters). We are planning to increase title of article up to 100 characters which tends to increase length of URL to 100+.
Will ...
2
votes
1answer
17 views
Page cannot be reached in Google Rich snippet test
When i test my webpage in google rich test, it say page cannot be reached. My website is on shopify so i guess there should be no issue from server side.
I need your help please... I cannot figure out ...
1
vote
0answers
25 views
Can alternate URLs for hreflinks be in different XML sitemap files when using <xml:link rel=“alternate”
Regarding the XML sitemaps (that use alternative languages), in this page it is specified the following:
Common Mistakes
Here are the most common mistakes with hreflang usage:
Missing ...
2
votes
1answer
23 views
Can robots.txt be used to prevent bots from seeing lazily loaded content?
Let's say that googlebot is scraping https://example.com/page.
example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page
/page lazy loads content using /...
1
vote
0answers
23 views
Is there a way to use js templating with wordpress without impacting seo or ux?
I have one JSON file that contains the text content for my hundreds of Wordpress posts. I am using templating, mustache.js, to insert the text content and render the posts. This saves me a lot of time,...
1
vote
1answer
32 views
Google Crawl Stats: What's included in “time spent downloading a page” metric?
Background
I'm the owner of a quite large site (millions of pages). I'm trying to optimize the site for "time spent downloading a page"-metric in google search console. Currently my stats in Search ...
7
votes
1answer
854 views
Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files
We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
2
votes
1answer
19 views
Is it OK for SEO to remove 10 of 16 translations of our site in subdirectories?
Our website has 16 different languages, each in a subdirectory. hreflang is not implemented.
We are now trying to redesign the website and we want 6 languages to be active. The rest of the languages ...
7
votes
2answers
4k views
Googlebot crawls my page too often
I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot).
But I have 1 article that have ...
1
vote
1answer
20 views
How does a live test in Google search console affect ranking if it doesn't pick up all the images and other content?
In google search console I conducted live test URL for my site's home page. However, the screenshot of the live test does not look like my home page.
There are not many images and text.
My home page ...
15
votes
3answers
9k views
SEO - Responsive Website and Duplicated Menus
Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
1
vote
0answers
38 views
404 errors as a result of …/false urls
I have been reviewing 404 error logs on a WooCommerce site I have been working on (logs held within the website server) and there are a lot of requests to product pages which have /false on the end of ...
3
votes
2answers
56 views
Will I get penalized by Google for cloaking if I redirect mobile users to a sub domain?
I have a website https://example.com which is responsive but it came a time where it displays too much text. So I decided to create a progressive web application version and would like to deploy it ...
3
votes
0answers
1k views
URL will be indexed only if certain conditions are met
I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
2
votes
2answers
26 views
How does google treat h1-3's that are styled to be smaller than a lower class of heading?
For example:
You have you h1
<h1>Epic SEO Keywords</h1>
then right below
<h2 class="Massive text">Enjoy the future of you, visualise your dreams</h2>
Will Google render ...
2
votes
1answer
211 views
Google AdWords ads are disapproved because “destination not working”
We have been having an issue setting up adwords for one of our clients and despite multiple phone calls to Google we cannot get an answer out of them as to why our ads are being disapproved.
...
3
votes
1answer
92 views
How to change Google crawl rate in new Google search console?
Google has now completely removed the old search console (https://searchengineland.com/the-old-google-search-console-is-no-longer-available-321650) where you used to be able to request a custom crawl ...
3
votes
2answers
161 views
Can Google crawl dynamic structured data added in a window.onload() event?
I inject some JSON-LD into a page dynamically via Javascript, when I test the page using Google's Structured Data Testing Tool, the expected output appears and I can see the Product Element is ...
1
vote
1answer
47 views
Shrinking site's size by removing pages from sitemap and from internal links
We are an eCommerce site with about 50,000 pages. We decided to focus on about 2,000 so google will crawl our site better.
We planning to remove 48,000 pages from the sitemaps and from the site links, ...
1
vote
1answer
338 views
Security Issue - Harmful content
I am trying to solve an issue with my website already for some time but without any luck.
Google has detected somewhere in my website harmful content (Malware). But the problem is, I could not find ...
2
votes
0answers
23 views
URL crawlable in old Google search console, but blocked in new
Google stopped crawling images for our site, when we moved images to cdn subdomain.
Old search console states image is crawlable, but new one states it is blocked by robots.txt (for same link).
our ...
3
votes
1answer
51 views
Can I hide ads for Googlebot so it loads my site faster?
I have ads which load sometimes slower, I want to hide those ads for Googlebot. I plan to hide that ad code for googlebot only and show them only to users.
Do I risk a Google penalty if I hide ads, ...
2
votes
1answer
3k views
In robots.txt, why would there be “Allow: /*” below a Disallow rule?
I want to allow full access to all Google
I am specially confused about Allow: /* below Disallow:
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
# Google AdSense
User-agent: ...
1
vote
0answers
43 views
Crawl anomaly caused by googlebot crawling secure_link mp4 files with md5 and expires parameters
Googlebot continues to crawl mp4 files in the format www.example.com/video/12345.mp4?md5=2029zanlkdain312on&expires=1523327783 which has obviously expired (only valid timestamp+30minutes, after ...
3
votes
2answers
155 views
Googlebot keeps on crawling stale/nonexistent(410) resources and then shows a crawl anomaly followed by de-indexing pages
We have been experiencing constant crawl anomalies caused by googlebot continued crawling of 'rnd.js?asdfasdfasfs3423' (hash is random on each pageload). It has been 3 months since we removed rnd.js ...
0
votes
0answers
46 views
Referencement issue with irrelevant suffix
I have a problem concerning the "coverage" on Google Search Console. The website of the company I'm working for has weird errors on this page. Google is telling that it had issues referencing pages ...
3
votes
0answers
54 views
Pantheon Dev Site Trapped in Google Index (with injected robots.txt)
We have a bizarre scenario I'd like opinions on. A site at example.com is being built on pantheon, using their example.pantheon.io staging / dev URL*.
Pantheon's development platform injects a ...
2
votes
0answers
376 views
Disqus comments cause “clickable elements too close together” in Google Search Console mobile usabality report
I just got an email from Google that they found some problems on my site:
clickable elements too close together
Content wider than screen
I fetched problematic URLs as Googlebot and noticed that it ...
1
vote
1answer
61 views
Google is ignoring meta tags and robots.txt
I have the following meta tags on my website (dev server):
<meta name="robots" content="noindex">
<meta name="googlebot" content="noindex">
And I also configured a robots.txt (...
11
votes
1answer
9k views
link rel=“alternate”, multiple languages and canonical urls
Context: We've got a website which is available in multiple language versions. However, the content is the same. Each translated version of the site is available on a distinct subdomain.
On some ...
0
votes
0answers
9 views
Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]
Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...
