Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
1
vote
0answers
19 views

JSON-LD structured data not showing up on live url test(search console)

I am appending dynamic structured data to the head of html as and when my ajax response comes. Something like:- function appendSchemaToScript(schema){ var el = document.createElement('script'); ...
0
votes
0answers
9 views

Angular SEO friendly product detail page URL

Hello folks currently I am searching for best SEO friendly URL for product detail page for our live site Right-on EC where currently we use the following pattern for product detail page it seems not ...
0
votes
0answers
9 views

Is there a way to get Google Search crawler to detect meta tags when they are injected asynchronously?

I have an issue with the Google search crawler that I need some help resolving. We have a vue.js application that reaches out to our Prismic CMS for meta tag data. For example, title, description, etc....
0
votes
0answers
9 views

How can I created tracking redirects that are neither crawled nor indexed?

I have tracking redirects on my site. I use server-side tracking on them. They are used for things like: Advertisements Social media sharing The URLs are of the forms: /ad-click?id=12345 /...
0
votes
0answers
12 views

Meta tags - Meta keywords [duplicate]

I have a question or an huge doubt on "meta keywords in the meta tags". Some members are saying "meta keywords" is not important & some of them are saying it is also important. So kindly provide ...
2
votes
1answer
102 views

SEO cloaking rules when deferring loading content only after user interacting

what are the rules with google (or any other search engine) in regards to cloaking (e.g. showing different content for bots vs real users) The question is what if we want to load certain content only ...
1
vote
1answer
38 views

What is “ineedthispage”? Where is it coming from? It is killing the server

I am seeing the following in my "access_log": ip.address.output.here - - [12/Dec/2019:15:17:07 +0000] "GET /pathto/article/15104133/?ineedthispage=yes&ineedthispage=yes&ineedthispage=yes&...
0
votes
0answers
28 views

How does Googlebot crawl pages having cookiepro integration

Google bot is appending the link present in cookiepro banner on my main product-details page and hence giving 404. On the webapp the link is going to the proper URL, but I can see a lot of URLS in my ...
1
vote
0answers
20 views

What means the “average Bytes” metric in logfiles?

I'm currently studying the logfiles - namely those from Googlebot access. I realized, that the average Bytes metric is in the majority of cases (urls) 10-20% of the page size. Many questions were ...
2
votes
0answers
15 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
3
votes
1answer
59 views

What headers used in request by google bot?

What headers used in request by google bot? Must be user agent, but what else? I'm interested in cache control headers, does bot try to get fresh page, but not from cache? The related question: Do ...
2
votes
1answer
60 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
1
vote
2answers
238 views

What are the SEO implications of an established website shifting from server-side rendering to client side?

I run a decently established website (20k monthly users, already indexed by Googlebot, etc) that has been built on Django a year and a half ago. The vast majority of traffic on my site comes from the ...
2
votes
1answer
36 views

Is a child element set to visible while its parent set to hidden bad for SEO?

To modify the style of a component within a framework, I need to set the visibility of the parent container to hidden and the visibility of its child that contains the content to visible. Is that bad ...
2
votes
1answer
62 views

Can a JS redirect to a not found page work in place of a real 404?

I need assistance with some issues on a Big Commerce site. To preface, it is not my site and the IT team has a complex technology stack being utilized to build the site. The issue is that adding an ...
0
votes
0answers
13 views

In a sitemap, should I update the lastmod tag of a url based on the text content or html content? [duplicate]

Imagine I have this blogging / ecommerce website with 1000 posts / products. And I've built a sitemap for it, which is dynamically generated. Basically it's a list with a bunch of <url> and <...
1
vote
1answer
100 views

If I make changes in the template I use for a product page, do I need to update all <lastmod> tags for all product pages in my sitemap.xml?

I'm currently building a sitemap for my web app. Note: my app is a single page app built with React and Firebase. Basically I wan to index blogPost and product pages. So I intend to generate my ...
1
vote
0answers
16 views

Can I Prevent Adsense Loading For GoogleBot?

I have been working on lots of performance updates to my site and I have gotten pretty much every page type to load in under 2 seconds. Most single post pages are coming in at around 1.5 seconds.When ...
3
votes
1answer
60 views

Would Google accept a text sitemap if the URL had a .php extension?

Google accepts a sitemap as .txt file with a list of URLs separated each by a newline. Would Google also allow the same text file format with a .php? Ie.mysitemap.php instead of mysitemap.txt?
2
votes
2answers
38 views

Will lengthening our URLs change how Googlebot crawls our site?

Currently we are using URL fetched from main title of article (55 characters). We are planning to increase title of article up to 100 characters which tends to increase length of URL to 100+. Will ...
2
votes
1answer
19 views

Page cannot be reached in Google Rich snippet test

When i test my webpage in google rich test, it say page cannot be reached. My website is on shopify so i guess there should be no issue from server side. I need your help please... I cannot figure out ...
1
vote
0answers
32 views

Can alternate URLs for hreflinks be in different XML sitemap files when using <xml:link rel=“alternate”

Regarding the XML sitemaps (that use alternative languages), in this page it is specified the following: Common Mistakes Here are the most common mistakes with hreflang usage: Missing ...
2
votes
1answer
25 views

Can robots.txt be used to prevent bots from seeing lazily loaded content?

Let's say that googlebot is scraping https://example.com/page. example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page /page lazy loads content using /...
1
vote
0answers
23 views

Is there a way to use js templating with wordpress without impacting seo or ux?

I have one JSON file that contains the text content for my hundreds of Wordpress posts. I am using templating, mustache.js, to insert the text content and render the posts. This saves me a lot of time,...
1
vote
1answer
44 views

Google Crawl Stats: What's included in “time spent downloading a page” metric?

Background I'm the owner of a quite large site (millions of pages). I'm trying to optimize the site for "time spent downloading a page"-metric in google search console. Currently my stats in Search ...
2
votes
1answer
19 views

Is it OK for SEO to remove 10 of 16 translations of our site in subdirectories?

Our website has 16 different languages, each in a subdirectory. hreflang is not implemented. We are now trying to redesign the website and we want 6 languages to be active. The rest of the languages ...
7
votes
2answers
4k views

Googlebot crawls my page too often

I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot). But I have 1 article that have ...
1
vote
1answer
21 views

How does a live test in Google search console affect ranking if it doesn't pick up all the images and other content?

In google search console I conducted live test URL for my site's home page. However, the screenshot of the live test does not look like my home page. There are not many images and text. My home page ...
2
votes
1answer
56 views

404 errors as a result of …/false urls

I have been reviewing 404 error logs on a WooCommerce site I have been working on (logs held within the website server) and there are a lot of requests to product pages which have /false on the end of ...
2
votes
2answers
28 views

How does google treat h1-3's that are styled to be smaller than a lower class of heading?

For example: You have you h1 <h1>Epic SEO Keywords</h1> then right below <h2 class="Massive text">Enjoy the future of you, visualise your dreams</h2> Will Google render ...
3
votes
1answer
118 views

How to change Google crawl rate in new Google search console?

Google has now completely removed the old search console (https://searchengineland.com/the-old-google-search-console-is-no-longer-available-321650) where you used to be able to request a custom crawl ...
3
votes
2answers
213 views

Can Google crawl dynamic structured data added in a window.onload() event?

I inject some JSON-LD into a page dynamically via Javascript, when I test the page using Google's Structured Data Testing Tool, the expected output appears and I can see the Product Element is ...
2
votes
0answers
25 views

URL crawlable in old Google search console, but blocked in new

Google stopped crawling images for our site, when we moved images to cdn subdomain. Old search console states image is crawlable, but new one states it is blocked by robots.txt (for same link). our ...
1
vote
1answer
54 views

Google live test can't load AngularJS XHR resources due to CORS policy

I implemented my website using AngularJs, It works correctly on any browsers and locations I've tested. Although they claimed that Google Search Engine sees exactly what modern browsers display, my ...
3
votes
1answer
52 views

Can I hide ads for Googlebot so it loads my site faster?

I have ads which load sometimes slower, I want to hide those ads for Googlebot. I plan to hide that ad code for googlebot only and show them only to users. Do I risk a Google penalty if I hide ads, ...
1
vote
0answers
46 views

Crawl anomaly caused by googlebot crawling secure_link mp4 files with md5 and expires parameters

Googlebot continues to crawl mp4 files in the format www.example.com/video/12345.mp4?md5=2029zanlkdain312on&expires=1523327783 which has obviously expired (only valid timestamp+30minutes, after ...
3
votes
2answers
203 views

Googlebot keeps on crawling stale/nonexistent(410) resources and then shows a crawl anomaly followed by de-indexing pages

We have been experiencing constant crawl anomalies caused by googlebot continued crawling of 'rnd.js?asdfasdfasfs3423' (hash is random on each pageload). It has been 3 months since we removed rnd.js ...
3
votes
0answers
59 views

Pantheon Dev Site Trapped in Google Index (with injected robots.txt)

We have a bizarre scenario I'd like opinions on. A site at example.com is being built on pantheon, using their example.pantheon.io staging / dev URL*. Pantheon's development platform injects a ...
1
vote
1answer
62 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...
0
votes
0answers
9 views

Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]

Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...
2
votes
3answers
66 views

Stop googlebot crawling URL more than once?

I have a site that usually creates a few thousand pages a day, which don't change after they have been created. Recently my dedicated server has crashed due to googlebot crawling the site too often. ...
0
votes
0answers
11 views

How to submit and get links indexed faster on google search console? [duplicate]

I've created a website which has a lot of links. I've added it to google search console in order to show them on google search but it is indexing around 2,000 links every week. I have 200,000+ links. ...
0
votes
1answer
54 views

Robots.txt, php.ini, .htaccess, etc

This will be my first website, so I am completely new to all of this. I cannot seem to find an answer anywhere, as to whether or not I should disallow configuration files like /php.ini or hidden ...
0
votes
0answers
5 views

What are the seemingly obvious things to improve search engine rank? [duplicate]

One thing that seems like the most obvious thing I can think of, at least when it comes to businesses, is to register your business with Google My Business. I can't think of anything more obvious that ...
1
vote
1answer
145 views

How to resolve Google “Indexed, though blocked by robots.txt” warning for a user login protected page?

Or, in other words, how can I tell google this is a user login protected page? The website is basically a single page app with server side rendering. That said, if the visitor is asking for a web ...
3
votes
2answers
65 views

Will I get penalized by Google for cloaking if I redirect mobile users to a sub domain?

I have a website https://example.com which is responsive but it came a time where it displays too much text. So I decided to create a progressive web application version and would like to deploy it ...
1
vote
1answer
39 views

Which page speed google SEO uses? [duplicate]

My SEO team told me Google SEO engine uses the page speed in its algo to rank the pages. It is confirmed by this blog too. In my case, page speed is different for end-user coming from browser than ...
1
vote
0answers
33 views

Huge differences between webmaster tools' crawl stats and apache logs

We noticed huge difference in the metrics. For example, last saturday : According to Webmaster Tools : 6,082 pages crawled 76,119 kilobytes downloaded 1,550 ms in average to download a page Now ...
0
votes
0answers
68 views

Angular 7 - will removing hash on my angular app make my website SEO friendly?

I have an angular website thats been running for more than 2 years but sadly when i checked, only the homepage was indexed. My about us, contact us page which are serving static content was not also ...
3
votes
1answer
73 views

GoogleBot crawls thousands of urls like 2487763877595434670.htm that doesn´t exist

in our serverlogs there are thousands of requests (50-100k) from googlebot every day for urls like /2487763877595434670.htm (as I can see always 19 random digits with .htm at the end) First the bot ...