Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Too many hits from GoogleBot

The last week I have about 500.000 hits/day from googlebot in one of my websites. As a result, the server has serious issues to handle the rest of the requests. What I have done Verified that it is ...
1
vote
2answers
19 views

Google Search Console shows two different “Last crawled” dates for the same URL. Which one is correct?

For the same URL (x): Coverage > Indexed, not submitted on sitemap > Examples > URL x > Last Crawled shows "Mar 29, 2020". Inspect URL > URL x > Coverage > Last crawl shows "Apr 4, 2020, 2:47:23 AM"....
1
vote
0answers
14 views

Should we use a canonical tag even if we are sure the page doesn't have any duplicate?

If my pages don't have any duplicates, and their URLs are all clean (no '?id=123456' stuff), should I still use canonical tags? (from a SEO perspective) In other words, would bots downgrade me (or ...
2
votes
2answers
27 views
+50

How to limit Google crawling certain URL patterns?

recently we have updated our e-commerce website, where each product category has product filter. There is around 30 attributes which can be used for filtering products shown in category. These ...
2
votes
1answer
22 views

How to simulate googlebot to see which links in a React app would be indexed?

I am developing a React app. I’ve had poor indexing coverage until now (only the home page was indexed). I recently implemented server side rendering (SSR) and indexing coverage appears to be ...
1
vote
1answer
11 views

How to check if sitemap index has been successfully submitted and can be properly read

Recently, I have submitted a sitemap index with a slew of xml files and URL links. I see that the sitemap index has been successfully submitted and I understand that the discovered URLs is possible ...
2
votes
0answers
14 views

Disallow Image Loading to Reduce Number of “Other Error” in Crawled Page Resources?

We have a hotel search website. Many landing pages of our site load resources asynchrously (e.g. Ajax), but very often (about 25% of the time) when we click "View Crawled Page" in Google Search ...
3
votes
0answers
16 views

Would adding links between pages help Google index more of our site?

We have a hotel search website. We have identified some concepts that we would like Google to index, for example, we have a landing page for "Boutique Hotels in Chicago, IL", Given number of cities in ...
0
votes
1answer
8 views

Google Search Console: issues detected “Top Errors” Server Error (5xx). BUG?

Just got this email from Google Search Console: This were the issues: But when I inspected those live URL's (using Google Search Console tool) they came back as "available to Google" and one of them ...
0
votes
1answer
16 views

Excluding URLs to avoid thin content: just “noindex”? also “dofollow”?

My website has 100,000 URLs with auto-generated content and I'm suffering from thin content (drop in ranking and alerts of duplicate content in Google Search Console). I want now to exclude those ...
1
vote
1answer
23 views

Impact on Google Shopping of Blocking UTM Parameters in Robots.txt File

I am optimising the crawling experience of the website as a large portion of the site isn't being crawled. A large portion of the 'crawl budget' is being used on crawling URLs with utm parameters for ...
2
votes
1answer
26 views

Google Search Console Warning: “ Indexed, though blocked by robots.txt ” (BUG)

Two days ago I've got an email with a warning from Google Search Console, telling me that 3 URLs from my site where indexed, but were blocked by robots.txt. Since I first deployed my website, I haven'...
0
votes
0answers
14 views

Can I use the thumbnails of external links in a news aggregator website?

I'm building a site where I need to post some news (links) about a specific subject. I'll gather them around and aggregate them on my site. The question is: Can I use their thumbnail to promote ...
1
vote
1answer
26 views

Why does Googlebot attempt to crawl /admin/install.php?

On one site I own, I recently started seeing Googlebot checking for non-existing URIs: 66.249.76.89 - - [23/Feb/2020:10:18:48 +0100] "GET /robots.txt HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; ...
1
vote
1answer
21 views

Are these expiration headers wrong or could they hurt SEO?

I cannot figure out why my rankings dropped, at the same time I deployed 4 Nginx cache servers on different continents. When I serve the PHP pages, I set this expiration header from the origin server,...
2
votes
1answer
71 views

Why is Google making up pages to crawl on my site?

For some reason Google is listing a series of pages that do not exist on my website such as: https://www.my_domain.com/index.php/about_us.php It lists them as 'Duplicate, Google chose different ...
0
votes
0answers
14 views

getting google to index anchored URLs with one dynamic image per anchor tag

Background I have a site (https://example.com) that shows a slideshow of images. The site uses a single <img> tag in the DOM. On page load, the JavaScript loads the 30 or so total images into &...
4
votes
1answer
59 views

Googlebot ignores robots.txt

Two months ago i disallowed some directories from crawling with robots.txt. Since then i monitor logfiles everyday and realize, that Googlebot absolutely ignores robots.txt. Really, it crawls every ...
1
vote
1answer
21 views

Is it worth handling SEO friendly URLs that might change over time? Or should I use generic ones that will not change?

I will use SEO friendly URL slugs for my website pages. But I would like to know how far should I go down this road. My specific question is about whether I should be choosing SEO friendly URL's ...
0
votes
1answer
17 views

Changed URL for a page that was indexed by Googlebot. Will redirect 301 from the old URL to the new one. But what to do with my Sitemap?

I'm planning to change a url for one of my site's page. Example: From: https://www.example.com/old-post-slug To: https://www.example.com/new-post-slug The fact is that Google has already ...
1
vote
1answer
24 views

What are the SEO ramifications of duplicate content on the same page?

My question has some similarities to this one. But since I am asking about a very specific type of section on the page I think it warrants some thought as to whether the approach I am looking to take ...
0
votes
1answer
30 views

Tracking outbound links with Google Tag Manager without hurting SEO

My website contains a widget with links to social media and I'm using a script to open the links. Because I'm not adding the links directly into href attribute I think the web crawlers may be missing ...
0
votes
1answer
21 views

Google Crawl rate & the log file mystery

According to Google Search Console for a website I am working on, Googlebot crawls ~5000 pages per day (min 2500, max 8500). However, when looking at the Apache log files, GoogleBot only shows up ~10 ...
3
votes
2answers
107 views

Why is my meta description blank in SERPs?

My client's site, which runs on Nuxt and Vue, is not showing meta descriptions on Google SERPs. They used to and as of last week they don't. Any idea why meta description tags would be stripped from ...
1
vote
2answers
32 views

Why does Googlebot keep coming back to recrawl pages that have been returning 410 for a long time?

It was early 2018, almost two years ago, when I retired a bunch of pages from my site that were no longer needed/relevant/maintained/etc. I did that by returning a 410-Gone HTTP status. To my surprise,...
1
vote
1answer
23 views

Is it better to create a new level in page hierarchy or add pages at same level?

If the existing site structure is www.example.com/abc/def With the page hierarchy as home>>abc>>def If I want to add a new set of pages, as home>>abc>>def>>ghi, which format of the URL would be ...
1
vote
0answers
54 views

JSON-LD structured data not showing up on live url test(search console)

I am appending dynamic structured data to the head of html as and when my ajax response comes. Something like:- function appendSchemaToScript(schema){ var el = document.createElement('script'); ...
0
votes
0answers
12 views

Angular SEO friendly product detail page URL

Hello folks currently I am searching for best SEO friendly URL for product detail page for our live site Right-on EC where currently we use the following pattern for product detail page it seems not ...
0
votes
0answers
16 views

Is there a way to get Google Search crawler to detect meta tags when they are injected asynchronously?

I have an issue with the Google search crawler that I need some help resolving. We have a vue.js application that reaches out to our Prismic CMS for meta tag data. For example, title, description, etc....
0
votes
0answers
9 views

How can I created tracking redirects that are neither crawled nor indexed?

I have tracking redirects on my site. I use server-side tracking on them. They are used for things like: Advertisements Social media sharing The URLs are of the forms: /ad-click?id=12345 /...
0
votes
0answers
13 views

Meta tags - Meta keywords [duplicate]

I have a question or an huge doubt on "meta keywords in the meta tags". Some members are saying "meta keywords" is not important & some of them are saying it is also important. So kindly provide ...
2
votes
1answer
107 views

SEO cloaking rules when deferring loading content only after user interacting

what are the rules with google (or any other search engine) in regards to cloaking (e.g. showing different content for bots vs real users) The question is what if we want to load certain content only ...
1
vote
2answers
48 views

What is “ineedthispage”? Where is it coming from? It is killing the server

I am seeing the following in my "access_log": ip.address.output.here - - [12/Dec/2019:15:17:07 +0000] "GET /pathto/article/15104133/?ineedthispage=yes&ineedthispage=yes&ineedthispage=yes&...
0
votes
0answers
38 views

How does Googlebot crawl pages having cookiepro integration

Google bot is appending the link present in cookiepro banner on my main product-details page and hence giving 404. On the webapp the link is going to the proper URL, but I can see a lot of URLS in my ...
1
vote
0answers
20 views

What means the “average Bytes” metric in logfiles?

I'm currently studying the logfiles - namely those from Googlebot access. I realized, that the average Bytes metric is in the majority of cases (urls) 10-20% of the page size. Many questions were ...
3
votes
1answer
27 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
3
votes
1answer
151 views

What headers used in request by google bot?

What headers used in request by google bot? Must be user agent, but what else? I'm interested in cache control headers, does bot try to get fresh page, but not from cache? The related question: Do ...
2
votes
1answer
68 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
1
vote
2answers
244 views

What are the SEO implications of an established website shifting from server-side rendering to client side?

I run a decently established website (20k monthly users, already indexed by Googlebot, etc) that has been built on Django a year and a half ago. The vast majority of traffic on my site comes from the ...
2
votes
1answer
38 views

Is a child element set to visible while its parent set to hidden bad for SEO?

To modify the style of a component within a framework, I need to set the visibility of the parent container to hidden and the visibility of its child that contains the content to visible. Is that bad ...
2
votes
1answer
165 views

Can a JS redirect to a not found page work in place of a real 404?

I need assistance with some issues on a Big Commerce site. To preface, it is not my site and the IT team has a complex technology stack being utilized to build the site. The issue is that adding an ...
0
votes
0answers
14 views

In a sitemap, should I update the lastmod tag of a url based on the text content or html content? [duplicate]

Imagine I have this blogging / ecommerce website with 1000 posts / products. And I've built a sitemap for it, which is dynamically generated. Basically it's a list with a bunch of <url> and <...
1
vote
1answer
108 views

If I make changes in the template I use for a product page, do I need to update all <lastmod> tags for all product pages in my sitemap.xml?

I'm currently building a sitemap for my web app. Note: my app is a single page app built with React and Firebase. Basically I wan to index blogPost and product pages. So I intend to generate my ...
1
vote
0answers
22 views

Can I Prevent Adsense Loading For GoogleBot?

I have been working on lots of performance updates to my site and I have gotten pretty much every page type to load in under 2 seconds. Most single post pages are coming in at around 1.5 seconds.When ...
3
votes
1answer
65 views

Would Google accept a text sitemap if the URL had a .php extension?

Google accepts a sitemap as .txt file with a list of URLs separated each by a newline. Would Google also allow the same text file format with a .php? Ie.mysitemap.php instead of mysitemap.txt?
2
votes
2answers
38 views

Will lengthening our URLs change how Googlebot crawls our site?

Currently we are using URL fetched from main title of article (55 characters). We are planning to increase title of article up to 100 characters which tends to increase length of URL to 100+. Will ...
2
votes
1answer
27 views

Page cannot be reached in Google Rich snippet test

When i test my webpage in google rich test, it say page cannot be reached. My website is on shopify so i guess there should be no issue from server side. I need your help please... I cannot figure out ...
1
vote
0answers
51 views

Can alternate URLs for hreflinks be in different XML sitemap files when using <xml:link rel=“alternate”

Regarding the XML sitemaps (that use alternative languages), in this page it is specified the following: Common Mistakes Here are the most common mistakes with hreflang usage: Missing ...
2
votes
1answer
25 views

Can robots.txt be used to prevent bots from seeing lazily loaded content?

Let's say that googlebot is scraping https://example.com/page. example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page /page lazy loads content using /...
1
vote
0answers
26 views

Is there a way to use js templating with wordpress without impacting seo or ux?

I have one JSON file that contains the text content for my hundreds of Wordpress posts. I am using templating, mustache.js, to insert the text content and render the posts. This saves me a lot of time,...

1
2 3 4 5
16