Questions tagged [googlebot]
Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.
929
questions
1
vote
1
answer
27
views
Bots trying to crawl non-existant movie related pages which appear to be from IMDB on my project management website
In the Visitors log in cPanel I can see that googlebot, semrushbot, yandex and others are trying to access non-existant urls. The urls are related to movies & actors and appear to be on IMDB. They ...
0
votes
1
answer
32
views
Website is not showing any Traffic on Google, SEMrush and also on other Platforms
I have an issue regarding a website as when I check it on Google Analytics, Semrush, and other tools, it does not show traffic results. I have attached the images of the search and also the admin ...
- 113
0
votes
1
answer
30
views
Risk-benefit of cleaning up thousands of "Alternate page with proper canonical tag"?
The canonical URL to our products is example.com/product-alias in the sitemap, inside link rel="canonical" etc., and Google picks it up.
Yet from the category pages, we link to our products ...
- 425
0
votes
0
answers
25
views
Indexing speed of 500k+ pages site
I am reaching out for some guidance regarding the Google Search Console (GSC) indexing of my website. I launched this site about a month ago, and I'm encountering some challenges with Google's ...
- 1
1
vote
0
answers
29
views
Getting indexed on Google from mysql database [closed]
I have a website that sells 100,000 items. The item data is stored in a mysql database. So, there aren't 100,000 html files on my site. Therefore, google can't see them. I read that the solution to ...
- 121
0
votes
0
answers
15
views
Block Googlebots from fetching robots.txt on a microservice subdomain
We have some microservices and as an example, Language Service's URL is the following;
http://prod-lang-service.example.com/
We have been getting errors (NotFoundException) on Rollbar that Google ...
- 101
1
vote
0
answers
46
views
How to allow indexing images for Favicon only?
Generally I like to disallow search engines to index images on my site.
There is one exception to this:
I like the Google Favicon Bot to index images in order to show the icon in search results.
I am ...
- 198
3
votes
1
answer
26
views
Why site:example.com doesn't give the same amount of results in incognito mode
Did you ever tried to search on Google site:domain.example on incognito mode? I am never getting the same amount of results than when I am doing it in normal mode. Sometimes more, sometimes less, do ...
- 141
2
votes
1
answer
53
views
Validation of canonical links not working in Google Search Console
I internationalized my site's canonical links using the guide from here: https://yoast.com/hreflang-ultimate-guide/#hreflang-canonical
Which has an output like this:
<link rel="canonical" ...
- 217
1
vote
1
answer
20
views
How to properly block indexing my admin subdomain?
I have a website hosted on https://admin.example.com.
However, the public website displayed is on this sub-directory : https://www.example.com/website/
For technical reasons, we cannot install it or ...
- 131
0
votes
1
answer
46
views
How to overcome Google indexing issue? [duplicate]
It's almost been more than 6 months i'm facing Google indexing issue my site can not be indexed. The issues are described below:
1- "Excluded by ‘noindex’ tag" although I have searched all ...
2
votes
1
answer
169
views
Google Search console: Blocked due to access forbidden (403) when blocking all country except (India, Srilanka)
I am a very old user of cloudflare for the last 4 Years, I have 14+ websites of different categories which has have added to cloudflare last year. My website was accessible and ranked on google which ...
- 21
2
votes
0
answers
255
views
How good is Google in indexing and crawling SPA webpages?
I am about to relaunch our website, but I am uncertain whether I should go with a Vue.js SPA webpage or a Nuxt.js SSR webpage. The big advantage of an SPA webpage is that it has an app-like feeling ...
- 121
1
vote
1
answer
85
views
Googlebot Spam / DDoS?
I recently saw a significant googlebot crawl on my website. We're talking 10-20 requests per second. It looked like googlebot, and seemed to be legit googlebot IPs, but the requests had query params ...
- 111
0
votes
0
answers
37
views
Why does our site name keep reverting back to an incorrect name?
We have two websites living under two different domains. Domain A and domain B. Domain B was decommissioned in February and migrated over to domain A with a URL path /en-us/. All of the old URLs from ...
1
vote
1
answer
97
views
Large increase in 'Discovered - currently not indexed' pages in Google Webmasters
For last few months, we noticed a large increase in 'Discovered - currently not indexed' pages in Webmaster tools.
By last month, we had almost half of our site pages were classified as discovered but ...
- 43
1
vote
0
answers
17
views
Website pages crawled entirely, thereafter only a 50% of pages are [duplicate]
Any technical or implementation tips on how to improve the crawling of site pages, when a given site used to have most pages crawled on its launch and thereafter only > 50% of them are. This ...
- 23
2
votes
1
answer
231
views
How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?
I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
- 21
4
votes
3
answers
70
views
Use query parameters or hashes when implementing referral codes?
We're implementing a feature where users can generate unique referral links that they can share and receive rewards when people sign up. The pages themselves have identical content.
There are two ...
- 161
1
vote
1
answer
18
views
How can a react/vue apps like these rank just by using client side rendering?
By client-side rendering I mean, you can't see any content (h1,title,meta tags etc) in the source code, which is the only place where google bot can learn what a page is about.
here is this site: ...
- 11
1
vote
1
answer
104
views
0 discovered links in Google Search Console and only my homepage is indexed [duplicate]
Last month I launched my website.
It is using SSR, fixed almost all high/medium priority SEO issues existing in screaming frog, all pages are indexable and SEO friendly and from my index page I have a ...
1
vote
1
answer
77
views
If Both Article Page and PDF File Have the Same Content, Will It Cause SEO Problems?
I have a blog page and I publish articles on this blog page. You can download these articles as PDFs by clicking the download button on that article.
PDFs are the same content found in the article. ...
- 1,017
0
votes
1
answer
185
views
Does blocking api URL hurt rendering by googlebot
In my GSC crawl stats most of the 404s and URLs using page resource load are:
API URLs for a JavaScript part of our content
Cloudflare's managed challenge
If I add an entry to robots.txt for 1, will ...
- 1
0
votes
0
answers
10
views
Having trouble dealing with backticks in keyword phrase: Yoast
I have a site I wrote in WordPress. I've been using the Yoast SEO plugin to see if I can get my content more easily found.
I just wrote an article on my site about the SQL SELECT statement. It begins ...
- 103
1
vote
0
answers
248
views
Why is the Chrome-Lighthouse user agent hitting my server every day?
Every day, I find lines like this one in my access.log file:
74.125.210.42 - - [08/Mar/2023:08:11:13 +0100] "GET /fr/ HTTP/2.0" 200 24160 "-" "Mozilla/5.0 (Macintosh; Intel ...
2
votes
1
answer
130
views
Google Spidering ajax Call URL as Stand Alone Content
My page has an ajax call that loads fresh html into the mother page.
Google has now spidered these ajax urls and runs it's search console algorythm over them as if it were a stand alone page.
In the ...
- 21
0
votes
0
answers
18
views
Does pinging a sitemap under a certain path affect crawling of parent directories/sitemaps?
I couldn't find much information about this topic in the official docs, but under Google's best practices:
"You can host your sitemaps anywhere on your site, but a sitemap affects only ...
- 101
1
vote
1
answer
122
views
How did Google crawler view my password-protected pages?
I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped.
I am trying to figure out how this could have happened. It is a large site ...
- 217
1
vote
1
answer
72
views
Site search form getting spammed, but by Google IP
I have the search form on a section of a site that is getting bombarded with requests like:
search query = baccarat philippe starck furniture, ➡️g188.ph⬅️, play free slots with bonus gcash logo cash, ...
- 111
2
votes
1
answer
40
views
Will search engines deindex the entire resource when returning HTTP 410 for a parameterized URL?
Will the HTTP 410 response code make search engines deindex the underlying page resource? Or will Googlebot and Bingbot only deindex the URL with query it's parameters as whole?
The topic Redirection ...
- 7,650
2
votes
1
answer
2k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
1
vote
0
answers
468
views
Removing __NEXT_DATA__ script tag for google bots
Our e commerce website is built on next js. We wanted to optimise the bit crawl bandwidth by reducing html size shipped to google bots. We want to treat user and bot differently. As we know bots don’t ...
- 141
0
votes
0
answers
105
views
Google Bots Crawling Invalid URL
Google Bots crawl URLs by appending trailing 200 and false for all the article pages. I am unsure what the issue is here and why that is getting appended on every article page.
I have tried scanning ...
3
votes
1
answer
109
views
What is the actual HTML rendered to bots?
We have a product page in which 4 item carousels are present, showing mostly related products. All these 4 carousels are client side rendered.
1. Scenario1: Google rich results
Now, we have observed ...
- 141
2
votes
1
answer
63
views
How to handle additional content on paginated pages?
we have category pages with a few hundred products per category.
The category pages are properly paginated according to Googles guidelines for pagination.
One page shows 16 products. In addition to ...
- 141
1
vote
1
answer
52
views
Is blocking indexing and crawling at the same time bad?
have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
- 65
1
vote
0
answers
59
views
Google Crawls mostly Javascript files of my HTML website [closed]
I have an issue with my HTML static page website, and the issue in my HTML(DOM) crawls only 11% and javascript crawls 70%
- 111
2
votes
1
answer
47
views
Is disallowing all useragents sufficient to keep a site from being crawled by search engines?
We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
- 123
2
votes
0
answers
41
views
One day after putting my site online and registering it with Google I got spam mail. How?
I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
2
votes
0
answers
29
views
Googlebot increased the number of requests for a certain URL even though it has no links
As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
- 51
0
votes
0
answers
19
views
MODx issue, cannot be crawled [duplicate]
I have a page https://zabery365.ru/removal/bath for my client. Please look at the code. I have a big trouble - for a 3 weeks it has not been crawled, pagespeed also said that there are no data. GSC ...
1
vote
1
answer
61
views
Alternative to AWS lambda redirect that doesn't affect Googlebot
We have the following geographic subdirectories of our website:
www.example.com (for USA and worldwide visitors)
www.example.com/en-au/ (for Australian visitors)
www.example.com/en-gb/ (for UK ...
- 1,456
3
votes
0
answers
45
views
Can I use canonical tags in place of 301 redirect that remove a parameter that had been used by a meta refresh?
For one of my sites, I had used a the meta tag refresh:
<meta http-equiv="refresh" content="...;url=...">
A few months ago, I removed this tag and instead implemented 301 ...
- 131
3
votes
0
answers
75
views
Does Google crawl domains with dynamic IP addresses?
Does Google crawl web sites with dynamic IP addresses and dynamic DNS entries?
2
votes
0
answers
234
views
What is correct way to stop query indexing using Robots.txt?
How to stop indexing search result using Robots.txt?
hello.
Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.
These pages are mostly search query (/?s=) and ...
- 21
2
votes
2
answers
549
views
Will Googlebot click on tab navigation / buttons and index links?
I have the following UI:
There are tabs which are <button>'s and when you select one there will be <a>-Links to different pages. It seems like Google will only index the Links on the ...
- 153
2
votes
4
answers
376
views
How long does Googlebot take to crawl a page?
Googlebot crawls a certain number of pages daily, depending on the size of the site. However, how long does it take for a particular page. In other words, how many seconds or minutes does it spend on ...
- 21
0
votes
0
answers
43
views
Mobile Navbar SSR
Currently, we serve our site using SSR. On some parts of the site, we render components based on the screen size (mobile navbar for example).
The SSR is serving the desktop version.
So, Googlebot ...
- 51
3
votes
1
answer
108
views
Why does Google discover a web page but not crawl it?
Several years ago, posts on my site would be indexed very quickly (usually 1-2 days after I publish them). But now, it takes 2-3 months after publish before pages get indexed.
This is not due to low ...
- 171
0
votes
0
answers
243
views
Are .JSON and сhunk .js in Next.js files important for Googlebot to render page correctly?
I have a problem with Googlebot indexing my website. He scans a lot of JSON and chunks .js files instead of scanning. HTML files. These unnecessary files reduce the crawl budget of my website. So ...
