Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
1
vote
0answers
35 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
1
vote
0answers
49 views

What is the correct response to Googlebot for a search_term_string from site search markup with _escaped_fragment?

I have a AJAX web online-shop with hashbang URL and added SiteLinks as JSON-LD. I follow this introduction. { "@context":"http://schema.org", "@type":"WebSite", "url":"https://...
1
vote
0answers
11 views

My most important pages does not get indexed, google search console [duplicate]

I have a website with blog articles. I added it's sitemap to google search console and it say that the sitemap has been processed. However, half of my articles (the important pages) have not been ...
2
votes
0answers
30 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
1
vote
0answers
37 views

Google cache is not showing up for a page despite trying twice to use the fetch and render tool [duplicate]

I did Fetch and Render twice on a particular page but the cached version of the page is still not showing up on Google. I did a Fetch and Render on other posts on the site and they all showed cached ...
0
votes
0answers
45 views

Any relation between adsense lack of impressions and incorrect sitemap?

I have recently submitted a sitemap in search console for at least 200 new pages on my site. However, I did a mistake by not including a slash ("/") at the end of each url. But the pages without ...
2
votes
2answers
264 views

CloudFront - disable access to distribution via d1234.cloudfront.net

I have a CloudFront distribution with a configured alternative name (e.g. cdn.example.com). Is it possible to restrict access to the CDN via d1234.cloudfront.net name? I don't want it to be indexed ...
2
votes
1answer
501 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
0
votes
2answers
323 views

Status Code 500 to Googlebot but url working

I have a URL on my site that is working perfectly. I have the robots with the index and follow, but when I Fetch as Google in the WMT says Unreachable and gives me this message in the crawl errors ...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
4
votes
0answers
69 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
4
votes
1answer
353 views

Dynamic content using .append() or .innerHTML() and SEO

I was looking for some clarification on something I am looking to implement, I would like to know how a Google Crawl will see this and if it would be detrimental to SEO. On my webpage I have an empty ...
4
votes
1answer
62 views

Does Google follow and count a backlink that goes through a JavaScript location.href redirect page?

example.com has a link on their website which is on same domain like, <a href="https://example.com?ref=mywebsite.tld">Reference</a> example.com?ref=mywebsite.tld doesn't redirect with ...
0
votes
0answers
14 views

Removing Paginated URLs from Google Search [duplicate]

I have a website running on Django, Python. I'm using django pagination so that users can click next & see further content. For example, my website is example.com, so when user requests new page ...
2
votes
0answers
163 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
1
vote
0answers
22 views

which status code should be returned for null parameter

having a website with many urls in js code (ajax calls), the google bot is calling these but without the necessary parameters atm 200 code is returned and the html for the error page, and this is ...
1
vote
2answers
96 views

How to get Googlebot to crawl my website faster to pick up my new meta description

I've tried to put an new meta description for my website, but when I look it up I get the wrong description. After some research I found out that I need Google to crawl my site. The cache:website ...
6
votes
1answer
187 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
3
votes
1answer
141 views

Why would googlebot be trying to crawl a URL “https://www.example.com/https://www.example.com/other/things”?

I've been trying to clean up crawl errors on my company's website. In one menu, google bot is trying to crawl every link with an extra-base URL attached. These pages (perhaps obviously) don't exist ...
1
vote
1answer
34 views

How to get a live-steam video into the carousel at the top of the Google search results?

I want whenever a live stream content is available on my website like a sports match then I want to Google to show a live stream preview in its result from the video tag. What I think is I need to ...
3
votes
2answers
252 views

Googlebot says access denied on non-existant resource

Google Search Console is telling me: Access Denied Googlebot couldn't crawl your URL because your server either requires authentication to access the page, or it is blocking Googlebot from ...
-1
votes
1answer
71 views

Maximum time to cache HTTPS URL after HTTP to HTTPS migration?

We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
1
vote
1answer
228 views

Googlebot smartphone redirect

We are using two different domains for our desktop and mobile websites. "www.x.com" and "m.x.com". The users with mobile devices ending on "www.x.com" are (302) redirected to "m.x.com" I have just ...
0
votes
2answers
44 views

Using canonicals for image gallery items on sub pages

I have a sub page on a website that features an image gallery /gallery/gallery-name/ - this page contains a common description about the content of the gallery and thumbnails for the gallery items. ...
1
vote
0answers
315 views

How does StackOverflow know that the referral traffic is coming from Google organic search?

A lot of StackOverflow's results in Google search append a parameter at the end to indicate that the traffic came from Google. I assume this is largely used for internal search volume tracking. ?...
1
vote
1answer
138 views

Can Googlebot crawl JavaScript charts?

I am thinking of using Google Charts API to generate client-side charts on my website. However, my concern is whether Googlebot (for search) will be able to recognize the data?
2
votes
1answer
92 views

Indexed pages dashboard: New or old, which is correct?

There is a discrepancy between the data in my "Index" dashboard in the new search console versus the old one and I am not sure which I should believe. Plausibly of more concern, the old one shows a ...
1
vote
1answer
196 views

After changing the wrong canonical URL, the Google cache still points to the previous canonical URL

We have accidentally used a wrong canonical URL and Google cached the page with the wrong URLs data. When we found this and updated the correct canonical URL, it still points to the same old wrong ...
0
votes
1answer
29 views

delete indexed page from google [duplicate]

I want to delete all page indexed in google and use url seo to index my page again using url seo and delete old urls. I tried <meta name="robots" content="noindex"> but does not work. for ...
1
vote
3answers
796 views

410 or 404 or 301 for SEO for old pages after changing URLs in a site update?

Our site have around 2.1M pages indexed at Google. We made a change to the URL for the indexed pages, for managing international changes, that reflects change in almost all indexed pages. When the new ...
1
vote
2answers
247 views

How often do search bots typically visit a website?

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed. When a search engine returns for ...
1
vote
3answers
553 views

SEO impact of onclick handlers in place of links

I have a question similar to the one posted here. I'm building a React application which has very strict SEO requirements, so I'm wondering if the following approach is an issue: For all of our ...
1
vote
1answer
291 views

Will geolocation DNS blocking some countries hurt search rank?

I am using geolocation DNS to block some countries. Visitors coming from these countries will be directed to another web server with a blank web page with few words and contact email address. I see ...
-1
votes
1answer
102 views

Built a site on a domain that was previously 301 redirected, Google won't crawl it now

I registered a domain and built a site on it. I didn't realize that the domain had a previous owner who 301 redirected the entire domain to another domain of his. I have this domain listed in my ...
2
votes
1answer
148 views

Google search results show “Loading, downloading assets” for all site pages because Googlebot is not fully running JavaScript

I host maps. Each map has its own unique URL. But all these URLs return the exact same small HTML/CSS code, just to show something to the user while the assets and content are being loaded via Ajax. ...
1
vote
2answers
1k views

Googlebot flooding server with requests for junk URLs with random data

I'm having some trouble with GoogleBot. It keeps requesting a random URL that doesn't exist. It is trying to access:www.example.com/index.php/{TOKEN} That {TOKEN} is really random, no idea where it ...
1
vote
1answer
58 views

Website disappeared from Google after redirecting Googlebot to HTML with only the metadata I want indexed

I have a website on which I use a redirect only for Googlebot to a HTML file. It only contains metadata that I want to show in the search results. I have removed everything else from the HTML: <!...
2
votes
1answer
383 views

Can I disallow JS scripts in robots.txt for Googlebot if the website degrades to offer equivelant non-JS functionality?

To my knowledge, Googlebot is currently fully capable of rendering complex SPA applications, and this is recommended as a rule of thumb. The website is currently routed from server side and served as ...
3
votes
1answer
298 views

Does Googlebot execute Google Tag Manager?

I wanted to understand how Googlebot (and other crawlers) crawls my site. Specifically whether it passes a document.referrer and if it maintains localStorage keys, so I implemented a script via Google ...
2
votes
2answers
327 views

AMP pages not getting updated when served from Google's cache

I have an amp page at the following UNS and its already cached by Google. When this paged is served from Google's cache, its showing a 2 day old version of page. Original AMP URL http://www....
2
votes
1answer
100 views

Prevent “Fetch as Google” from creating a wishlist in the database of a Woocommerce site

Can anybody explain why when I 'Fetch as Google' and ask it to index a new page, Google creates a new Wishlist on the back end of the site? Is this normal? The site was built by Yell and we have had ...
1
vote
1answer
149 views

404 or 302 Redirect - what to use for a url which may be used in the future but not available at the moment

My site lists blogs like this example.com/?status=blog&id=number I only have 3 blogs now (id=1, id=2 and id=3). I checked my log files and saw that search bots are manipulating the URLs and ...
3
votes
1answer
268 views

Google doesn't recognize a 404 status code

There are 404 pages with two kinds of response headers (copypasted in full length from Chrome DevTools, Network tab): Response headers: cache-control:max-age=0, no-store content-type:text/html date:...
1
vote
1answer
330 views

Website Home Page URL Not Show On Google Search [duplicate]

If i search like site:www.paka.tv my main homepage URL not showing, but categories of my site pages showing on results. How to show my website URL (www.pakka.tv) on google search.
5
votes
0answers
1k views

Search Console: Server Connectivity Error - Connect Timeout

Over the last 4 days server connectivity errors have started to pop up in my Search Console. Specifically, it is the connection timeout error. I have been talking to my hosting provider and they ...
1
vote
1answer
1k views

How to find out the referrer of Googlebot's crawling URL?

Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links? Is there anything like HTTP Referrer? In detail: Googlebot crawls example.com/no_such_files.php ...
0
votes
0answers
24 views

How to handle the duplicate block of content?

Please refer: https://support.google.com/webmasters/answer/66359?hl=en Googlebot don't like duplicate content and also duplicate block of content repeatedly copy and pasted in many pages. I think, I ...
2
votes
1answer
42 views

Can we have Google crawl but ignore our paginated category pages and prefer our individual post pages in the index?

We have a site with a massive content back-end (50.000+) with, lets say, posts. We want to make these posts available to users, so people searching for the content on Google will find one of these ...
1
vote
0answers
427 views

If “_escaped_fragment_” shows up in Search Console URL Parameters, should you “Let Googlebot decide”?

As far as we know, there should be no links to our site anywhere that include the _escaped_fragment_ parameter used for fetching HTML snapshot versions of our JavaScript-heavy pages. We use the ...
0
votes
1answer
142 views

How frequently should I expect Google AdsBot to visit my site?

I bought paid advertising for my website through an internet advertising company. I noticed that, among all the visits coming from that source, a large amount of them have AdsBot-Google (+http://www....