Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

68 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
1answer
854 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
5
votes
0answers
1k views

Search Console: Server Connectivity Error - Connect Timeout

Over the last 4 days server connectivity errors have started to pop up in my Search Console. Specifically, it is the connection timeout error. I have been talking to my hosting provider and they ...
4
votes
0answers
146 views

isAccessibleForFree and paywalled content delivered to Googlebots

I've got a site with some paywalled articles. We want to ensure that Googlebots can index the paywalled content, but we don't want to be accused of cloaking. I've got two types of articles: Premium ...
4
votes
0answers
70 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
3
votes
0answers
54 views

Pantheon Dev Site Trapped in Google Index (with injected robots.txt)

We have a bizarre scenario I'd like opinions on. A site at example.com is being built on pantheon, using their example.pantheon.io staging / dev URL*. Pantheon's development platform injects a ...
3
votes
0answers
1k views

URL will be indexed only if certain conditions are met

I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
2
votes
0answers
15 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
2
votes
0answers
23 views

URL crawlable in old Google search console, but blocked in new

Google stopped crawling images for our site, when we moved images to cdn subdomain. Old search console states image is crawlable, but new one states it is blocked by robots.txt (for same link). our ...
2
votes
1answer
309 views

Crawled page's screenshot is blank in the Google Search Console

I set up a SPA page through Angular, and achieved the effect of Server Side Render through Rendertron, In the facebook debug tool, I could get my dynamically generated Meta data, but the preview of ...
2
votes
0answers
376 views

Disqus comments cause “clickable elements too close together” in Google Search Console mobile usabality report

I just got an email from Google that they found some problems on my site: clickable elements too close together Content wider than screen I fetched problematic URLs as Googlebot and noticed that it ...
2
votes
0answers
36 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
2
votes
0answers
167 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
2
votes
0answers
239 views

Google cache does not have prerendered _escaped_fragment version of page, rather it is showing another page altogether

I have read multiple questions around it but still could not find a solution. We have a page in the form https://example.com/trending. We are using Prerender using the ?_escaped_fragment url to show ...
2
votes
0answers
255 views

Google Blocked Resources showing incorrect data

In Google Webmaster Blocked Resources tab it is showing 500+ blocked resource (like .js and .css), but when I tested those URL in robots.txt Tester it is showing Allowed, then why not Google bot is ...
2
votes
0answers
135 views

Googlebot not indexing https pages with 301 redirect

I recently installed a new SSL in my blog and i forced 301 redirect on all http to https pages. It took like 2 months for google to index https content, infact i am seeing a small boost in rankings. ...
2
votes
0answers
320 views

access.log requests from UA “Google favicon”…? What do these mean?

I have a bunch of these in my nginx log file, on a range of URLs: on 66.102.9.23 - - [12/Nov/2015:10:40:25 +0000] "GET /myadminfolder/page.html HTTP/1.1" 200 1247 1800 "-" "Google favicon" "...
2
votes
0answers
77 views

Search engine ranking of paginated categories, subcategories and tag pages

Suppose we have an E-commerce domain with the following structure: domain/items - contains all items. domain/items/category - contains a narrower spectrum of results than above. domain/items/category/...
2
votes
0answers
660 views

google displays wrong language on multi language website because it takes cookies into consideration

I have a multiple language website that uses both subdomains and cookies to display the right language. Here is the logic on the website: de subdomain will display the German version fr subdomain ...
2
votes
0answers
914 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
2
votes
1answer
452 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
1answer
22 views

What is “ineedthispage”? Where is it coming from? It is killing the server

I am seeing the following in my "access_log": ip.address.output.here - - [12/Dec/2019:15:17:07 +0000] "GET /pathto/article/15104133/?ineedthispage=yes&ineedthispage=yes&ineedthispage=yes&...
1
vote
0answers
19 views

What means the “average Bytes” metric in logfiles?

I'm currently studying the logfiles - namely those from Googlebot access. I realized, that the average Bytes metric is in the majority of cases (urls) 10-20% of the page size. Many questions were ...
1
vote
0answers
15 views

Can I Prevent Adsense Loading For GoogleBot?

I have been working on lots of performance updates to my site and I have gotten pretty much every page type to load in under 2 seconds. Most single post pages are coming in at around 1.5 seconds.When ...
1
vote
0answers
25 views

Can alternate URLs for hreflinks be in different XML sitemap files when using <xml:link rel=“alternate”

Regarding the XML sitemaps (that use alternative languages), in this page it is specified the following: Common Mistakes Here are the most common mistakes with hreflang usage: Missing ...
1
vote
0answers
23 views

Is there a way to use js templating with wordpress without impacting seo or ux?

I have one JSON file that contains the text content for my hundreds of Wordpress posts. I am using templating, mustache.js, to insert the text content and render the posts. This saves me a lot of time,...
1
vote
0answers
38 views

404 errors as a result of …/false urls

I have been reviewing 404 error logs on a WooCommerce site I have been working on (logs held within the website server) and there are a lot of requests to product pages which have /false on the end of ...
1
vote
0answers
43 views

Crawl anomaly caused by googlebot crawling secure_link mp4 files with md5 and expires parameters

Googlebot continues to crawl mp4 files in the format www.example.com/video/12345.mp4?md5=2029zanlkdain312on&expires=1523327783 which has obviously expired (only valid timestamp+30minutes, after ...
1
vote
0answers
32 views

Huge differences between webmaster tools' crawl stats and apache logs

We noticed huge difference in the metrics. For example, last saturday : According to Webmaster Tools : 6,082 pages crawled 76,119 kilobytes downloaded 1,550 ms in average to download a page Now ...
1
vote
1answer
47 views

Shrinking site's size by removing pages from sitemap and from internal links

We are an eCommerce site with about 50,000 pages. We decided to focus on about 2,000 so google will crawl our site better. We planning to remove 48,000 pages from the sitemaps and from the site links, ...
1
vote
0answers
28 views

How to verify that Googlebot can access the URL of a video thumbnail?

I need to check if Googlebot can see my image thumbnails. When I use the URL Inspection Tool, Google cannot see my images in the page. In the rendered screen shot, it displays a black box instead of ...
1
vote
0answers
109 views

Googlebot intermittently fails to download ressources from AWS CloudFront CDN

I have a weird issue and after a few weeks of investigation I think I may have found the cause : Cloudfront & Googlebot. Context : I launched a French website that compares and combines all ...
1
vote
0answers
31 views

When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?

I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl. At the ...
1
vote
0answers
40 views

Does Google index ajax data that load after 1 second by setTimeout?

I use load Javascrpt and use Ajax for load content. And i should use setTimeout for run Ajax load function after 1 or 2 second(s). Does Google index and follow this content?
1
vote
0answers
16 views

Download link expired, going back to software presentation page, what redirect method to use?

When someone (user or bot) reaches an old download link like /download/yellow-wallpaper.zip?key=some-expired-download-key , user is sent back to file details page, e.g /yellow-wallpaper.html, where ...
1
vote
0answers
97 views

Angular2 - Google indexing loader instead of redirected home page

I'm working on a multilanguage app in Angular. Universal is already setup and working but I have issue with googlebot indexing the page. When user navigates to url https://example.com only loader ...
1
vote
0answers
35 views

Could using the Google URL removal tool for a specific page also have removed the home page?

I noticed today my Homepage URL is not on Google Search Result anymore. Firstly for specific keyword where i ranked #1 than for domain itself. it is my brand keyword which I get most traffic from. I ...
1
vote
0answers
170 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
35 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
1
vote
0answers
60 views

What is the correct response to Googlebot for a search_term_string from site search markup with _escaped_fragment?

I have a AJAX web online-shop with hashbang URL and added SiteLinks as JSON-LD. I follow this introduction. { "@context":"http://schema.org", "@type":"WebSite", "url":"https://...
1
vote
0answers
22 views

which status code should be returned for null parameter

having a website with many urls in js code (ajax calls), the google bot is calling these but without the necessary parameters atm 200 code is returned and the html for the error page, and this is ...
1
vote
0answers
334 views

How does StackOverflow know that the referral traffic is coming from Google organic search?

A lot of StackOverflow's results in Google search append a parameter at the end to indicate that the traffic came from Google. I assume this is largely used for internal search volume tracking. ?...
1
vote
0answers
453 views

If “_escaped_fragment_” shows up in Search Console URL Parameters, should you “Let Googlebot decide”?

As far as we know, there should be no links to our site anywhere that include the _escaped_fragment_ parameter used for fetching HTML snapshot versions of our JavaScript-heavy pages. We use the ...
1
vote
0answers
475 views

How to stop Bot generate Traffic in Wordpress?

My WordPress website recent facing a big problem. Spamming problem. Using a plugin i detect that every single second here coming huge number of traffic which is fake and send by BOT. The traffic also ...
1
vote
0answers
472 views

Does Google crawl AJAX API calls?

We have been monitoring our logs for Googlebot and there are many requests to API calls (requests via AJAX) which are used to fetch the page and not the complete page url. How does Google ...
1
vote
0answers
56 views

Dealing with Google spidering JavaScript incorrectly

Google spiders JavaScript now to look for possible URLs. This is mostly a good thing, but occasionally it gets it wrong. Is there any way to hide possible URLs in JavaScript from googlebot? For ...
1
vote
0answers
153 views

Submit and Index page/url changes in Google Search Console

I made some changes to a page and it's url, and have configured my web server to redirect from the old to the new. After fetching the old and new url's in the search console the results appear as ...
1
vote
0answers
695 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
1
vote
0answers
165 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
1
vote
0answers
95 views

I still get “Googlebot can't access your site” messages after removing a site from Google Webmaster Tools

I had a client who moved their site a while back to another developer. When they moved it they took down the site and put up a coming soon page ( which is currently what's there). Googlebot warned me ...
1
vote
0answers
181 views

Why does Google keep hitting only one MP3 file?

I have a bunch of Mp3 files on my site, and all if them are in the sitemap to allow search engines that log mp3 files to know about them. For some reason, every day or two, Google keeps hitting the ...