Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
3
votes
3answers
83 views

Is Google Bot indexing the content of a textarea?

For one of my projects that provides pages with different content in a textarea, I need to make sure that Google Bot is indexing the content of those textareas. I could not find an answer online. ...
2
votes
1answer
657 views

Google started to reports root url blocked by robots.txt - but it's not -

My website has been running for multiple years without problem, however I recently noticed my root url on Google does not show any description/title. The Webmaster Tool reports this error: Crawl ...
1
vote
0answers
10 views

Why the Download Pages of our products are not indexed by Google? [duplicate]

I have a question about pages indexed, hope you can help me out, thank you in advance. Recently, we found our French site's two download pages are not indexed by Google. The URL of the French site ...
1
vote
1answer
324 views

Security Issue - Harmful content

I am trying to solve an issue with my website already for some time but without any luck. Google has detected somewhere in my website harmful content (Malware). But the problem is, I could not find ...
1
vote
0answers
96 views

Angular2 - Google indexing loader instead of redirected home page

I'm working on a multilanguage app in Angular. Universal is already setup and working but I have issue with googlebot indexing the page. When user navigates to url https://example.com only loader ...
5
votes
1answer
171 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
3
votes
1answer
53 views

Status code vs. Response Header Status: what is the rule for bots?

Assuming we have a page, which has as Status Code 200 OK and as Response Headers Status HTTP/1.1 410 Gone, like on image: What is the behavior of bots / Googlebot? To crawl/index or not?
1
vote
1answer
84 views

Googlebot refuses to render more than index.html in Ember app

I am using Ember 3.0.0 to create a booking site. Title and meta tags are handled via ember-cli-meta-tags and seem to be rendered correctly when pages are opened in the web browser. However, Googlebot ...
1
vote
2answers
362 views

Does Google's own bot contribute to my Google Maps API fees?

Does Googlebot, when crawling pages which display maps using the Javascript API, cause a billable API usage on each page request?
0
votes
2answers
96 views

Stop Google from accsesing pages without robots.txt

Is it possible to get Google to stop crawling certain pages without using robots.txt? My understanding is robots.txt can block Google from accsesing a page, but if using noindex (meta or header) ...
1
vote
0answers
35 views

Could using the Google URL removal tool for a specific page also have removed the home page?

I noticed today my Homepage URL is not on Google Search Result anymore. Firstly for specific keyword where i ranked #1 than for domain itself. it is my brand keyword which I get most traffic from. I ...
1
vote
1answer
74 views

Sitewide A/B or Sanity Test

We are switching to a new Wordpress template and need some help. We will monitor if the test version improves the sites performance (CTR, Revenue) Technically we will: Split A Records by 50% for ...
0
votes
1answer
21 views

Does Google take into account page title changes due to JavaScript? [duplicate]

One of my pages is set up so that the initial page title is just some placeholder title, and it changes after an AJAX call. Here's what the process looks like: Initial page load AJAX call to ...
1
vote
0answers
157 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
2
votes
1answer
202 views

Google AdWords ads are disapproved because “destination not working”

We have been having an issue setting up adwords for one of our clients and despite multiple phone calls to Google we cannot get an answer out of them as to why our ads are being disapproved. ...
1
vote
2answers
83 views

SEO impact of changing versions (and names) of external CSS

A website of mine uses a WordPress plugin for CSS minification. The plugin regenerates the CSS file on every update to the style-sheet. When it does, the plugin also changes the version and hence name ...
1
vote
0answers
35 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
1
vote
0answers
55 views

What is the correct response to Googlebot for a search_term_string from site search markup with _escaped_fragment?

I have a AJAX web online-shop with hashbang URL and added SiteLinks as JSON-LD. I follow this introduction. { "@context":"http://schema.org", "@type":"WebSite", "url":"https://...
1
vote
0answers
11 views

My most important pages does not get indexed, google search console [duplicate]

I have a website with blog articles. I added it's sitemap to google search console and it say that the sitemap has been processed. However, half of my articles (the important pages) have not been ...
2
votes
0answers
36 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
1
vote
0answers
38 views

Google cache is not showing up for a page despite trying twice to use the fetch and render tool [duplicate]

I did Fetch and Render twice on a particular page but the cached version of the page is still not showing up on Google. I did a Fetch and Render on other posts on the site and they all showed cached ...
0
votes
0answers
45 views

Any relation between adsense lack of impressions and incorrect sitemap?

I have recently submitted a sitemap in search console for at least 200 new pages on my site. However, I did a mistake by not including a slash ("/") at the end of each url. But the pages without ...
2
votes
2answers
307 views

CloudFront - disable access to distribution via d1234.cloudfront.net

I have a CloudFront distribution with a configured alternative name (e.g. cdn.example.com). Is it possible to restrict access to the CDN via d1234.cloudfront.net name? I don't want it to be indexed ...
2
votes
1answer
522 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
0
votes
2answers
347 views

Status Code 500 to Googlebot but url working

I have a URL on my site that is working perfectly. I have the robots with the index and follow, but when I Fetch as Google in the WMT says Unreachable and gives me this message in the crawl errors ...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
4
votes
0answers
70 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
4
votes
1answer
386 views

Dynamic content using .append() or .innerHTML() and SEO

I was looking for some clarification on something I am looking to implement, I would like to know how a Google Crawl will see this and if it would be detrimental to SEO. On my webpage I have an empty ...
4
votes
1answer
63 views

Does Google follow and count a backlink that goes through a JavaScript location.href redirect page?

example.com has a link on their website which is on same domain like, <a href="https://example.com?ref=mywebsite.tld">Reference</a> example.com?ref=mywebsite.tld doesn't redirect with ...
0
votes
0answers
16 views

Removing Paginated URLs from Google Search [duplicate]

I have a website running on Django, Python. I'm using django pagination so that users can click next & see further content. For example, my website is example.com, so when user requests new page ...
2
votes
0answers
165 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
1
vote
0answers
22 views

which status code should be returned for null parameter

having a website with many urls in js code (ajax calls), the google bot is calling these but without the necessary parameters atm 200 code is returned and the html for the error page, and this is ...
1
vote
2answers
101 views

How to get Googlebot to crawl my website faster to pick up my new meta description

I've tried to put an new meta description for my website, but when I look it up I get the wrong description. After some research I found out that I need Google to crawl my site. The cache:website ...
6
votes
1answer
200 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
3
votes
1answer
145 views

Why would googlebot be trying to crawl a URL “https://www.example.com/https://www.example.com/other/things”?

I've been trying to clean up crawl errors on my company's website. In one menu, google bot is trying to crawl every link with an extra-base URL attached. These pages (perhaps obviously) don't exist ...
1
vote
1answer
35 views

How to get a live-steam video into the carousel at the top of the Google search results?

I want whenever a live stream content is available on my website like a sports match then I want to Google to show a live stream preview in its result from the video tag. What I think is I need to ...
3
votes
2answers
256 views

Googlebot says access denied on non-existant resource

Google Search Console is telling me: Access Denied Googlebot couldn't crawl your URL because your server either requires authentication to access the page, or it is blocking Googlebot from ...
-1
votes
1answer
71 views

Maximum time to cache HTTPS URL after HTTP to HTTPS migration?

We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
1
vote
1answer
237 views

Googlebot smartphone redirect

We are using two different domains for our desktop and mobile websites. "www.x.com" and "m.x.com". The users with mobile devices ending on "www.x.com" are (302) redirected to "m.x.com" I have just ...
0
votes
2answers
45 views

Using canonicals for image gallery items on sub pages

I have a sub page on a website that features an image gallery /gallery/gallery-name/ - this page contains a common description about the content of the gallery and thumbnails for the gallery items. ...
1
vote
0answers
330 views

How does StackOverflow know that the referral traffic is coming from Google organic search?

A lot of StackOverflow's results in Google search append a parameter at the end to indicate that the traffic came from Google. I assume this is largely used for internal search volume tracking. ?...
1
vote
1answer
154 views

Can Googlebot crawl JavaScript charts?

I am thinking of using Google Charts API to generate client-side charts on my website. However, my concern is whether Googlebot (for search) will be able to recognize the data?
2
votes
1answer
93 views

Indexed pages dashboard: New or old, which is correct?

There is a discrepancy between the data in my "Index" dashboard in the new search console versus the old one and I am not sure which I should believe. Plausibly of more concern, the old one shows a ...
1
vote
1answer
201 views

After changing the wrong canonical URL, the Google cache still points to the previous canonical URL

We have accidentally used a wrong canonical URL and Google cached the page with the wrong URLs data. When we found this and updated the correct canonical URL, it still points to the same old wrong ...
0
votes
1answer
31 views

delete indexed page from google [duplicate]

I want to delete all page indexed in google and use url seo to index my page again using url seo and delete old urls. I tried <meta name="robots" content="noindex"> but does not work. for ...
1
vote
3answers
834 views

410 or 404 or 301 for SEO for old pages after changing URLs in a site update?

Our site have around 2.1M pages indexed at Google. We made a change to the URL for the indexed pages, for managing international changes, that reflects change in almost all indexed pages. When the new ...
1
vote
2answers
301 views

How often do search bots typically visit a website?

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed. When a search engine returns for ...
1
vote
3answers
663 views

SEO impact of onclick handlers in place of links

I have a question similar to the one posted here. I'm building a React application which has very strict SEO requirements, so I'm wondering if the following approach is an issue: For all of our ...
1
vote
1answer
304 views

Will geolocation DNS blocking some countries hurt search rank?

I am using geolocation DNS to block some countries. Visitors coming from these countries will be directed to another web server with a blank web page with few words and contact email address. I see ...
-1
votes
1answer
103 views

Built a site on a domain that was previously 301 redirected, Google won't crawl it now

I registered a domain and built a site on it. I didn't realize that the domain had a previous owner who 301 redirected the entire domain to another domain of his. I have this domain listed in my ...