Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
1
vote
1answer
40 views

Linked desktop and mobile pages- does SEO rank of mobile website even matter?

The question is somewhat similar to this post. The only difference is, we have desktop version as https://www.example.com/ and mobile version as https://www.example.com/m/. The desktop page has mobile ...
1
vote
1answer
52 views

Link to product detail pages without allowing product list pages with many filter variants to all be indexed

I have a website for managing trading card game (TCG) collections which is built around a central search engine/filter system. This filter system is located on the cards page which lists all cards ...
5
votes
1answer
862 views

Google crawler and dynamic websocket website

I have a a big problem that I'm only considering after I have finished development of a webpage. My page is written with only a skeleton HTML. All the actual content of my site is sent through ...
2
votes
1answer
796 views

Why is Google crawling non-existant URLs?

I can see in live traffic of my wordpress website that goggle bot crawl non existing pages. www.example.gr/search/search-results/password-reset%252Fpassword-reset/password-reset%252Fpassword-reset%...
0
votes
1answer
91 views

How can I prevent the Googlebot from indexing duplicate URLs?

I have duplicate URLs for each ad page. For example: https://example.com/ads/2161/%D9%84%D9%83%D8%B2%D8%B3_GS350.html https://example.com/2161/ I used the second URL for Twitter and Instagram posts....
0
votes
1answer
575 views

Can i do a 301 redirect for the Googlebot Only?

Lets say a have a webpage called example.com, and a have a subdomain called sellingpage.example.com I use sellingpage.example.com as a landing page for all my paid traffic, this is a landing page ...
1
vote
1answer
2k views

Firebase hosting, sitemaps and robots.txt

i tried searching on internet about this question and how Firebase handle indexing and how owner of the content running on Firebase hosting can influence crawling by Google bot. Personally, i am ...
3
votes
1answer
345 views

AdsBot-Google visiting domain which no longer has advertisements

During analyzing traffic generated by bots to our website, we found the strange issue. We found that AdsBot-Google crawling domain which we stopped to advertise in the middle of 2016. Right now ...
4
votes
1answer
3k views

Does changing from HTTPS to HTTP kill my backlinks?

I had only my main page changed to HTTP to test out if I want to change from HTTPS to HTTP. All my backlinks are to my HTTPS site. Will I lose all of my backlinks and have to start over when switching ...
0
votes
1answer
60 views

If i redirected my website from “HTTPS” to “HTTP”, will i still be getting a ranking boost?

Google announced that going HTTPS will give you a ranking boost. So i bought SSL certificate to boost my rank and redirected all "HTTP" to "HTTPS" from .htacess... then i realized that it's not ...
1
vote
1answer
169 views

Google Search Console - Scan Request does not work

I have a problem with Google's Search Console. When I "Fetch as Google" and status is Complete, I click on "Request Indexing", fill in the Captcha and Submit. Nothing happens after i click on Submit,...
3
votes
1answer
393 views

Rule out third party scraping, but allow Google crawling

How to make scraping of own content through wget, httrack etc. impossible, but allow crawling through googlebot? This should be done without showing to googlebot other content, as to other user ...
2
votes
2answers
505 views

How to Remove URL from Google Cache which is not available in Sitemap?

I have a site Hot Deal INDIA. I have added around 1200 Links in my sitemap. Around all the Links are cached by google, so there is no problem with Caching. But when I checking Google Index-> Total ...
3
votes
1answer
558 views

What is the “google-xrawler” user agent used for?

I'm getting over 100 requests per day from user agent google-xrawler. The requests are all for the same page. Some of the IP addresses are known to be Google IPs, although they don't have valid ...
5
votes
4answers
153 views

How to make Google index users posts when there are no direct links and the posts are loaded with infinite scroll?

In a website that has posts system similar to Facebook, where the user has option to: Keep his posts private (needs login to view) Make them public (anyone can see them) In case #2 I want to allow ...
2
votes
1answer
320 views

Will Google automatically re-index my page when I remove the 'noindex' tag

On my request, our company has implemented a function in our website that adds <meta name="robots" content="noindex"> to blog articles and whitepaper pages that haven't been published yet. ...
4
votes
2answers
111 views

Least SEO-disruptive HTTPS migration strategy

We are moving a website from HTTP to HTTPS and I have read Google's support article which is quite good: https://support.google.com/webmasters/answer/6033049 As I understand, 301 redirects may ...
1
vote
0answers
1k views

Angular2 : 'Fetch as Google' doesn't render page [closed]

I am working on Angular2 based web app. I used Angular CLI to generate app and then to build it for prod. I have hosted website on AWS S3 & Cloudfront. When I use 'Fetch as Google' tool from the ...
2
votes
0answers
254 views

Google Blocked Resources showing incorrect data

In Google Webmaster Blocked Resources tab it is showing 500+ blocked resource (like .js and .css), but when I tested those URL in robots.txt Tester it is showing Allowed, then why not Google bot is ...
1
vote
1answer
416 views

Avoid “Cloaking” with GeoIP Redirects

I'm currently working on a US-based site that consists of the following: National Landing Site (call this WWW, i.e. www.site.com) 50 regional subdomains, one for each state (i.e. nevada.site.com, ...
0
votes
1answer
266 views

Does adding CDN stops Google from crawling?

I want to add Cloudflare free CDN to my site but worried about whether Google bots will be able to crawl my site later on or not.
0
votes
1answer
6k views

How block only Yandex bot

Can you show me how will look robots.txt when I block only yandex bot, allow Google bot and block Yandex bot.
4
votes
2answers
98 views

GoogleBot is crawling my datepicker to inifnity

Our website is an agenda of events taking place in our city. We have a datepicker for users to look at events scheduled in the next week, next month, etc... We recently discovered that GoogleBot is ...
1
vote
1answer
28 views

How do you properly SEO-tag an app that is only a catalogue of images?

I am making a little helper app that mostly links a lot of graphs and images. There is also some text explaining a couple of things, but more importantly that text will probably never (or very rarely) ...
1
vote
1answer
286 views

Site not indexed after moving site from WordPress to Magento

I moved a site from WordPress to Magento. But now Google webmaster does not index the site. I have submitted the sitemap, but it remains pending. I also have a robots.txt file which has the following ...
2
votes
2answers
853 views

Will Google respect changing the <meta> description with JavaScript?

I am building a single-page-application/SPA that uses JavaScript to control all the pages. Naturally, this means only one meta description embedded in HTML, even though there are many different pages ...
2
votes
3answers
1k views

Google not crawling my site (robots.txt error)

I'm currently performing SEO for my client's project. I'm kind of new to this, so please bear with me. I've read a lot mixed reviews on inclusion of robots.txt (some say it's good to include even if ...
3
votes
1answer
147 views

How to check whether the infinite page scroll data is crawled and indexed by google or not?

I am having a music platform website where at a time 20 songs are visible right now then after scroll the other 20 are visible. So how can I confirm whether the songs which are coming after scrolling ...
3
votes
1answer
58 views

SEO: Moving domain without having access to the old domain

The mother of my girlfriend has a contract with a company that provides her website and domain. The contract costs 360€ per year and it expires on May 1st. The website and domain will be probably ...
1
vote
1answer
77 views

Google bot requesting URLs from old 301 redirectd site and then reporting a bunch of 404 errors

I'm reviewing some errors in Google Webmaster Tools / Search Console. I'm getting a lot of 404 errors for pages that don't exist such as: example.com/page12.php example.com/page111.html example....
1
vote
0answers
36 views

Sudden traffic fall related with DNS Error [closed]

We juts notice a real time traffic fall in our website , from 1500 visits to 600 real time. This is extremely rare, I do a check on the behavor from the past weeks and we notice , that GoogleBot ...
2
votes
0answers
161 views

Sudden Spike in mobile app crawl errors - Search console [closed]

We are seeing something weird in our mobile app crawl dashboard on the search console. Up until 3rd Feb, we had only 3 crawl errors but suddenly that number is now 107k and it went up between 3rd ...
3
votes
2answers
404 views

Google Search Console: 404 errors on existing pages

There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
5
votes
1answer
3k views

How do I stop Google from indexing AJAX calls within Javascript?

We are getting a number of odd results come up on Google EG: http://www.somedomain.com/somepage/AjaxFunction.aspx?stuff=XXX&other=XXX When I looked at "somepage" the ajax function is not ...
2
votes
1answer
67 views

Does Search Console 404s from relative path links affect Crawl Budget?

Search Console is reporting 404s for URLs that don't exist on my site. However, GoogleBot seems to be crawling a URL by guessing a page based off of relative paths in my links. For example: I have a ...
2
votes
0answers
135 views

Googlebot not indexing https pages with 301 redirect

I recently installed a new SSL in my blog and i forced 301 redirect on all http to https pages. It took like 2 months for google to index https content, infact i am seeing a small boost in rankings. ...
2
votes
1answer
634 views

“Is robots.txt blocking important pages?” in Search Console regarding static folder

This is the contents of my robots.txt file: User-agent: * Disallow: /static/* Disallow: /templates/* Disallow: /translations/* Sitemap: https://xx.xxx.com/sitemap.xml However I don't know why I am ...
2
votes
1answer
102 views

robots.txt for website hosted in a subdirectory

I have 2 website which is hosted in a shared hosting. 1st website example.com hosted in a root directory as /public_html/ 2nd website example2.com hosted in a sub-directory directory as /public_html/...
1
vote
1answer
52 views

Google Confused (Two Sites on One Server)

I have two sites / domains (https://www.pcfixes.com & https://www.flcleaner.com) on same server. Having tweaked the nginx profiles for both sites, I believe Gbot went on PCfixes.com and was ...
1
vote
1answer
122 views

What is the Google sandbox?

What is the Google sandbox? What happens in the Google sandbox as far as SEO is concerned? When I submit my site to Google Seach Console, what is the process to move from the sandbox to the Google ...
0
votes
2answers
38 views

Landing page and development site

We currently have a Landing page for our website with no navigation or links only a contact form. However, I am adding content to other pages on the LIVE site that only I can get to for now. Will ...
3
votes
1answer
806 views

Can Googlebot crawl reCAPTCHA v2?

I'm building a website that contains certain forms protected by reCAPTCHA v2. Can Googlebot crawl content protected by its own reCAPTCHA v2?
8
votes
1answer
367 views

Will implementing HSTS prevent Googlebot from seeing 301 permanent redirects from HTTP to HTTPS?

I did a migration from http:// to https:// for a big site with more than 2 million of URLs indexed on Google. As others have mentioned I'm also experiencing a bit of fluctuation on organic traffic (-...
2
votes
1answer
59 views

Will the Google web crawler stop crawling and indexing pages that take longer than 2.8 seconds to load?

I just received this statement from my client: our PR company works on ads for us and they spoke to a representative from Google . they said after 2.8 seconds the Google web crawlers stop ...
5
votes
1answer
439 views

My site fails to do content negotiation with Googlebot using the Accept-Language header

HTTP offers Content Negotiation (by reacting on the Accept-Language request header) to deliver translated representations of a resource under one URI. In January 2015 Google finally implemented this ...
2
votes
1answer
2k views

How to block Googlebot from accessing one specific page

I am getting spammed by this IP: 66.249.79.70, and this website is telling me that it is from Google: https://ipinfo.io/66.249.79.70 It is filling one of my input boxes and keeps posting. I want to ...
0
votes
1answer
764 views

Googlebot not respecting HTTP basic auth

I have basic auth set up and it has always worked. Suddenly Google started crawling my pages. The auth is still there (I have checked it using different browsers). I am at a loss how it's possible. ...
1
vote
3answers
104 views

Will Googlebot take text appearance into account?

The Google bot will rank words by the surrounding tag. A text between h1 tags will have more score than the content of another h{n} element or the p element, of course. However, is the Google bot ...
1
vote
1answer
496 views

Will Google index text in inline SVG that was rendered through JavaScript?

As far as I know the Google bot will index text inside of inline SVG, this has been answered already. However I don't know this applies to text that was generated through JavaScript. Here is an ...
0
votes
1answer
759 views

SEO issues with Elm?

Do search engines, particularly Google, render JavaScript created from transpiled Elm code when crawling? Can they follow links, even internal that modify the existing page?