New answers tagged

1

Most good search engine crawlers (i.e. Googlebot) can process JavaScript, JQuery, etc. and render websites just like a user sees them. These crawlers will be able to crawl your menu and sub-menu items. Use best practices, though, so as not to run into cloaking problems such as using "hidden" tags.


1

Check if the these pages are linked from any other page. Remove those links and mark the errors as fixed. Make sure there are no backlinks on that pages. If there are any backlinks coming to that page make sure you display 404 and submit all those domains to disavow file.


3

RFC 5785 can be paraphrased: If you create a new URL such as robots.txt or favicon.ico that is expected to be at a certain place on every website, you shall henceforth make such a URL start with /.well-known/. Furthermore, you will register all such URLs with the IETF so that there is a big central list of all these URLs. The registry appears to ...


0

Are you displaying ads on your site from Amobee (formerly known as Kontera)? The PTR record for this IP is nat.aws.kontera.com, suggesting that it is a crawler looking for your page's content in order to determine what ads may be relevant to that URL. If you've loaded the Kontera JavaScript on login-protected pages, then you will find that they are crawled ...


1

While the Googlebot could be at fault here as you ask in the comments it is unlikely. Based on your question and your follow up comments it sounds as though your server may have malware or a virus. I would recommend running antivirus toolkits, and malware scanners, and potentially even reloading your site from a backup to deal with any changes that may have ...


0

The Googlebot doesn't add query string to the end of pages it indexes. There is one of two options here, either the Google SERP has a glitch on a few web servers which is adding that query string to the end of the URL (unlikely) or two two the links on your page are being either rendered in a malformed way by Code Igniter or being altered at run time by ...


0

Crawling for the Internet Archive is done both by Alex and by the Internet Archive's own crawlers. Support for the Crawl-Delay directive in the robots.txt file is vairly hap-hazard between the two due to the directive not being part of the official robots.txt standard. In addition the way both companies treat the Crawl-Delay directive when they do accept it ...


1

Just tested it and Bing found page with all text build purely by JavaScript. Google also finds it of course.


2

There's a fairly small range of mobile device viewport sizes. Everything else will be regarded as "desktop". And: When the meta viewport element is absent, mobile browsers default to rendering the page at a desktop screen width (usually about 980px, though this varies across devices). [source] Googlebot Smartphone is basically a mobile browser, so ...


0

In your robots.txt file add a crawl delay directive. User-agent: * Crawl-delay: 1 Any IPs that don't obey this should be blocked. Anything in the 66.249 range is Google in my understanding.


0

If there's code showing min-width values of 1200px+ the crawler can detect and read that content just as it does for the mobile format content.


1

Google has written a blog post at https://webmasters.googleblog.com/2012/08/website-testing-google-search.html on the best way to handle indexing with A/B testing and multi-variant testing such as what you are doing. The first thing to note in the article is that cloaking is still not allowed even during testing. It doesn't look as though any of their ...


0

What I finally did was to always block the price to bots. Used a code similar to this: function _bot_detected() { if (isset($_SERVER['HTTP_USER_AGENT']) && preg_match('/bot|crawl|slurp|spider/i', $_SERVER['HTTP_USER_AGENT'])) { return TRUE; } else { return FALSE; } } More info at: http://stackoverflow.com/questions/677419/how-to-...


0

The reason for this is that Google detected the term .NET in the word and .NET is a well established keyword for the .NET framework. As such it would have made the mistaken decision that the word was simply missing a space and added it as a spelling correction. Not much that can be done about fixing this though.


0

There is nothing to be concerned about with this as it commonly shows up with secured applications on the internet. A user accesses a link through their bookmarks bar or through their browser history to go back to a page that they want to go back to and because their session has expired they are redirected to the login page to login first. No worries here. ...


0

It's possible those accesses are page redirects or supplementary document files such as images. Perhaps each of your pages has a large number of assets attached to it that must load in order for the page to be complete, such as external javascript files, CSS files, and image files. Like closetnoc said, you can try google search console to limit crawl rate, ...


1

The example IP addresses are Google alright! However, it should be no more than 1 request per 2 seconds though that may have changed in the past few years. For the record, Google has always behaved rather well and has always stayed within the 1 request per 2 seconds rule. Still, it is not uncommon for small flurries of requests throughout the day and maybe ...


1

There are two ways you could handle it - unfortunately both are not ideal solutions... You could remove the microformats markup from your pages - but this will affect the people both in the US and outside looking at your results You could create different versions of your website for different countries to be indexed in results for these countries. This ...



Top 50 recent answers are included