New answers tagged

1

I've noticed that Google does the same thing to some of my sites, but they don't actually index the content they've crawled. Read here: https://support.google.com/webmasters/answer/35308?hl=en According to Google, using that in the robots.txt prevents them from indexing the content, it doesn't say that it will stop crawling it entirely.


0

Most good search engine crawlers (i.e. Googlebot) can process JavaScript, JQuery, etc. and render websites just like a user sees them. These crawlers will be able to crawl your menu and sub-menu items. Use best practices, though, so as not to run into cloaking problems such as using "hidden" tags.


1

Allow: /services/online.html Disallow: /*?dir= The most specific rule (based on the length of the path argument) wins when resolving Allow: / Disallow: conflicts - regardless of the order of the directives in the file. So, for the given URL, the first rule wins because it is the most specific path that matches the requested URL. To resolve this you can ...


0

It appears that you are looking forward to block URL with appearing with sorting parameters. The best thing to do is add canonical tags. It the recommended method Coming to the above, The robots test in webmaster often takes time to update and display the new data add in live robots.txt page. Please confirm that it has been updated.


0

GET requests could be seen as unique url. So any bot can access get data more quickly as unique url and judge contents. You can also cross verify all GET ajax calls againsts request url and desired format.


3

Googlebot only performs POST requests under very limited circumstances where it is believed by the Googlebot that it is safe and appropriate. Google takes precautions to avoid performing tasks on a site that could result in executing an unintended user action and Google making POST requests is for crawling purposes only to index what the end user would see. ...


1

This is because what you had read is not fully true, as Google says: Google tries to associate framed content with the page containing the frames, but we don't guarantee that we will. Or it was before 2011, when Matt Cutts from Google said: Googlebot keeps getting smarter. Now has the ability to execute AJAX/JS to index some dynamic comments ...


0

This question was answered by the person who asked it in a comment: Well I found the answer after some research. Google states that hidden content on your page is probably not critical for that particular page. This includes click to expand content. If you want this content indexed you should make sure it's visible for users when they land at that page. ...


0

I think I may be able to follow what you are talking about. Based on your question and your comments you seem to be indicating that you are concerned that your website will appear to be a Philippine website as it is being managed by a Philippine webmaster with a Philippine IP address and you are concerned about this by using the examples... I have seen ...


0

The Googlebot doesn't add query string to the end of pages it indexes. There is one of two options here, either the Google SERP has a glitch on a few web servers which is adding that query string to the end of the URL (unlikely) or two two the links on your page are being either rendered in a malformed way by Code Igniter or being altered at run time by ...


2

There's a fairly small range of mobile device viewport sizes. Everything else will be regarded as "desktop". And: When the meta viewport element is absent, mobile browsers default to rendering the page at a desktop screen width (usually about 980px, though this varies across devices). [source] Googlebot Smartphone is basically a mobile browser, so ...


0

In your robots.txt file add a crawl delay directive. User-agent: * Crawl-delay: 1 Any IPs that don't obey this should be blocked. Anything in the 66.249 range is Google in my understanding.


0

If there's code showing min-width values of 1200px+ the crawler can detect and read that content just as it does for the mobile format content.


1

Google has written a blog post at https://webmasters.googleblog.com/2012/08/website-testing-google-search.html on the best way to handle indexing with A/B testing and multi-variant testing such as what you are doing. The first thing to note in the article is that cloaking is still not allowed even during testing. It doesn't look as though any of their ...


1

Regarding the first aspect of your question regarding censorship thanks to the first amendment there are no laws forcing censorship on user generated content to the extent that the service you are providing should not be designed for the breach of the law. A prime example of this is the torrent site The Pirate Bay. While the argument could be made all ...


1

No, a noindex isn't necessary. The canonical link element should ensure that only the canonical version is returned in search results – so no duplicates – and will benefit from "ranking signals" of the canonicalised (i.e. variant) pages. https://support.google.com/webmasters/answer/139066?hl=en


0

The reason for this is that Google detected the term .NET in the word and .NET is a well established keyword for the .NET framework. As such it would have made the mistaken decision that the word was simply missing a space and added it as a spelling correction. Not much that can be done about fixing this though.


0

Unfortunately there is no way to do what you ask. There are a number of companies out there who's brands are based on mis-spelling common words and they aren't able to do it either. Google will have the page rank for dezign but every time a user tries to search for dezign the SERP will say "did you mean design? showing you results for "design" instead. and ...


0

This may be an old question but for the purposes of providing it an up to date answer, this is no longer required. Google no longer needs, and in fact encourages users not, to use escape fragments for the URL's as the Googlebot is now able to understand javascript generated page content without needing a URL escape fragment. This can be seen in the ...


0

rel="nofollow" is not meant to prevent indexing of the linked page as that is the way that Google locates new pages on the internet. All it does is tell Google not to pass link juice to that linked page in an attempt to mitigate link spamming. The only way to block those old pages from Google would be to add each one to your robots.txt file as a disallowed ...


1

There is no way through any Google tools to hurry or delay the crawl of a site. When you use tools such as Fetch and Render + submit it won't affect any queued crawling of the page in question, all it will do is add further details to the queue and submit the detected links to the queue as additional pages to crawl. Google can take anywhere from a few hours ...


0

It's possible those accesses are page redirects or supplementary document files such as images. Perhaps each of your pages has a large number of assets attached to it that must load in order for the page to be complete, such as external javascript files, CSS files, and image files. Like closetnoc said, you can try google search console to limit crawl rate, ...


1

The example IP addresses are Google alright! However, it should be no more than 1 request per 2 seconds though that may have changed in the past few years. For the record, Google has always behaved rather well and has always stayed within the 1 request per 2 seconds rule. Still, it is not uncommon for small flurries of requests throughout the day and maybe ...


1

I recommend your option of using the URL Parameter Tool in search console. For each of your tracking parameters, add it to the tool and set it to "Doesn't effect page content (ex. tracks usage)." When you tell Googlebot that these are passive parameters, Googlebot will usually crawl just one URL with a specific parameter value. If the parameter can be ...


0

There is no way to tell Googlebot to forget about something it has crawled. Your only recourse is to: Fix the problem with your HTML. Redirect any faulty URLs that were caused by the problem. Wait until Googlebot has recrawled all the pages with faulty HTML and all the bad links that those pages might have generated. NOARCHIVE prevents Google from ...


1

But, without Googlebot being able to crawl the live URLs on pages, the way they're appearing on pages, it won't be able to follow the links and give them any weight, authority and relevance to these pages. Is that right? In theory yes, but my personal interpretation of having backlinks to pages that are not accessible to search engines is that they do play ...



Top 50 recent answers are included