The Stack Overflow podcast is back! Listen to an interview with our new CEO.
16

I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link: The site might previously have had those links The site may be cloaking and serving those links only to Googlebot and not to visitors It is a waste of resources, but it won't confuse Google and it won't hurt ...


9

Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/ Google crawled stuff like <script>var test="www.example.net/page.html";</script> or simply www.example.net/page.html So i think they only look for a ...


7

In Webmaster Tools, Google will tell you where they found the URL. In the error report, click on the URL In the resulting popup, click on the "Linked From" tab Google will then display the URL of the page that contains that link. Unfortunately, there is no way to prevent these junk URLs from appearing as 404. They come from a variety of sources: Scraper ...


6

This was covered in a Webmaster Central blog post a few months ago. One thing we’re really excited about in this new version of the Crawl errors feature is that you can really focus on fixing what’s most important first. We’ve ranked the errors so that those at the top of the priority list will be ones where there’s something you can do, whether ...


5

Let's analyse your examples. All three have on thing in common: /enquiry/frmenquiry.aspx. That makes it worth googling for. One of the results is this question, but there are several others, a few of which actually display the HTML source code instead of the rendered HTML. That is a classic attack vector for compromising web sites. I would surmise that your ...


5

If you have a message saying that "Google couldn't crawl your site because we were unable to access the robots.txt file". Then it is not the contents of the robots.txt file that is in question, it is that Google simply couldn't access the file. And when Google can't access a robots.txt file then it won't crawl the site. Using fetch as Googlebot in ...


5

Once Google Webmaster Tools reports 404 errors on your website, you have to fix the issue on the site, then you need to mark these errors as fixed in Google Webmaster Tools. Then if this error appears again on your site, Google Webmaster Tools will report it again, otherwise it's fixed.


5

There are tons of scripts out there that optimistically scan random IP addresses on the internet to find vulnerabilities known in various kinds of software. 99.99% of the time, they find nothing (like on your site,) and that 0.01% of the time, the script will pwn the machine and do whatever the script controller wants. Typically, these scripts are run by ...


4

Blocking by referrer won't help: Googlebot doesn't send a referrer header when it crawls, so it will never see your block. Here is what Google's John Mueller (who works on Webmaster Tools and Sitemaps) has to say about 404 errors that appear in Webmaster tools: HELP! MY SITE HAS 939 CRAWL ERRORS!!1 I see this kind of question several times a week; ...


4

It merely means that when the crawler visited some of your pages, the response speed was slow. It may or may not be an intermittent issue but I would recommend that you do a performance audit of your pages. If you're not sure where/how to proceed, Google page insights is a good bet: https://developers.google.com/speed/pagespeed/insights/


4

Unfortunately there's not much you can do. You can't prevent these other (hacked) sites from linking to you. Besides, once Google has found these (now erroneous) URLs, it is likely to continue crawling them for sometime to come, even when the inbound link is (eventually) removed. You are already doing the correct thing, in returning a 404. A genuine 404 is ...


4

Doubleclick is subsidiary of Google, which develops and provides Internet ad serving services. Using Adsense with dynamic allocation and targetting will automaticlly receive more relevant adds from the Doubleclick network. The reason that Google Search Console is nagging is because one or several resources are blocked, these can be internal resources or ...


3

Found the answer. Once I posted a followup comment above I managed to craft a better Google search term and found that Disqus is causing the issue. There is no solution that I can see of yet. There is an explanation of the problem here: http://www.seroundtable.com/disqus-google-errors-15663.html


3

Unless you see reported "Crawl Errors" or "Blocked URLs" then it would seem that missing graph data is simply "where the data is not available". However, Google does not appear to be particularly clear as to why these "gaps" occur, simply stating that it can take time for data to appear for new sites and to check the stats for both the www and bare domains (...


3

Once Google crawls a URL, it will continue to try to crawl that URL forever and will never quit complaining about it if it goes away. On one of my sites, we changed the URL structure about 15 years ago. To this day, Googlebot still requests a bunch of the old style URLs, even the ones with no links to them. You can mitigate this situation by returning ...


3

Xenu's Link Sleuth is what you're probably looking for


3

Looking at the source of your page, I see the following lines of JavaScript code: //call multiBox var initMultiBox = new multiBox({ ... addDownload: false,//do you want the files to be downloadable? pathToDownloadScript: './Scripts/ForceDownload.asp',//if above is true, specify path to download script (classicASP and ASP.NET versions included) ...


3

A sitemap may help with indexing your new URL structure, but remember Google makes no promises they'll crawl or index everything in there at all. However, even if they did index everything in your sitemap, that would not magically make your crawl errors go away. 404(not found) errors explicitly allow for the possibility that the URL may be available again ...


3

Google is telling you that some pages found in your sitemap loaded very slowly. There are online testing tools to check the speed of your site. There are several, but the first good looking service is http://gtmetrix.com/. But while this is an option people like, read on. Google does tell you what pages loaded slowly using Google Analytics, Site Speed, ...


3

This probably isn't actually an attack but a scan or probe. Depending on the scanner/prober, it might be benign, meaning it is just looking for issues in some type of research capacity or it could have a function to automatically attack if it finds an opening. Web browsers put valid referrer information but other programs can just make up whatever referrer ...


3

Indeed it sounds like bot frenzy. We have been getting hammered as well by thousands of IP's across many hosts, most likely unbeknownst to the site OP. Before i offer some helpful solutions, one question back that i have is: Q: How are you seeing 404's from your site as a whole in Google webmaster tools? GWT is the output of Googlebots findings, not the ...


3

Your PHP code isn't modifying the robots.txt file. You've merely commented out the link to the page. Try creating a robots.txt file in the main public directory of your website. Then add the following: User-Agent: * Disallow: /terms-and-conditions/ You could also add a noindex meta tag to the file in question: <meta name="robots" content="noindex"> ...


3

It sounds like your registrar and/or host had some downtime. If your registrar is not your host, then I would be the registrar DNS failed to respond to requests. This is one of the reasons why I always recommend high quality registrars and hosts though this can happen for other reasons too such as network error along the route. It is possible that neither ...


3

Hi this is due to a 3rd party site linking to your domain to a page on your site that doesn't exist. So Google has followed a link from another website to yours - but because the link is incorrect at the other website when Google hits your site it gets a 404 error because the page that's being linked to doesn't exist. This isn't your fault as is usually due ...


3

Somewhere your site is linking to the "htmlprint" version. The first thing to do is figure out where that broken link is and fix it. View the source of your page and search for "htmlprint". Instead, it is possible that Googlebot is misinterpreting JavaScript code in your page. Googlebot tries to execute JavaScript to find URLs. You may have something ...


3

When Googlebot is crawling such a large number of bad URLs, it is almost always because your site is misconfigured and you are linking to the URLs incorrectly somewhere. In your case it is the "show all articles" link. For example on this page I see the following in the HTML source code: <a href="http://ermagazin.com/zakopao-zivu-djevojku-8-mjeseci-...


3

I wouldn't worry too much because 404 errors are not penalized. It basically represents files that have been temporarily hidden. If removed, it will clear itself from the search. I usually suggest not blocking Google for any reasons. As you will see Google recommends 404 pages and noindex tag opposed to blocks. Make removal permanent' The Remove URLs tool ...


3

If Google is reporting these pages as a 404 (and this has been confirmed with the "Fetch as Google" tool in Google Search Console), yet they appear to be perfectly accessible and return a 200 OK when visiting the site then check your .htaccess file for any suspicious code that might be redirecting the Googlebot. If a hacker is able to inject code into ....


3

You explicitely requested an indexing of some urls, through sitemap or webmaster tools, which are duplicates without canonical. The question is rather why do you want duplicated urls are explicitely indexed? It is not according to good SEO practices. Set your duplicated urls to noindex or, at least set a canonical so Google knows what to rank instead of ...


2

Leading up to the event search traffic may have been higher than it is now and not have to do with your server having problems. Just like traffic goes up during the holidays. I guess it would depend on the event if it was locally or nationally promoted. 30 minutes of downtime shouldn't affect your traffic.


Only top voted, non community-wiki answers of a minimum length are eligible