17

I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link: The site might previously have had those links The site may be cloaking and serving those links only to Googlebot and not to visitors It is a waste of resources, but it won't confuse Google and it won't hurt ...


9

Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/ Google crawled stuff like <script>var test="www.example.net/page.html";</script> or simply www.example.net/page.html So i think they only look for a ...


7

In Webmaster Tools, Google will tell you where they found the URL. In the error report, click on the URL In the resulting popup, click on the "Linked From" tab Google will then display the URL of the page that contains that link. Unfortunately, there is no way to prevent these junk URLs from appearing as 404. They come from a variety of sources: Scraper ...


6

Doubleclick is subsidiary of Google, which develops and provides Internet ad serving services. Using Adsense with dynamic allocation and targetting will automaticlly receive more relevant adds from the Doubleclick network. The reason that Google Search Console is nagging is because one or several resources are blocked, these can be internal resources or ...


5

If you have a message saying that "Google couldn't crawl your site because we were unable to access the robots.txt file". Then it is not the contents of the robots.txt file that is in question, it is that Google simply couldn't access the file. And when Google can't access a robots.txt file then it won't crawl the site. Using fetch as Googlebot in ...


5

Let's analyse your examples. All three have on thing in common: /enquiry/frmenquiry.aspx. That makes it worth googling for. One of the results is this question, but there are several others, a few of which actually display the HTML source code instead of the rendered HTML. That is a classic attack vector for compromising web sites. I would surmise that your ...


5

There are tons of scripts out there that optimistically scan random IP addresses on the internet to find vulnerabilities known in various kinds of software. 99.99% of the time, they find nothing (like on your site,) and that 0.01% of the time, the script will pwn the machine and do whatever the script controller wants. Typically, these scripts are run by ...


4

Once Google Webmaster Tools reports 404 errors on your website, you have to fix the issue on the site, then you need to mark these errors as fixed in Google Webmaster Tools. Then if this error appears again on your site, Google Webmaster Tools will report it again, otherwise it's fixed.


4

Blocking by referrer won't help: Googlebot doesn't send a referrer header when it crawls, so it will never see your block. Here is what Google's John Mueller (who works on Webmaster Tools and Sitemaps) has to say about 404 errors that appear in Webmaster tools: HELP! MY SITE HAS 939 CRAWL ERRORS!!1 I see this kind of question several times a week; ...


4

It merely means that when the crawler visited some of your pages, the response speed was slow. It may or may not be an intermittent issue but I would recommend that you do a performance audit of your pages. If you're not sure where/how to proceed, Google page insights is a good bet: https://developers.google.com/speed/pagespeed/insights/


4

Unfortunately there's not much you can do. You can't prevent these other (hacked) sites from linking to you. Besides, once Google has found these (now erroneous) URLs, it is likely to continue crawling them for sometime to come, even when the inbound link is (eventually) removed. You are already doing the correct thing, in returning a 404. A genuine 404 is ...


3

Unless you see reported "Crawl Errors" or "Blocked URLs" then it would seem that missing graph data is simply "where the data is not available". However, Google does not appear to be particularly clear as to why these "gaps" occur, simply stating that it can take time for data to appear for new sites and to check the stats for both the www and bare domains (...


3

Once Google crawls a URL, it will continue to try to crawl that URL forever and will never quit complaining about it if it goes away. On one of my sites, we changed the URL structure about 15 years ago. To this day, Googlebot still requests a bunch of the old style URLs, even the ones with no links to them. You can mitigate this situation by returning ...


3

Google is telling you that some pages found in your sitemap loaded very slowly. There are online testing tools to check the speed of your site. There are several, but the first good looking service is http://gtmetrix.com/. But while this is an option people like, read on. Google does tell you what pages loaded slowly using Google Analytics, Site Speed, ...


3

This probably isn't actually an attack but a scan or probe. Depending on the scanner/prober, it might be benign, meaning it is just looking for issues in some type of research capacity or it could have a function to automatically attack if it finds an opening. Web browsers put valid referrer information but other programs can just make up whatever referrer ...


3

Indeed it sounds like bot frenzy. We have been getting hammered as well by thousands of IP's across many hosts, most likely unbeknownst to the site OP. Before i offer some helpful solutions, one question back that i have is: Q: How are you seeing 404's from your site as a whole in Google webmaster tools? GWT is the output of Googlebots findings, not the ...


3

Your PHP code isn't modifying the robots.txt file. You've merely commented out the link to the page. Try creating a robots.txt file in the main public directory of your website. Then add the following: User-Agent: * Disallow: /terms-and-conditions/ You could also add a noindex meta tag to the file in question: <meta name="robots" content="noindex"> ...


3

It sounds like your registrar and/or host had some downtime. If your registrar is not your host, then I would be the registrar DNS failed to respond to requests. This is one of the reasons why I always recommend high quality registrars and hosts though this can happen for other reasons too such as network error along the route. It is possible that neither ...


3

Hi this is due to a 3rd party site linking to your domain to a page on your site that doesn't exist. So Google has followed a link from another website to yours - but because the link is incorrect at the other website when Google hits your site it gets a 404 error because the page that's being linked to doesn't exist. This isn't your fault as is usually due ...


3

Somewhere your site is linking to the "htmlprint" version. The first thing to do is figure out where that broken link is and fix it. View the source of your page and search for "htmlprint". Instead, it is possible that Googlebot is misinterpreting JavaScript code in your page. Googlebot tries to execute JavaScript to find URLs. You may have something ...


3

When Googlebot is crawling such a large number of bad URLs, it is almost always because your site is misconfigured and you are linking to the URLs incorrectly somewhere. In your case it is the "show all articles" link. For example on this page I see the following in the HTML source code: <a href="http://ermagazin.com/zakopao-zivu-djevojku-8-mjeseci-...


3

I wouldn't worry too much because 404 errors are not penalized. It basically represents files that have been temporarily hidden. If removed, it will clear itself from the search. I usually suggest not blocking Google for any reasons. As you will see Google recommends 404 pages and noindex tag opposed to blocks. Make removal permanent' The Remove URLs tool ...


3

If Google is reporting these pages as a 404 (and this has been confirmed with the "Fetch as Google" tool in Google Search Console), yet they appear to be perfectly accessible and return a 200 OK when visiting the site then check your .htaccess file for any suspicious code that might be redirecting the Googlebot. If a hacker is able to inject code into ....


3

You explicitely requested an indexing of some urls, through sitemap or webmaster tools, which are duplicates without canonical. The question is rather why do you want duplicated urls are explicitely indexed? It is not according to good SEO practices. Set your duplicated urls to noindex or, at least set a canonical so Google knows what to rank instead of ...


3

The screenshots indicate you are looking at the http and non www version of your site. The home page inspection report indicates that the home page for that site (http without www) is redirecting to the https with www version . https://www. and that it is not indexed because of that. That's from the "page with redirect" statement and what Google says it has ...


3

Googlebot never makes up URLs. It crawls URLs because it finds links to them. If Google is crawling a URL, it must be linked from somewhere on the web. In this case, one bad link into a page with an extra trailing slash probably caused your site to show additional broken links to Googlebot. By default, web servers and PHP allow any additional path to be ...


2

This is the case of URL masking, in simple words your site has been attacked with malicious codes, so your site is showing the url of other sites in your url. You have to do the fresh installation of your site. The best way is to take the backup of your database and then go ahead with the fresh installation. Note: Try not to use any code files from the ...


2

The 170 not found 404 errors are likely because Google can no longer find the previous PHP pages or content. If your new WordPress pages and content match your previous ones, then create 301 redirects in your web server configuration from the old URLs to the new ones to let Google know that they moved. If you just wish to add the new URLs and remove the ...


2

The damage has been done. You are in deep kimchi and will be for a while. The best thing you can do is wait. It will take quite a bit of time, but in the meantime, make sure that these links remain broken. It is okay to allow them to 404. A 410 is better, but not worth the effort to implement. If you have not registered for Google Webmaster Tools, you can ...


2

When you have a URL in the page without a nofollow, loaded directly, it is most likely that Google will follow it. In your case, when Google is following it, it is being served with a page by your script, so it is not entirely Google's fault. You have really two options here: Add a canonical link. You mentioned that you have already done that. Add a rel=...


Only top voted, non community-wiki answers of a minimum length are eligible