I have been checking the errors of my website in the google console , and I fount it face an error Googlebot blocked by robots.txt and it shows a link https://static.doubleclick.net/instream/ad_status.js , that I did not used in my project. what is it ? my robots is this

User-agent: *
Disallow: /

How could I fix this error?

  • 1
    That robots.txt disallows everything on your site, is that what you want? – Stephen Ostermiller Jun 19 at 14:28
  • Doubleclick is an ad company owned by Google. Do you you AdSense or other ads on your website? – Stephen Ostermiller Jun 19 at 14:29
  • 1
    You may not have used that script in your project, but AdSense scripts are sometimes pre-installed with and supported by various modules, plugins, third party services, etc, because this is a popular way to monetize your site. This is a script for instream ads (could be video ads related). You can manually remove it from your backend files if you want, but make sure you know what it'll affect. Google wants to crawl your JS files, but your robots file blocks this, so it's letting you know there are JS files it can't access. If you want your site indexed, remove the forward slash from line 2. – Henry Visotski Jun 19 at 15:07
  • what thing will be disallowed ? No I do not have any advertisement on my pages . @StephenOstermiller – neda Derakhshesh Jun 19 at 15:29
  • @HenryVisotski Thank you very much for valuable informations. You mean it should be like this :? User-agent: * Disallow: ( Disallow with just : ) ? – neda Derakhshesh Jun 19 at 15:31

I suspect you are showing us the robots.txt file for the link that is blocked. robots.txt files vary per site. In this case https://static.doubleclick.net/robots.txt.

doubleclick are themselves saying, don't crawl our site. That is fine as Googlebot is not interested in ads. Quite a few third party tracking systems will do the same. It's nothing to worry about.

The robots.txt file for your own site will look different, and is probably fine.

| improve this answer | |
  • Ok Tony so you mean there is no need to change anything? – neda Derakhshesh Jun 21 at 10:01
  • Yep. Focus on something else. – Tony McCreath Jun 22 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.