4

At the moment Google (I can not find similar informations for Bing, Yahoo!, etc.) does not index Data-URI.

But until it is possible I like to know if it is possible to exclude Data-URIs of being crawled. For example:

<img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" alt="trans-gif" />

or

<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA
AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO
9TXL0Y4OHwAAAABJRU5ErkJggg==" alt="Red dot" />

Do I need to add /data: or data: to my robots.txt or isn't it supported at all?

Improve this question
8

It's not supported at all since it is a not a HTTP call. It is embedded in the document and is essentially part of the HTML.

Improve this answer
  • We are seeing requests from both BingBot and YahooSlurp for urls like: /folder/path/data:image/gif;base64,... which is causing issues as we're then getting "Potentially dangerous request" errors in IIS. – Zhaph - Ben Duguid Feb 25 '16 at 9:49
  • @Zhaph-BenDuguid Presumably the data URI is complete and does actually match a valid data URI that exists in the HTML of your site? Are you seeing sustained requests for these URIs over a period of time or is this a "blip"? – MrWhite Feb 25 '16 at 16:55
  • @w3dk The Data URI scheme is supposed to be a fully contained base64 encoded representation of the binary object - meaning that the browser shouldn't be making another request - everything it needs is in the page already - see the question for an example of format - that's not a link to another part of the site. Yes, we've seen these for quite some time in our log files now. – Zhaph - Ben Duguid Feb 26 '16 at 10:24
  • @Zhaph-BenDuguid I was just really clarifying whether the data URI you are seeing in the request is indeed the same as that in your HTML - I expect it is - or whether it's entirely different (which would be "even" stranger). How does your server respond to such a request? Status code? – MrWhite Feb 26 '16 at 10:43
  • @w3dk - Sorry for the delay - yes the request is identical to that in the HTML. My server (IIS 7.5 running ASP.NET 4) responds with 500 Internal Server Error because it thinks this is "A potentially dangerous Request.Form value" (see white paper on this here). However digging deeper I think this isn't Bing/Yahoo but overzealous browser pre-fetches (mostly Chrome on ChromeOS, Safari on iPad and some Firefox instances - which is more painful). – Zhaph - Ben Duguid Feb 29 '16 at 18:24
4
+100

We are seeing requests from both BingBot and YahooSlurp for urls like: /folder/path/data:image/gif;base64,...

If you are actually getting an HTTP request for these resources then you should be able to block them in robots.txt, providing robots.txt is being honoured for these "strange" requests (Bing and Yahoo! should), with something like:

User-agent: *
Disallow /folder/path/data

Or, more generically using a * wildcard (an extension to the original robots.txt "standard" but is reportedly supported by Bing and Yahoo!):

User-agent: *
Disallow /*data:image/

Although, as John suggests in his answer, there really shouldn't be an HTTP request. Nothing is required from the server in order to get the resource (they already have it). I can't actually imagine a server ever responding with a valid response (at least not intentionally)?

Is this perhaps just a bot "glitch"?

Improve this answer
  • Yes, I'm thinking it's an error with these bots - as they should be recognising that the data: scheme doesn't represent a relative link. Interesting to see that both BingBot and YahooSlurp might be able to be blocked via that extension - I'll give that a go and see how we get on. – Zhaph - Ben Duguid Feb 26 '16 at 10:26
  • Apologies to BingBot/Yahoo - I think I'd mixed up two log lines - looks like it's actually some browsers attempting to pre-fetch content or similar ((mostly Chrome on ChromeOS, Safari on iPad and some Firefox instances) which is more painful :( – Zhaph - Ben Duguid Feb 29 '16 at 18:25
  • 1
    @Zhaph-BenDuguid That's even odder; robots.txt is not going to help if these are real browsers (or bad bots in disguise?). Also strange that several different browsers appear to exhibit the same behaviour. If these are real browsers, is it possible they are old versions that don't support data URIs? Although, according to caniuse.com they've been supported by Chrome and Firefox since forever (no specific mention of ChromeOS though and Firefox Android might be lagging - at least the data is missing). – MrWhite Feb 29 '16 at 20:05
  • 1
    @Zhaph-BenDuguid With regards to iOS Safari, caniuse.com reports support from version 3.2, no data before that. The other thought was whether there could be a rogue browser plugin - but I didn't think iOS Safari even supports plugins? – MrWhite Feb 29 '16 at 20:08
  • 2
    Do you have any javascript running on your page that is trying to pre-load or cache images, or provide any kind of preview feature? Sometimes these scripts if poorly written will parse your HTML looking for <img/> tags and just assume that the src= attribute represents an external rather than embedded resource. – richhallstoke Mar 1 '16 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.