5

Google is discovering some URLs in GWT which I'm surprised it could discover. The only place these are linked from are as follows:

<li class="multiselect__item" data-key="filter__brand--examplebrand2" data-name="examplebrand2" data-count="92" data-url="/example-category/examplebrand1--examplebrand2/" title="examplebrand2">    <input type="checkbox" id="checkboxfilter__brand--examplebrand2">    <label for="checkboxfilter__brand--examplebrand2"><span></span>examplebrand2<i class="multiselect__item-count"> (92)</i></label></li>

The code below being the interesting bit:

data-url="/example-category/examplebrand1--examplebrand2/"

GWT is reporting that these URLs are linked from a page on the site, and not externally. My question, should Google be able to crawl these URLs?

2
  • 1
    You seem to have answered this question already? No one knows for sure what Google "should be able to do". Google simply tries its best to discover content (by whatever means possible) and they are certainly getting better at this. You seem to have already shown that Google is discovering this content (at least examining the URL). Is Google requesting this URL? Is it even a valid "public" URL? However, what Google does with it in this case is another matter.
    – MrWhite
    Dec 29, 2014 at 11:03
  • Àre you doing something with this URL? Maybe linking it via JavaScript?
    – unor
    Dec 29, 2014 at 14:48

1 Answer 1

6

Googlebot uses heuristics to pick out anything from the page that looks like it might be a URL. It can crawl:

  • URLs in SELECT option values
  • URLs in data attributes
  • URLs in JavaScript variables

In my experience, just having a slash (/) in a string is enough for Googlbot to think it may be a URL. The fact that you name you attribute data-URL may also contribute to Googlebot recognizing it.

As far as I know, Googlebot only uses these "links" for content discovery. I don't believe that it passes link juice across them the same way that it does for normal a href links.

Unfortunately, if Googlebot gets it "wrong" and crawls something that isn't a URL, it will still report it in your site errors in Google Webmaster Tools. In my opinion, it shouldn't do that when it used a heuristic to try to find something that looks like a URL.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.