Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 7 down vote favorite

I need to search the web for sites that are in our industry that use the same Adwords management company, to ensure that the said company is not violating our contract, as they have been accused of doing.

They use a tracking code in the template of every page which has a certain domain in the URL, and I'm wondering if it's possible "Google" the source code using some bot that crawls the code rather than the content?

For example, I bought an unlimited license for an image gallery, and I was asked to type the license number in a comment just before the script. I thought it was just so a human could look at the source and find out if someone paid, but it turned out that it was actually that they had a crawler looking for their source code and that comment. If it ran across the code on your site, it would look for the comment, and if it found one, it would check to see if it was an existing one. If not, it would first notify you of your noncompliance, and then notify the owner of the script.

Edit: I'm looking to index HTML and JavaScript only, not the server-side languages or Java.

share|improve this question
Before anyone posts it, Koders.com is NOT what I'm looking for. That does not index HTML or other markup languages, only open-source scripting languages. It would be awesome if I could find one where I could suggest Google keywords and have it crawl the source code of the top n Google results. – Dexter Jan 6 '11 at 16:26

migrated from stackoverflow.com Jan 8 '11 at 15:13

1 Answer

up vote 1 down vote

You can only crawl the "output" of that code. when the comment is in the output - you will see it.

If I use a very bad code or good code to generate some html tags, you will not see it, you will only see the generated page. otherwise- there will be no secrets in the world (code can contain passwords, and other staff that you don't want anyone to see - so web server should never let you or any other crawler to see it).

EDIT ** you can get the "source" of an HTML page with any "HTTPREQUEST" API - in any language (maybe by other names), and you can parse the results your self. you need to make 1 call to Google- to get the "first 10 pages" - parse it for the urls' and then generate a call to each one of them, getting the page as "source code" and do what ever you need to do

share|improve this answer
Secrets? I can go to View >> Page Source and see all of the HTML code I need to see. Surely someone's written a crawler that can do this? – Dexter Jan 6 '11 at 16:32
I think the OP is referring to the HTML and Javascript source that's served to a browser, not the server-side source. – Wooble Jan 6 '11 at 16:33
Sorry, I should have been more specific. Editing the question now. – Dexter Jan 6 '11 at 16:36
1  
@dexter - you can get the "source" of an HTML page with any "HTTPREQUEST" API - in any language (maybe by other names), and you can parse the results your self. you need to make 1 call to Google- to get the "first 10 pages" - parse it for the urls' and then generate a call to each one of them, getting the page as "source code" and do what ever you need to do. – Dani Jan 6 '11 at 20:11
@Dani - I guess I can do that with some simple PHP then or ASP. – Dexter Jan 6 '11 at 21:40
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.