Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We currently monitor our webapps using cURL. More and more of our webapps use the GWT framework, which uses tons of JavaScript, and we can't rely on our cURL system to monitor anymore. Therefore, we search the right tool to monitor, but it seems difficult to find a crawler which is light (no Selenium please) but handles JavaScript correctly.

P.S. : we host our webapps as well as the probes, we don't want any Internet monitoring service.

share|improve this question

migrated from webapps.stackexchange.com Sep 22 '12 at 17:46

This question came from our site for power users of web applications.

Did you take a look at this thread stackoverflow.com/questions/2670082/… ? – StevenLeRoux Aug 27 '12 at 16:10
Studying watir.com with headless mode – Raf Aug 28 '12 at 14:46
New idea : defining a REST API on each webapp (/probe/step1, /probe/step2, ...) – Raf Oct 10 '12 at 15:28
I think the right answer is CasperJS/PhantomJS. It works really well on GWT sites, it's a high level descriptive language with no reference to GWT RPC. – Raf Nov 8 '12 at 9:18

I would highly recommend investing time into PhantomJS or CasperJS.

They are browser simulations that interact with HTTP and DOM level JS. We use it at Top Hat to test our app and it's a very heavy JS client.

PhantomJS isn't for the feint of heart as getting it to work exactly like you want it to can be challenging however, it's worth the time.

Good luck, let me know how it goes @kentf.

share|improve this answer

Create a PHP page that downloads whatever is passed to $_GET["URL"] and saves it to temp.html. Then create an <iframe> with the source temp.html. Then use JavaScript to get the iframe's innerHTML. Pass that innerHTML to another PHP page that saves it as domain.html and have your monitor check out domain.html to get a look at the JavaScript compiled version of the site. You can just add a line to open the browser that starts this whole process and then check domain.html 30 seconds later.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.