1

I have a personal weather station connected to my LAN that provides its info (e.g. temperature, rainfall, humidity) via a web UI. I would like to scrape this info so I can make a better webpage of my own, and keep a record of the daily weather in a spreadsheet.

I know there are scrapers like parsehub and import.io, but they seem to work off the cloud which means they cannot access the websites residing locally on my home network.

What are some of the best ways to scrape content off a local website on a regular basis?

2

if your site is available with browser, than it is scrapeable on regular basis, because it has a regular internet address, like an IP-address.

Your task seems to me to be pretty simple:

  • open the web UI of the weather station and
  • recognize XPath of a site element(s), which you want to scrape. Use Chrome dev.tools->right click->copyXPath
  • scrape them into Google Spreadsheats and =importXML()

how to scrape with =importxml & Gspreadsheats

3
  • As I mentioned, the website in question is on a LAN i.e. it is on a 192.168.x.x address and not accessible publicly on the internet. Thus, cloud-based services like Google Docs, and the two web-scrapers already mentioned in the question, cannot access the website and its elements.
    – Islay
    Oct 12 '15 at 22:17
  • @Abhi, surely it can. Run your application from htdocs folder in your MAMP/XAMPP, so you get the web ui of it with localhost/your-app
    – Evgeniy
    Oct 13 '15 at 6:59
  • 1
    Yes, I can access the web UI using my browser (it sits on another embedded system connected to the LAN). But entering a localhost/my-app address in the =importXML() function does not work because Google Spreadsheets run the function on their servers which cannot access my localhost/intranet. Same applies to the other scrapers. [I emailed both - ParseHub and import.io - and they've confirmed that local scraping is not possible with their tools.]
    – Islay
    Oct 14 '15 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.