Emoncms local vs remote

Emoncms
1

The introduction of billing on emoncms.org brings with it some interesting questions about monitoring system design (or IOT in a wider sense) which I think it would be good to discuss openly. We don’t want to push people to use and pay for emoncms.org where it’s not suitable, we only want people to use it if it is valuable. Emoncms can be run locally and has been installed for some time on the OpenEnergyMonitor EmonPi/EmonBase base stations which are based on the RaspberryPi platform.

There are I think advantages and disadvantages to both remote and local data storage and visualisation and it’s likely a matter of choosing the right tool for the application, or a combination of both. As a starting point, I thought I would list a number of points for and against local and then remote storage. I’m sure there are plenty more to add.

There are then a few ideas at the end for how some of the disadvantages of local storage could be solved, although not easily.

Local Storage (EmonPi/EmonBase/Pi)

Advantages

  • More user control and ownership of the whole system.
  • Data mostly sourced and used locally, does it need to go out to a remote server?
  • Distributed design, scalable.
  • Non-reliant on WAN, especially important for control and home automation, although remote control is also useful.
  • Combined heat and computing, heat not lost in a datacenter?

Disadvantages

  • Not as easy to access data remotely, requires setup of 3rd party services e.g, port forwarding with dynDNS. Dataplicity, where such a service has been setup, data access can be slow.
  • Have to handle own backups and security updates.
  • If there are problems you end up being a linux admin, higher technical barrier to entry?

Remote Storage on emoncms.org

Advantages

  • Easy remote access
  • Public dashboard horsepower, ideal for community renewable energy projects
  • Easier multi-account administration potential, useful for solar, heat pump, retrofit installers/organisations and research projects.
  • Backup and security updates handled for you
  • Support from server admin available
  • Easier tech level, less need for technical expertise, no linux command line required.
  • Easier? potential to do community energy aggregation, open data projects e.g a solar output map

Disadvantages

  • Reliant on service provider, less user control and ownership of whole system.
  • Requires reliable WAN connection to upload data to server, can be mitigated with buffering.
  • Hardware with plenty of processing and storage capability to do much of the same data storage and visualisation service has likely already been paid for in the basestation, remote server adds further costs.
  • Large monitoring systems use a lot of network bandwidth uploading relatively small amounts of data at regular intervals.

Ideas that could solve some of the disadvantages of local storage

The first two ideas would require considerable development, and are more ideas for the longer term.

  • Socket based remote access to local data. Login to your Pi remotely from emoncms.org, view feed list, graphs, control elements, integrated in a seamless way so that it feels like you’re accessing a remote site - this would be really neat.
  • Automated backup system to a remote server (a periodic backup could upload a larger chunk of data say on a daily basis, the data could be compressed. Data could be saved on cheaper spinning disk storage without option to view graphs remotely) simplifying remote requirements and lowering costs. Easy backup/restore functionality. Upload/Download bandwidth could be a significant problem here.
  • Host specific feeds on the remote server if shared on a public dashboard or if particularly useful for remote access, store a larger number of feeds locally that are only used occasionally for more in-depth analysis.
2

I think Paul wrote a script to do this to DropBox IIRC.

I have plenty of other ‘boxes’ locally so being able to FTP the backups or RSYNC to my NAS would be good. I’m not that precious about the data as it is more a real time thing - I record meter readings monthly for a long term view on usage. Backing up of settings, visualisations, dashboards etc is more important to me.

I was thinking recently it would be good to be able to limit the data stored to a fixed period of time.

1 Like
3

For remote access to rPi we could use OpenVPN with split tunneling, i’m setting up one now on a spare raspberry to check if it is easy enough to do.

It still needs to port forward one port and map the dynamic IP address with a DNS name, but it is more secure than the other alternatives and it’s more flexible because when you connect you can reach your internal LAN and can access multiple resources (emoncms, openhab, etc)

There are clients to smartphones and laptops, the only thing i am not sure is if the Raspberry pi can handle it. What do you think?

4

Ok this is really easy to set up and it’s working.

  • install pivpn
  • comment some lines on the config file to disable pushing DNS and default route
  • permit port in ufw
  • setup port forwarding in router
  • copy .ovpn profile to device
  • connect!

If you want i can do a small tutorial.

5

Don’t forget to give your Pi’s local IP address a ‘reserved’ IP address in your router settings, otherwise if you reboot you may find that it’s been allocated a different local address and your port forwarding will not work.

Also, most internet providers (in the UK) assign public IP’s dynamically, so it can change, and regularly does. How are you intending to overcome this with your VPN?

1 Like
6

My ISP router gives infinite leases for local LAN devices, but you are totally right, there are two options, keep DHCP on the RPI and configure a fixed IP on the internet router or configure a static IP on the RPI that does not collide with the internet router’s DHCP pool.

Regarding the dynamic allocation of public IP addresses on residential internet you need a name that resolves to that ever changing IP. There are lot’s of free services that are supported on the Internet router, just check one that is supported, create account and get the name and then configure the credentials back on the internet router so it can automatically update every time the public ip changes.

no-ip is a popular choice.

PIVPN is really easy, it handles everything in one run. In our case we have to tweak it a bit to do split-tunnel but with some screenshots i think it’s doable for everybody.

EDIT: Actually the LAN would not be available, only the raspberry pi, but i think it’s not a problem.