The introduction of billing on emoncms.org brings with it some interesting questions about monitoring system design (or IOT in a wider sense) which I think it would be good to discuss openly. We don’t want to push people to use and pay for emoncms.org where it’s not suitable, we only want people to use it if it is valuable. Emoncms can be run locally and has been installed for some time on the OpenEnergyMonitor EmonPi/EmonBase base stations which are based on the RaspberryPi platform.
There are I think advantages and disadvantages to both remote and local data storage and visualisation and it’s likely a matter of choosing the right tool for the application, or a combination of both. As a starting point, I thought I would list a number of points for and against local and then remote storage. I’m sure there are plenty more to add.
There are then a few ideas at the end for how some of the disadvantages of local storage could be solved, although not easily.
Local Storage (EmonPi/EmonBase/Pi)
- More user control and ownership of the whole system.
- Data mostly sourced and used locally, does it need to go out to a remote server?
- Distributed design, scalable.
- Non-reliant on WAN, especially important for control and home automation, although remote control is also useful.
- Combined heat and computing, heat not lost in a datacenter?
- Not as easy to access data remotely, requires setup of 3rd party services e.g, port forwarding with dynDNS. Dataplicity, where such a service has been setup, data access can be slow.
- Have to handle own backups and security updates.
- If there are problems you end up being a linux admin, higher technical barrier to entry?
Remote Storage on emoncms.org
- Easy remote access
- Public dashboard horsepower, ideal for community renewable energy projects
- Easier multi-account administration potential, useful for solar, heat pump, retrofit installers/organisations and research projects.
- Backup and security updates handled for you
- Support from server admin available
- Easier tech level, less need for technical expertise, no linux command line required.
- Easier? potential to do community energy aggregation, open data projects e.g a solar output map
- Reliant on service provider, less user control and ownership of whole system.
- Requires reliable WAN connection to upload data to server, can be mitigated with buffering.
- Hardware with plenty of processing and storage capability to do much of the same data storage and visualisation service has likely already been paid for in the basestation, remote server adds further costs.
- Large monitoring systems use a lot of network bandwidth uploading relatively small amounts of data at regular intervals.
Ideas that could solve some of the disadvantages of local storage
The first two ideas would require considerable development, and are more ideas for the longer term.
- Socket based remote access to local data. Login to your Pi remotely from emoncms.org, view feed list, graphs, control elements, integrated in a seamless way so that it feels like you’re accessing a remote site - this would be really neat.
- Automated backup system to a remote server (a periodic backup could upload a larger chunk of data say on a daily basis, the data could be compressed. Data could be saved on cheaper spinning disk storage without option to view graphs remotely) simplifying remote requirements and lowering costs. Easy backup/restore functionality. Upload/Download bandwidth could be a significant problem here.
- Host specific feeds on the remote server if shared on a public dashboard or if particularly useful for remote access, store a larger number of feeds locally that are only used occasionally for more in-depth analysis.