Community
OpenEnergyMonitor

Community

Restore missing data

Hey,

Sometimes emoncms stops logging data. Either something goes haywire with the box itself and it stops recording, or the network connection goes down and emoncms.org stops receiving. So, I have all the data, but sometimes it’s split over the two instances (local or emoncms.org). I would love to be able to sync the two, even if that means downloading everything and writing a script to overwrite the NaNs.

Is there an easier way?

Bruce

What version? What hardware?

Can you click on the button Copy as Markdown next to Server Information on the Admin page and paste into a reply here please (no further formatting required).

I’m sure it’s just benign stuff caused by my own development. Restarting the wrong daemon without restarting others breaks the connection and if I don’t notice for a few hours I lose a bunch of data. In this case I’m only interested in correcting the existing gaps in my logs.

However, here is the result of your request:

Server Information

Server Information

Services

  • emonhub :- Active Running
  • emoncms_mqtt :- Active Running
  • feedwriter :- Active Running - sleep 60s 66 feed points pending write
  • service-runner :- Active Running
  • emonPiLCD :- Active Exited
  • redis-server :- Active Running
  • mosquitto :- Active Running

Emoncms

  • Version :- low-write 10.2.0
  • Modules :- Administration | App v2.0.9 | Backup v2.1.7 | EmonHub Config v2.0.4 | Dashboard v2.0.5 | Device v2.0.3 | energyaudit | EventProcesses | Feed | Graph v2.0.7 | Input | Postprocess v2.1.2 | CoreProcess | Schedule | Network Setup v1.0.0 | sync | Time | User | Visualisation | WiFi v2.0.2
  • Git :-

Server

  • OS :- Linux 4.19.97-v7+
  • Host :- emonpi | emonpi | (192.168.42.124)
  • Date :- 2020-02-10 16:25:38 UTC
  • Uptime :- 16:25:38 up 3:43, 3 users, load average: 1.19, 0.52, 0.28

Memory

  • RAM :- Used: 38.45%
    • Total :- 937.97 MB
    • Used :- 360.66 MB
    • Free :- 577.3 MB
  • Swap :- Used: 3.75%
    • Total :- 100 MB
    • Used :- 3.75 MB
    • Free :- 96.25 MB
      Write Load Period

Disk

  • / :- Used: 75.49%
    • Total :- 3.81 GB
    • Used :- 2.88 GB
    • Free :- 773.19 MB
    • Write Load :- 97.1 B/s (83 days 17 hours 24 mins)
  • /home/pi/data :- Used: 50.70%
    • Total :- 3.21 GB
    • Used :- 1.63 GB
    • Free :- 1.42 GB
    • Write Load :- 1.19 B/s (83 days 17 hours 24 mins)
  • /boot :- Used: 53.28%
    • Total :- 42.52 MB
    • Used :- 22.65 MB
    • Free :- 19.86 MB
    • Write Load :- 0 B/s (83 days 17 hours 24 mins)

HTTP

  • Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.3.22-MariaDB-0+deb10u1
  • Host :- localhost:6379 (127.0.0.1)
  • Date :- 2020-02-10 16:25:37 (UTC 00:00‌​)
  • Stats :- Uptime: 13395 Threads: 10 Questions: 10582 Slow queries: 0 Opens: 25 Flush tables: 1 Open tables: 19 Queries per second avg: 0.789

Redis

  • Version :-
    • Redis Server :- 5.0.3
    • PHP Redis :- 4.2.0
  • Host :- localhost:6379
  • Size :- 802 keys (876.95K)
  • Uptime :- 0 days

MQTT Server

  • Version :- Mosquitto 1.5.7
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.3.11-1~deb10u1 (Zend Version 3.3.11)
  • Modules :- apache2handler | calendar v7.3.11-1~deb10u1 | Core v7.3.11-1~deb10u1 | ctype v7.3.11-1~deb10u1 | date v7.3.11-1~deb10u1 | dom v20031129 | exif v7.3.11-1~deb10u1 | fileinfo v7.3.11-1~deb10u1 | filter v7.3.11-1~deb10u1 | ftp v7.3.11-1~deb10u1 | gettext v7.3.11-1~deb10u1 | hash v7.3.11-1~deb10u1 | iconv v7.3.11-1~deb10u1 | igbinary v3.0.0 | json v1.7.0 | libxml v7.3.11-1~deb10u1 | mysqli v7.3.11-1~deb10u1 | mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $ | openssl v7.3.11-1~deb10u1 | pcre v7.3.11-1~deb10u1 | PDO v7.3.11-1~deb10u1 | pdo_mysql v7.3.11-1~deb10u1 | Phar v7.3.11-1~deb10u1 | posix v7.3.11-1~deb10u1 | readline v7.3.11-1~deb10u1 | redis v4.2.0 | Reflection v7.3.11-1~deb10u1 | session v7.3.11-1~deb10u1 | shmop v7.3.11-1~deb10u1 | SimpleXML v7.3.11-1~deb10u1 | sockets v7.3.11-1~deb10u1 | sodium v7.3.11-1~deb10u1 | SPL v7.3.11-1~deb10u1 | standard v7.3.11-1~deb10u1 | sysvmsg v7.3.11-1~deb10u1 | sysvsem v7.3.11-1~deb10u1 | sysvshm v7.3.11-1~deb10u1 | tokenizer v7.3.11-1~deb10u1 | wddx v7.3.11-1~deb10u1 | xml v7.3.11-1~deb10u1 | xmlreader v7.3.11-1~deb10u1 | xmlwriter v7.3.11-1~deb10u1 | xsl v7.3.11-1~deb10u1 | Zend OPcache v7.3.11-1~deb10u1 | zlib v7.3.11-1~deb10u1

Pi

  • Model :- Raspberry Pi 2 Model B Rev 1.1 - 1GB (Sony UK)

  • Serial num. :- 9C7F0B26

  • Temperature :- 34.70°C - 35.8°C

  • emonpiRelease :- - File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0
  • Language :- en-GB,en;q=0.5

Window

  • Size :- 1904 x 1097

Screen

  • Resolution :- 1920 x 1200

Oh, OK. yes possibly. There is the sync module, but I’m not really clear on what it does exactly @TrystanLea shoudl be able to advise.

Thanks for the pointer. I tried the sync module but the feed-list API returns this:

<b>Notice</b>:  Trying to get property 'host' of non-object in <b>/home/pi/sync/sync-module/sync_controller.php</b> on line <b>51</b><br />
<br />
<b>Notice</b>:  Trying to get property 'apikey_read' of non-object in <b>/home/pi/sync/sync-module/sync_controller.php</b> on line <b>51</b><br />
<br />
<b>Warning</b>:  file_get_contents(/feed/listwithmeta.json?apikey=): failed to open stream: No such file or directory in <b>/home/pi/sync/sync-module/sync_controller.php</b> on line <b>51</b><br />
<br />
<b>Warning</b>:  Invalid argument supplied for foreach() in <b>/home/pi/sync/sync-module/sync_controller.php</b> on line <b>83</b><br />
{"solar_power":{"local":{"exists":true,"id":2,"tag":"","engine":"5","datatype":"1","start_time":1452338220,"interval":10,"npoints":12903083},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"vrms":{"local":{"exists":true,"id":3,"tag":"","engine":"5","datatype":"1","start_time":1452338220,"interval":10,"npoints":12903083},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"solar_energy":{"local":{"exists":true,"id":6,"tag":"","engine":"5","datatype":"1","start_time":1452338720,"interval":10,"npoints":12903033},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"temperature":{"local":{"exists":true,"id":9,"tag":"","engine":"5","datatype":"1","start_time":1452338760,"interval":10,"npoints":12903029},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"house_power":{"local":{"exists":true,"id":12,"tag":"","engine":"5","datatype":"1","start_time":1452431490,"interval":10,"npoints":12893756},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"house_energy":{"local":{"exists":true,"id":13,"tag":"","engine":"5","datatype":"1","start_time":1452431490,"interval":10,"npoints":12893756},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"grid_power":{"local":{"exists":true,"id":14,"tag":"","engine":"5","datatype":"1","start_time":1452431890,"interval":10,"npoints":12893716},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"grid_energy":{"local":{"exists":true,"id":15,"tag":"","engine":"5","datatype":"1","start_time":1452431910,"interval":10,"npoints":12893714},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"livingroom_raw_temperature":{"local":{"exists":true,"id":26,"tag":"","engine":"5","datatype":"1","start_time":1480886240,"interval":10,"npoints":10048280},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"boiler_raw_temperature":{"local":{"exists":true,"id":27,"tag":"","engine":"5","datatype":"1","start_time":1480886280,"interval":10,"npoints":6507101},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"livingroom_temperature":{"local":{"exists":true,"id":28,"tag":"","engine":"5","datatype":"1","start_time":1480886310,"interval":10,"npoints":2562650},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"boiler_temperature":{"local":{"exists":true,"id":29,"tag":"","engine":"5","datatype":"1","start_time":1480886320,"interval":10,"npoints":2562867},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"bedroom_temperature":{"local":{"exists":true,"id":33,"tag":"","engine":"5","datatype":"1","start_time":1481843100,"interval":60,"npoints":1658764},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"bedroom_temperature_external":{"local":{"exists":true,"id":34,"tag":"","engine":"5","datatype":"1","start_time":1481843100,"interval":60,"npoints":1658764},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"bedroom_humidity":{"local":{"exists":true,"id":35,"tag":"","engine":"5","datatype":"1","start_time":1481843160,"interval":60,"npoints":1658763},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"emonth_battery":{"local":{"exists":true,"id":36,"tag":"","engine":"5","datatype":"1","start_time":1481843160,"interval":60,"npoints":1658763},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"living_room_temperature_raw_2":{"local":{"exists":true,"id":37,"tag":"","engine":"5","datatype":"1","start_time":1504821080,"interval":10,"npoints":4163962},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"air_pressure":{"local":{"exists":true,"id":40,"tag":"","engine":"5","datatype":"1","start_time":1504821510,"interval":10,"npoints":7654753},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"emonth5:rssi":{"local":{"exists":true,"id":41,"tag":"","engine":"5","datatype":"1","start_time":1505829150,"interval":10,"npoints":7553979},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"dew_point":{"local":{"exists":true,"id":44,"tag":"","engine":"5","datatype":"1","start_time":1509614580,"interval":60,"npoints":1195906},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"kitchen_cput0":{"local":{"exists":true,"id":45,"tag":"Node kitchen","engine":"5","datatype":"1","start_time":1522530640,"interval":10,"npoints":2342665},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"lounge_cput0":{"local":{"exists":true,"id":46,"tag":"Node lounge","engine":"5","datatype":"1","start_time":1522530660,"interval":10,"npoints":5883838},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"emonpi_cput0":{"local":{"exists":true,"id":49,"tag":"Node emonpi","engine":"5","datatype":"1","start_time":1522531190,"interval":10,"npoints":5883785},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"fridge_temperature":{"local":{"exists":true,"id":50,"tag":"Node kitchen","engine":"5","datatype":"1","start_time":1563059850,"interval":30,"npoints":610306},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"fridge_humidity":{"local":{"exists":true,"id":51,"tag":"Node kitchen","engine":"5","datatype":"1","start_time":1563059940,"interval":30,"npoints":610303},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"freezer_temperature":{"local":{"exists":true,"id":52,"tag":"Node kitchen","engine":"5","datatype":"1","start_time":1563059940,"interval":30,"npoints":610303},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"freezer_humidity":{"local":{"exists":true,"id":53,"tag":"Node kitchen","engine":"5","datatype":"1","start_time":1563059970,"interval":30,"npoints":610302},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"fridge_dew_point":{"local":{"exists":true,"id":54,"tag":"Node kitchen","engine":"5","datatype":"1","start_time":1563280650,"interval":30,"npoints":602946},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}},"freezer_dew_point":{"local":{"exists":true,"id":55,"tag":"Node kitchen","engine":"5","datatype":"1","start_time":1563280680,"interval":30,"npoints":602945},"remote":{"exists":false,"start_time":"","interval":"","npoints":""}}}

and the page just retries forever because it can’t parse the JSON. Is this a known bug?

How did you install it?

I’m surprised at this path /home/pi/sync/sync-module/. @TrystanLea is the person who knows.

If you followed these instructions https://github.com/emoncms/sync/ it should work, but is not in line with the current filesystem layout. On an emonpi with the latest image, it will go into /opt/emoncms/modules.

I didn’t, it’s always been there.

That link doesn’t seem to have install instructions.

Sorry, I thought you had updated your image. I updated the link. Sorry I can’t help further it’ll need @TrystanLea.

Hello @bwduncan at present emoncms.org does not support uploading data using the sync module, it only supports downloading data. Have you considered using a remote access solution such as dataplicity for remote access to your local system? that way you don’t need to replicate everything in both places?
https://guide.openenergymonitor.org/setup/remote-access/

Hi Trystan,

That looks neat! I ended up setting up a wireguard VPN which I actually really like. I find it much more useful than I thought I would. Off-topic: KDE connect is so handy and can work anywhere with a VPN, samba browsing works seamlessly for the Mac user on my “LAN” when out and about…

The problem is that neither system has the full set of data. For example, if the Pi locks up because the SD card is getting old then it stops recording but it is still exporting the data to emoncms.org.

The format is pretty simple, I’ll just sync the feed from emoncms.org and write a bit of python to fill in the blanks. Thanks for your help!

Bruce

That’s an idea, if you use the sync module to download the feed data, you could then use the postprocess module mergefeeds process to combine both, it automatically fills in blanks in one feed if there is data available in the other. The output feed is an average of both feeds where data is present in both - but given that your feeds should be the same the output should be the same…

There’s a bit on the post process module here: https://guide.openenergymonitor.org/emoncms/postprocess/

2 Likes