[Solved] Daily kwh feed partially resetting after update

Hope someone can advise the best way of resolving this (and perhaps preventing it happening again). I’ll try and explain my steps prior to the issue, but it’s possible I’ve missed something; I didn’t notice this glitch in my use_kwh feed until today.

My kit is an rPi 2 Model B Rev 1.1 (yea, ancient, but it works), with a TPLink WN823N Rev 2 (rtl8192eu) WiFi dongle to get around the fact that the Pi2 doesn’t have onboard WiFi. I had some (unrelated) issues with my router recently, and in my fumblings, probably ran a ‘sudo apt update’ and ‘sudo apt upgrade’ on my homebrew EmonPi without thinking. Then I remembered there’s a tool in the EmonCMS Admin menu to do this, so I ran that as well. (18th Nov). I kept an update.log from that time, and a couple of times since (just to check that it really was as up to date as it can be).

My EmonSD in use is emonSD-17Oct19 and I think the previous last time I ran an update (via the EmonCMS Admin tool) was 25th Jan 2025.

As a result of my ongoing WiFi issues (now resolved, I think), I had cause to cleanly shutdown the rPi three times (or possibly sudo reboot - not sure how I can confirm which, to be honest).

It appears that the continuous process for turning my ‘use’ (pulse monitoring of smart-meter) feed to ‘use-kwh’ (the accumulating kwh feed used for daily/weekly/monthly totals, etc), has reset itself - but not to zero, as you might expect; instead to some other number which I think was the accumulated total from a previous restart or update. Again, not sure quite how to tell yet. Looking at the graph attached, it feels like it was the value of the accumulated kwh at the point of the last update attempt on 25th Nov (there’s a corresponding flatline there - not marked, but visible - which covers the time the unit was probably offline during my investigations with WiFi)

My main question now is - how do I correct this most safely? I’m tempted to simply delete all data on the use_kwh feed since the time of the update on 25th Nov, then somehow stifle new data coming in automatically (as part of the live feed process from ‘use’ feed - though I’m not sure how I’d suspend one without the other, safely). Then run a post-process from that same datetime, to do the ‘power-to-kwh’ accumulation and effectively ‘backfill’, then reinstate the original input-feed process (somehow). And hope it was all fairly seamless.

Is there a better way?

And is there a way to tell (i.e. not the hard way) whether any future hard-reboots are going to cause me to revert to this ancient use_kwh total (54505.something kwh)?

I am planning to move to emonSD-10Nov22 which I believe is the latest plausible candidate for my rPi2, but I’m in the middle of removing various other apps I’ve got on the same machine (Node-Red, and a mysql server I used for Octopus Agile data years ago), plus I need to copy over my homebrew pulse-reading software too - so it’s all a bit ‘tender’, and I need the EmonCMS/SD to be as near ‘normal’ as possible before I make that switchover.

Hence me wanting to do this step right. Any takers? I’d be really grateful for some insights or guiding steps, if anyone has anything to share. TIA

This is the log of the update from 25th Nov, in case it sheds any light. I have the ones from 19th and 18th Nov as well, if needed (but will wait until asked for those rather than eat up resources here!)

Starting update via service-runner-update.sh (v3.0) >
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        20G  4.0G   15G  22% /
devtmpfs        454M     0  454M   0% /dev
tmpfs           487M     0  487M   0% /dev/shm
tmpfs           487M   50M  438M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           487M     0  487M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs           1.0M  4.0K 1020K   1% /var/lib/php/sessions
tmpfs            30M     0   30M   0% /tmp
/dev/mmcblk0p3  9.9G  285M  9.1G   3% /var/opt/emoncms
/dev/mmcblk0p1  253M   49M  204M  20% /boot
log2ram          50M  3.1M   47M   7% /var/log
tmpfs            98M     0   98M   0% /run/user/1000
- emonSD version: emonSD-02Oct19
- supported images: emonSD-03Jul25 emonSD-01Feb24 emonSD-20Nov23 emonSD-10Nov22 emonSD-21Jul21 emonSD-08May21 emonSD-24Jul20 emonSD-02Oct19 emonSD-17Oct19
- emonSD base image check passed...continue update
git pull /opt/openenergymonitor/EmonScripts
  master
* stable
On branch stable
Your branch is up to date with 'origin/stable'.

nothing to commit, working tree clean
Already up to date.
-------------------------------------------------------------
Main Update Script
-------------------------------------------------------------
Date: Tue 25 Nov 12:00:57 GMT 2025
EUID: 1000
openenergymonitor_dir: /opt/openenergymonitor
type: all
serial_port: ttyAMA0
firmware: none
Hardware detected: rfm2pi
Reading package lists...
Building dependency tree...
Reading state information...
python3-pip is already the newest version (18.1-5+rpt1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: redis in /home/pi/.local/lib/python3.7/site-packages (3.5.3)
Running apt-get update
Get:1 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Get:2 http://archive.raspberrypi.org/debian buster InRelease [54.2 kB]
Get:3 https://deb.nodesource.com/node_12.x buster InRelease [4,584 B]
Get:4 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages [13.0 MB]
Get:5 http://archive.raspberrypi.org/debian buster/main armhf Packages [400 kB]
Fetched 13.5 MB in 18s (751 kB/s)
Reading package lists...
git pull /opt/openenergymonitor/emonpi
  master
* stable
On branch stable
Your branch is up to date with 'origin/stable'.

nothing to commit, working tree clean
Fetching origin
Already up to date.
git pull /opt/openenergymonitor/RFM2Pi
  master
* stable
On branch stable
Your branch is up to date with 'origin/stable'.

nothing to commit, working tree clean
Fetching origin
Already up to date.
Start emonhub update script:
-------------------------------------------------------------
emonHub update
-------------------------------------------------------------
Updating /opt/openenergymonitor/emonhub
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- running module install/update script (user=pi)
EmonHub directory: /opt/openenergymonitor/emonhub
emonSD_pi_env provided in arg = 1
user provided as arg = pi
installing or updating emonhub dependencies
Reading package lists...
Building dependency tree...
Reading state information...
bluetooth is already the newest version (5.50-1.2~deb10u5).
python3-configobj is already the newest version (5.0.6-3).
python3-pymodbus is already the newest version (2.1.0+dfsg-1).
python3-serial is already the newest version (3.4-4).
python3-pip is already the newest version (18.1-5+rpt1).
python3-spidev is already the newest version (20200602~200721-1~buster).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already up-to-date: paho-mqtt==1.6.1 in /home/pi/.local/lib/python2.7/site-packages (1.6.1)
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting requests
  Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl
Collecting py-sds011
  Could not find a version that satisfies the requirement py-sds011 (from versions: )
No matching distribution found for py-sds011
python3-rpi-lgpio not available, using python3-rpi.gpio instead
Remove console from /boot/cmdline.txt
Stop and disable serial service
Unit /etc/systemd/system/[email protected] is masked, ignoring.
/etc/emonhub directory already exists
Setting ownership of /var/log/emonhub to pi
Setting ownership of /var/log/emonhub/emonhub.log to pi and permissions to 644
Installing /usr/local/bin/emonhub symlink
emonhub.service already installed
- Service ActiveState=active
- component updated

Start emoncms update:
-------------------------------------------------------------
Update Emoncms Core
-------------------------------------------------------------
Updating /var/www/emoncms
- current branch: stable
- local changes: no
From https://github.com/emoncms/emoncms
   95e83491..4726e3ae  master     -> origin/master
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
-------------------------------------------------------------
Update Emoncms Services
-------------------------------------------------------------
emoncms_mqtt.service already installed
feedwriter.service already installed
service-runner.service already installed

Reloading systemctl deamon
Restarting Services...
- sudo systemctl restart feedwriter.service
--- ActiveState=active ---
- sudo systemctl restart emoncms_mqtt.service
--- ActiveState=active ---
- sudo systemctl restart emonhub.service
--- ActiveState=active ---
/opt/openenergymonitor/EmonScripts/sudoers.d/emoncms-rebootbutton: parsed OK
emonPi emoncms admin reboot button sudoers updated

-------------------------------------------------------------
Update Emoncms Modules
-------------------------------------------------------------
Updating /var/www/emoncms/Modules/app
- current branch: stable
- local changes: no
From https://github.com/emoncms/app
   8c586e0..b0ace4d  master     -> origin/master
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
Updating /var/www/emoncms/Modules/config
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
Updating /var/www/emoncms/Modules/dashboard
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
Updating /var/www/emoncms/Modules/device
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
Updating /var/www/emoncms/Modules/graph
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
Updating /var/www/emoncms/Modules/setup
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
Updating /var/www/emoncms/Modules/wifi
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
Updating /opt/emoncms/modules/backup
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- running module install/update script (user=pi)
- config.cfg already exists, left unmodified
- creating /etc/php/7.3/mods-available/emoncmsbackup.ini
post_max_size = 3G
upload_max_filesize = 3G
upload_tmp_dir = /var/opt/emoncms/backup/uploads
- phpenmod emoncmsbackup
- restarting apache
- component updated
Updating /opt/emoncms/modules/demandshaper
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- running module install/update script (user=pi)
- reinstalling demandshaper.service
- demandshaper service: ActiveState=active
- component updated
Updating /opt/emoncms/modules/postprocess
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
Updating /opt/emoncms/modules/sync
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- running module install/update script (user=pi)
[]
- reinstalling emoncms_sync.service
Removed /etc/systemd/system/multi-user.target.wants/emoncms_sync.service.
Created symlink /etc/systemd/system/multi-user.target.wants/emoncms_sync.service → /lib/systemd/system/emoncms_sync.service.
- Service ActiveState=active
- component updated
Updating /opt/emoncms/modules/usefulscripts
- current branch: stable
- local changes: no
- git fetch: Fetching origin
Already on 'stable'
- git checkout: Your branch is up to date with 'origin/stable'.
- git pull: Already up to date.
- database update: no changes
- component updated
/opt/openenergymonitor/EmonScripts/sudoers.d/wifi-sudoers: parsed OK
wifi sudoers entry updated

-------------------------------------------------------------
EmonPi LCD Update
-------------------------------------------------------------
Reading package lists...
Building dependency tree...
Reading state information...
i2c-tools is already the newest version (4.1-1).
python3-smbus is already the newest version (4.1-1).
python3-gpiozero is already the newest version (1.6.2-1).
python3-rpi.gpio is already the newest version (0.7.0-0.1~bpo10+4).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: redis in /home/pi/.local/lib/python3.7/site-packages (3.5.3)
Requirement already satisfied: paho-mqtt in /usr/local/lib/python3.7/dist-packages (1.5.0)
Requirement already satisfied: xmltodict in /home/pi/.local/lib/python3.7/site-packages (0.12.0)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.21.0)
Requirement already satisfied: adafruit-circuitpython-ssd1306 in /home/pi/.local/lib/python3.7/site-packages (2.12.15)
Requirement already satisfied: adafruit-circuitpython-framebuf in /home/pi/.local/lib/python3.7/site-packages (from adafruit-circuitpython-ssd1306) (1.6.4)
Requirement already satisfied: adafruit-circuitpython-busdevice in /home/pi/.local/lib/python3.7/site-packages (from adafruit-circuitpython-ssd1306) (5.2.6)
Requirement already satisfied: Adafruit-Blinka in /home/pi/.local/lib/python3.7/site-packages (from adafruit-circuitpython-ssd1306) (8.25.0)
Requirement already satisfied: adafruit-circuitpython-typing in /home/pi/.local/lib/python3.7/site-packages (from adafruit-circuitpython-busdevice->adafruit-circuitpython-ssd1306) (1.9.5)
Requirement already satisfied: Adafruit-PureIO>=1.1.7 in /home/pi/.local/lib/python3.7/site-packages (from Adafruit-Blinka->adafruit-circuitpython-ssd1306) (1.1.11)
Requirement already satisfied: Adafruit-PlatformDetect>=3.53.0 in /home/pi/.local/lib/python3.7/site-packages (from Adafruit-Blinka->adafruit-circuitpython-ssd1306) (3.54.0)
Requirement already satisfied: pyftdi>=0.40.0 in /home/pi/.local/lib/python3.7/site-packages (from Adafruit-Blinka->adafruit-circuitpython-ssd1306) (0.54.0)
Requirement already satisfied: typing-extensions~=4.0 in /home/pi/.local/lib/python3.7/site-packages (from adafruit-circuitpython-typing->adafruit-circuitpython-busdevice->adafruit-circuitpython-ssd1306) (4.7.1)
Requirement already satisfied: adafruit-circuitpython-requests in /home/pi/.local/lib/python3.7/site-packages (from adafruit-circuitpython-typing->adafruit-circuitpython-busdevice->adafruit-circuitpython-ssd1306) (2.0.2)
Requirement already satisfied: pyusb!=1.2.0,>=1.0.0 in /home/pi/.local/lib/python3.7/site-packages (from pyftdi>=0.40.0->Adafruit-Blinka->adafruit-circuitpython-ssd1306) (1.2.1)
Requirement already satisfied: pyserial>=3.0 in /usr/lib/python3/dist-packages (from pyftdi>=0.40.0->Adafruit-Blinka->adafruit-circuitpython-ssd1306) (3.4)
- reinstalling emonPiLCD.service
Unit /etc/systemd/system/emonPiLCD.service is masked, ignoring.
- installing emonPiLCD.service
Failed to enable unit: Unit file /etc/systemd/system/emonPiLCD.service is masked.
Failed to restart emonPiLCD.service: Unit emonPiLCD.service is masked.
- Service ActiveState=inactive
Stopping emonPiLCD service
service-runner will be restarted at the end

-------------------------------------------------------------
System update done: Tue 25 Nov 12:04:10 GMT 2025
-------------------------------------------------------------

Ok, so bravery won and I’ve solved this myself. Putting this here just in case it’s useful to anyone in the future.

Just run the post process power-to-kwh task with the ‘use’ feed as the input, and the existing ‘use_kwh’ feed as the target (i.e., overwrite, not create new).

It’ll take a moment especially if the feed is old and large, but it seems to have worked without needing any other feeds stored or altered in any way.

Success!