Data not stored through reboot

borpin · 2020-04-01T11:27:35.542Z

cmcg:

but the mysql file date has not changed since I first removed the card. So it seems like the database isn’t storing the data.

The mysql file only stores metadata so is not constantly updated. the other files store the actual data.

cmcg:

all data back to the back up point is lost, like it’s only in memory but not actually writing to the card or something.

How do you mean Lost if the feeds are updating?

If you see updates in the Inputs and updates in the Feeds, then the data is updating.

cmcg · 2020-04-01T11:54:25.039Z

emonpiGraph1616×955 119 KB

Here’s how it looks right now. On the left of the graph, you can see that I took it down to make the backup image at 2pm yesterday. It was only down for about a hour and then appeared normal after I restarted about 3pm, with just one hour of missing data. But at the next reboot, the data added after the restart is missing from the graph. I repeated this several times in the evening, last rebooting a little after 11pm, which is where this graph picks up. Each time, all data back to 2pm disappears.

The feeds do look normal, so I’m hoping that data loss is minimal, but are you able to see what I mean about it appearing to not store new data?

borpin · 2020-04-01T12:23:08.488Z

My guess is that after the attempted upgrade, one of the services did not start correctly. Did you check the service state in the Admin page (if it had not updated it might not show up).

It be the feedwriter was the issue, so yes the data is not written to disk.

What does your Admin page look like (screenshot)? What version does it say you are on?

cmcg · 2020-04-01T12:29:53.564Z

adminpage1221×1222 157 KB

Here it is. Version is still low-write 10.1.0 but some modules did update. Like, I previously downgraded Backup to get that working but it’s up to date again now. Feedwriter looks fairly normal doesn’t it?

borpin · 2020-04-01T12:52:27.215Z

cmcg:

Feedwriter looks fairly normal doesn’t it?

Yes it does.

Odd. Try clicking on the reboot button further down and see whether the data is there. It should be, but there is only one way to find out sadly.

cmcg · 2020-04-01T13:15:43.077Z

dashboard11298×774 82.8 KB

Sadly, it’s as before. I had to scroll back a little to show yesterday’s data. Then there is just a gap until the new current data begins.

dashboard21301×452 24 KB

Despite this, the .dat files continue to grow. This was earlier, repeated an hour apart…

datFiles887×1185 295 KB

And here they are after the reboot just now…

datFiles2881×579 151 KB

borpin · 2020-04-01T13:26:30.165Z

@TrystanLea one for you…

cmcg · 2020-04-01T13:47:42.206Z

Thanks for your time Brian. I’m definitely out of my depth with this one!

TrystanLea · 2020-04-01T16:43:56.604Z

Hello @cmcg in the screenshot above, next to feedwriter, it states ‘63 feed points pending write’ I assume that does keep resetting to 0 every 5minutes or so? The fact that the feeds show that they are updated on disk and that I can see both the disk usage and write update time updated suggest that that is happening.

I havent seen this behaviour before, so makes me wonder if there has been some sort of disk corruption. It might be worth writing the latest SD card image to a fresh SD card and then importing your historic data using this approach: Update & Upgrade - Guide | OpenEnergyMonitor

cmcg · 2020-04-01T17:03:32.980Z

Hi Trystan,

Yes, it seems normal. Sometimes there is more to do, sometimes less.

services787×343 32.5 KB

djh · 2020-04-01T19:35:58.963Z

Why is the ownership of some of the data files different to others? Some of the pi:pi ones haven’t been written to for quite while but others have.

cmcg · 2020-04-01T20:17:34.835Z

I’m guessing the default ownership must have changed with a previous version upgrade. The five that have not changed since August align with five feeds I stopped using when I added my EmonTX.

cmcg · 2020-04-02T12:21:21.513Z

An update to this. I’ve realised that the entire /home/pi/data partition is acting as tempfs. No changes to files in there remain after a reboot. The phpfina dat files give the impression that they are always growing, even through reboots, but I wonder if they just fill the time gap with zeros each time they start up again?
Could something have happened with an overlay file system to cause this?
I have a replacement sdcard arriving tomorrow so I’ll be able to take your advice then @TrystanLea
Does this look normal?

mount1181×451 134 KB

borpin · 2020-04-02T13:53:29.691Z

Rather than a screenshot, select the text and paste it directly with 3 backticks before and after.

cmcg:

I’ve realised that the entire /home/pi/data partition is acting as tempfs.

Not sure you are right…

You wouldn’t have any data if that was true I think.

What does df -h tell you?

It is odd…

djh · 2020-04-02T15:12:20.640Z

borpin:

What does df -h tell you?

df -hT

gives a little more information.

cmcg · 2020-04-02T16:38:00.129Z

Here’s the output of df-hT

pi@emonpi:~ $ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      3.9G  2.1G  1.6G  57% /
devtmpfs       devtmpfs  484M     0  484M   0% /dev
tmpfs          tmpfs     489M     0  489M   0% /dev/shm
tmpfs          tmpfs     489M  6.6M  482M   2% /run
tmpfs          tmpfs     5.0M  4.0K  5.0M   1% /run/lock
tmpfs          tmpfs     489M     0  489M   0% /sys/fs/cgroup
tmpfs          tmpfs      50M  3.0M   48M   6% /var/log
tmpfs          tmpfs      30M     0   30M   0% /tmp
tmpfs          tmpfs     1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p1 vfat       43M   22M   21M  52% /boot
/dev/mmcblk0p3 ext2      3.3G  233M  2.9G   8% /home/pi/data
tmpfs          tmpfs      98M     0   98M   0% /run/user/1000

Appreciate this probably looks pretty normal.
Here’s an example where I just did “touch data/test.txt” and then rebooted. As well as test.txt disappearing, notice how wificheck.log also reverts to a version from just before this issue began.

pi@emonpi:~ $ ls -al data/
total 87
drwxrwxrwx 12 pi       pi        1024 Apr  2 12:45 .
drwxr-xr-x 27 pi       pi        4096 Oct  3 19:58 ..
-rw-r--r--  1 root     root         0 Oct 30  2018 dhcpd.leases
drwxrwxrwx  3 root     root      1024 Oct 30  2018 @eaDir
-rw-r--r--  1 root     root         0 Jun 21  2019 emonbase
-rw-r--r--  1 pi       pi        8761 Jun 21  2019 emoncms-backup-2019-06-21.tar.gz
-rw-rw-r--  1 pi       www-data     0 Oct 30  2018 emoncms.conf
-rw-r--r--  1 pi       pi        1348 Jun 21  2019 emoncms-wifiscan.log
-rw-rw-rw-  1 pi       www-data  8063 Oct  3 19:57 emonhub.conf
-rw-r--r--  1 pi       pi       40286 Oct  6 15:53 emonpiupdate.log
drwx------  2 pi       pi       12288 Aug 16  2018 lost+found
drwxrwxrwx  5 mysql    mysql     1024 Mar 31 13:03 mysql
drwxrwxrwx  2 pi       pi        1024 Mar 27 12:16 phpfina
drwxr-xr-x  2 www-data root      1024 Jun 21  2019 phpfina1
drwxr-xr-x  2 pi       pi        1024 Jun 22  2019 phpfina2
drwxr-xr-x  2 www-data root      1024 Mar 27 12:17 phptimeseries
drwxrwxrwx  2 pi       pi        1024 Jun 21  2019 System Volume Information
-rw-r--r--  1 pi       pi           0 Apr  2 12:45 test.txt
drwxrwxrwt  2 root     root      1024 Oct 18  2018 @tmp
drwxr-xr-x  2 www-data root      1024 Oct 22  2018 uploads
-rw-r--r--  1 root     root       184 Apr  2 12:45 wificheck.log

pi@emonpi:~ $ ls -al data/
total 87
drwxrwxrwx 12 pi       pi        1024 Mar 30 08:18 .
drwxr-xr-x 27 pi       pi        4096 Oct  3 19:58 ..
-rw-r--r--  1 root     root         0 Oct 30  2018 dhcpd.leases
drwxrwxrwx  3 root     root      1024 Oct 30  2018 @eaDir
-rw-r--r--  1 root     root         0 Jun 21  2019 emonbase
-rw-r--r--  1 pi       pi        8761 Jun 21  2019 emoncms-backup-2019-06-21.tar.gz
-rw-rw-r--  1 pi       www-data     0 Oct 30  2018 emoncms.conf
-rw-r--r--  1 pi       pi        1348 Jun 21  2019 emoncms-wifiscan.log
-rw-rw-rw-  1 pi       www-data  8063 Oct  3 19:57 emonhub.conf
-rw-r--r--  1 pi       pi       40286 Oct  6 15:53 emonpiupdate.log
drwx------  2 pi       pi       12288 Aug 16  2018 lost+found
drwxrwxrwx  5 mysql    mysql     1024 Mar 31 13:03 mysql
drwxrwxrwx  2 pi       pi        1024 Mar 27 12:16 phpfina
drwxr-xr-x  2 www-data root      1024 Jun 21  2019 phpfina1
drwxr-xr-x  2 pi       pi        1024 Jun 22  2019 phpfina2
drwxr-xr-x  2 www-data root      1024 Mar 27 12:17 phptimeseries
drwxrwxrwx  2 pi       pi        1024 Jun 21  2019 System Volume Information
drwxrwxrwt  2 root     root      1024 Oct 18  2018 @tmp
drwxr-xr-x  2 www-data root      1024 Oct 22  2018 uploads
-rw-r--r--  1 root     root       184 Mar 31 13:00 wificheck.log

borpin · 2020-04-02T21:09:23.783Z

I have no idea - @TrystanLea one for you…

TrystanLea · 2020-04-03T08:26:03.076Z

cmcg:

I have a replacement sdcard arriving tomorrow so I’ll be able to take your advice then @TrystanLea

Great, I think this will be worthwhile anyway as your image version is an old version. It’s probably not worth delving deeper into the current one.

cmcg · 2020-04-03T15:38:18.361Z

Glad to say that a new SD card with emonSD-17Oct19.img fully updated and with my previous backup restored has got me to a fully up-to-date and working state. A little work to do on Node-Red flows and restoring my custom dark theme, but that’s all.
Thanks for your help @borpin @TrystanLea @djh

TrystanLea · 2020-04-04T07:52:51.503Z

Great to hear @cmcg!