I am using an emonpi since a year or so to monitor the power generated by my solar panels and the house power consumption. I wrote a an application based on node-red to control a water heater, this way I make a more efficient use of the energy generated by the panels. When running it on the SD card, the system works fine for 2-3 months, then eventually crashes, and upon restart, the lcd screen shows “rasberry pi booting” but it never completes boot. At that point I need to write a new image onto the sd card and off we go.I bought a power supply for the pi with battery integrated, to preserve the sd card getting corrupted because of power outages. But this did not really help.
I then decided to boot from sd, but have root partition on an external ssd drive. I did this following the tutorial https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=44177&sid=2f4a07f3853a8f79c8ece282a5072dd9
The drive is connected to a usb hub that is powered by a 2A supply. The emonpi boots and works fine for 1-2 days, but then stops working. I can not access the emonpi through via ssh anymore. I have noticed that my external ssd spins down and the small led light turns off regularly. Could this be the problem? shoudn’t the drive that is running the operating system be constantly up and running? Do I need to configure the ssd drive in a specific way for this application?
I am running out of ideas for getting a stable system… any help would be appreciated.
You mention an SSD, yet you also say it “spins down.” Are you referring to “sleep” mode?
Or do you have a rotating media hard drive vice an actual SSD?*
You also said that your system crashes after 2-3 months even when it’s running from the SD card.
it sounds like other issues are affecting your SD card data integrity. You may have an SD card
that’s near its end of life. Have you tried a different SD card?
Is the PSU that’s powering the emonPi one that was purchased from the OEM shop?
Have you tried a different PSU?
I’m going to ask what is in the system log when/after it crashes? That’s the first step in investigating causes.
I realize the system log is normally not persistent on emon systems, but I wrote instructions here - Persistent journald logs - because of exactly this type of situation.
Hi Bill,
thanks for taking the time.
sorry the ssd is solid, so it does not spin down. But the LED turns off for long periods of time.
I have tried with 3 different SD cards, and two different power supplies, one of them was the one I ordered from the openenergymonitor shop.
I am trying to get the persistent log as @djh suggests to see if I learn more on what is going on…
Hi, I forgot to mention that I am on an old emonSD-26Oct17. I decided to go for this one as it is a read-only and read through different forums that I could be more stable versus SD card corruption.
I don’t know how to get the contens of Admin > serve, but if you let me know I will be happy to get them for you.
Another topic is that I have noticed the temp files located in /var/cache/man cleaning is failing because the system is read-only. Could this be a source of problems? At this time all files in /var/cache/man are 1.9MB only (du /var/cache/man/ -h).
-Extract of journalctl:
Feb 16 20:02:37 emonpi systemd[1]: Starting Cleanup of Temporary Directories...
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/tr/cat1): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/tr/cat8): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: unlink(/var/cache/man/tr/index.db): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/tr/cat5): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: unlink(/var/cache/man/tr/CACHEDIR.TAG): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: utimensat(/var/cache/man/tr): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/tr): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs/cat1): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs/cat7): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs/cat8): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: unlink(/var/cache/man/cs/index.db): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs/cat5): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: unlink(/var/cache/man/cs/CACHEDIR.TAG): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: utimensat(/var/cache/man/cs): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/ru/cat1): Read-only file system
Did you make /var/log/journal a persisent directory in /home/pi/data as I suggested? If so then it makes perfect sense that it is quite large and larger than the /var/log filesystem, since it is in the /home/pi/data filesystem. Mine is 258M so 65M seems quite small You still have 3.4G to fill!
You have an older system image than mine (Oct2018) but it seems strange that there are various separate filesystems for /var/lib/* and that they are tmpfs especially since /var/lib subdirectories are supposed to be for persistent data, not temporary. Even /var/tmp is supposed to be persistent. So are those changes you have made or did your system come that way?
/var/cache should not be read-only. It seems it is part of the root filesystem and that seems to occupy quite a large space for read-only parts of the system.
In short, your system seems a bit messed up. I’d be tempted to start again with a new image to get to a known state. I believe you can backup and restore your data before and after.
edit: Oh, I don’t think that the inability to clear /var/cache will cause problems.
There is a sticky post on the RaspberryPi forum re SSDs and incompatibility with UAS for some devices although this is for Pi4. I’ve had issues with SSD/HDD but I seem to have tracked that down to me running DietPi. I’ve got a test setup using Raspbian and an SSD and it is working fine so far.
Ok, so I have just realised the correlation here with your other post.
Using the EmonSD image and an SSD is a little fraught because of the way the EmonSD partitions are modified. These modifications are not needed is you are going to run it on an SSD.
Have you anything other than EmonCMS installed on the EmonPi?
I think you might be best to start from a fresh image but I’m wondering how best to do that. Your data may not be where you expect it to be so there is a risk of data loss. Certainly use the Backup menu item and move the backup file off disk to your local machine before you start fiddling. I suspect the data is on the SD card (as I mentioned on the other thread).
Once you have EmonCMS installed and it is running OK, import your data and once that is OK, move the new rootfs to the SSD, overwriting all that was there. Make sure you stop all the emon services first.
If this was a fresh SSD or you are convinced the data is not on it, the better way, is to move the rootfs before installing EmonCMS I’d suggest.
This is all theory - I have not tried it (I have moved rootfs but not with EmonCMS)!
So I got an unexpected crash. Since then my node-red functions are not behaving correctly, and the shell reactivity is slow, it is as the system is dying in my arms
The log before the crash does not indicate anything strange, there was no power outage at that time:
note: sda1 is my external SSD where I have /
Feb 25 08:17:45 emonpi systemd[1]: Failed to start Advanced key-value store.
Feb 25 08:17:46 emonpi systemd[1]: Failed to start LSB: Apache2 web server.
-- Reboot --
Feb 25 09:00:21 emonpi kernel: EXT4-fs error (device sda1): ext4_find_entry:1463: inode #1179751: comm systemd-udevd: reading directory lblock
Feb 25 09:00:35 emonpi kernel: Aborting journal on device sda1-8.
Feb 25 09:00:35 emonpi kernel: JBD2: Error -5 detected when updating journal superblock for sda1-8.
Feb 25 09:00:35 emonpi kernel: EXT4-fs (sda1): Remounting filesystem read-only
Feb 25 09:00:36 emonpi kernel: EXT4-fs error (device sda1): ext4_find_entry:1463: inode #787106: comm systemd-udevd: reading directory lblock 0
Feb 25 09:00:37 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:37 emonpi kernel: EXT4-fs error (device sda1): ext4_find_entry:1463: inode #1179751: comm systemd-udevd: reading directory lblock
Feb 25 09:00:37 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:37 emonpi kernel: EXT4-fs error (device sda1): __ext4_get_inode_loc:4341: inode #790878: block 3146037: comm kworker/u8:0: unable
Feb 25 09:00:38 emonpi kernel: EXT4-fs error (device sda1): __ext4_get_inode_loc:4341: inode #264289: block 1048742: comm systemd: unable to re
Feb 25 09:00:38 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:38 emonpi kernel: EXT4-fs error (device sda1): __ext4_get_inode_loc:4341: inode #264289: block 1048742: comm systemd: unable to re
Feb 25 09:00:38 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:39 emonpi kernel: EXT4-fs error (device sda1): __ext4_get_inode_loc:4341: inode #131200: block 524327: comm (umount): unable to re
Feb 25 09:00:40 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:40 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:40 emonpi kernel: EXT4-fs error (device sda1): __ext4_get_inode_loc:4341: inode #790878: block 3146037: comm kworker/u8:0: unable
Feb 25 09:00:41 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:41 emonpi kernel: EXT4-fs error (device sda1): __ext4_get_inode_loc:4341: inode #1189204: block 4719221: comm java: unable to read
Feb 25 09:00:41 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:41 emonpi kernel: EXT4-fs error (device sda1): ext4_find_entry:1463: inode #1179650: comm kworker/u8:0: reading directory lblock 0
Feb 25 09:00:41 emonpi kernel: EXT4-fs (sda1): previous I/O error to superblock detected
Feb 25 09:00:42 emonpi kernel: EXT4-fs error (device sda1): ext4_find_entry:1463: inode #1179649: comm systemd-udevd: reading directory lblock
Feb 25 09:00:50 emonpi kernel: EXT4-fs error (device sda1): ext4_find_entry:1463: inode #787106: comm systemd-udevd: reading directory lblock 0
Thanks for the reply. Wait I hear here is that it is not such a good idea to run the emonSD on an SSD as there is not much heritage… so I might just buy a good SD card and go to the nominal install, maybe with the latest image.
Hmmm, looks like a corrupted filesystem, to me. Its flagging several inodes as unreadable.
That may mean a mangled inode table, mangled files, or bad media.
You still have 3.4G to fill!
