Community
OpenEnergyMonitor

Community

EmonPi crashes running OS on SSD

Tags: #<Tag:0x00007f8de5f11f88>

Hi,

I am using an emonpi since a year or so to monitor the power generated by my solar panels and the house power consumption. I wrote a an application based on node-red to control a water heater, this way I make a more efficient use of the energy generated by the panels. When running it on the SD card, the system works fine for 2-3 months, then eventually crashes, and upon restart, the lcd screen shows “rasberry pi booting” but it never completes boot. At that point I need to write a new image onto the sd card and off we go.I bought a power supply for the pi with battery integrated, to preserve the sd card getting corrupted because of power outages. But this did not really help.

I then decided to boot from sd, but have root partition on an external ssd drive. I did this following the tutorial https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=44177&sid=2f4a07f3853a8f79c8ece282a5072dd9
The drive is connected to a usb hub that is powered by a 2A supply. The emonpi boots and works fine for 1-2 days, but then stops working. I can not access the emonpi through via ssh anymore. I have noticed that my external ssd spins down and the small led light turns off regularly. Could this be the problem? shoudn’t the drive that is running the operating system be constantly up and running? Do I need to configure the ssd drive in a specific way for this application?

I am running out of ideas for getting a stable system… any help would be appreciated.

Luis

You mention an SSD, yet you also say it “spins down.” Are you referring to “sleep” mode?
Or do you have a rotating media hard drive vice an actual SSD?*

You also said that your system crashes after 2-3 months even when it’s running from the SD card.
it sounds like other issues are affecting your SD card data integrity. You may have an SD card
that’s near its end of life. Have you tried a different SD card?

Is the PSU that’s powering the emonPi one that was purchased from the OEM shop?
Have you tried a different PSU?

I’m going to ask what is in the system log when/after it crashes? That’s the first step in investigating causes.

I realize the system log is normally not persistent on emon systems, but I wrote instructions here - Persistent journald logs - because of exactly this type of situation.

Hi Bill,
thanks for taking the time.
sorry the ssd is solid, so it does not spin down. But the LED turns off for long periods of time.
I have tried with 3 different SD cards, and two different power supplies, one of them was the one I ordered from the openenergymonitor shop.
I am trying to get the persistent log as @djh suggests to see if I learn more on what is going on…

@luisrodriguez7 would it be possible to copy and paste here the contents of Admin > Server Information?
Have you tried the most recent image?

Are the logs filling up?

do a

df -h

and see if var/log is filling up.

Adding other programs (such as Node-Red) to an EmonPi can have unexpected consequences.

Hi, I forgot to mention that I am on an old emonSD-26Oct17. I decided to go for this one as it is a read-only and read through different forums that I could be more stable versus SD card corruption.
I don’t know how to get the contens of Admin > serve, but if you let me know I will be happy to get them for you.

Hi @borpin, the df -h shows the following:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        20G  2.3G   17G  13% /
devtmpfs        481M     0  481M   0% /dev
tmpfs           486M     0  486M   0% /dev/shm
tmpfs           486M  520K  485M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           486M     0  486M   0% /sys/fs/cgroup
tmpfs            40M  6.6M   34M  17% /var/lib/openhab
tmpfs           1.0M     0  1.0M   0% /var/lib/dhcpcd5
tmpfs           1.0M  8.0K 1016K   1% /var/lib/dhcp
tmpfs            50M  6.2M   44M  13% /var/log
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            30M   36K   30M   1% /tmp
/dev/mmcblk0p3  3.8G  237M  3.4G   7% /home/pi/data
/dev/mmcblk0p1   60M   22M   39M  37% /boot
/dev/sda2       897G   79M  851G   1% /mnt

So, /var/log at 6.2MB. But when I do du /var/log/ -h, I get 71MB!

0       /var/log/supervisor
4.0K    /var/log/mosquitto
8.0K    /var/log/logrotate
8.0K    /var/log/openhab
4.0K    /var/log/mysql
4.0K    /var/log/apache2
4.0K    /var/log/redis
4.0K    /var/log/emonpilcd
65M     /var/log/journal/5e3a8d5cbaff427898fd4b9ec446e38e
65M     /var/log/journal
71M     /var/log/

Another topic is that I have noticed the temp files located in /var/cache/man cleaning is failing because the system is read-only. Could this be a source of problems? At this time all files in /var/cache/man are 1.9MB only (du /var/cache/man/ -h).

-Extract of journalctl:

Feb 16 20:02:37 emonpi systemd[1]: Starting Cleanup of Temporary Directories...
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/tr/cat1): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/tr/cat8): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: unlink(/var/cache/man/tr/index.db): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/tr/cat5): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: unlink(/var/cache/man/tr/CACHEDIR.TAG): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: utimensat(/var/cache/man/tr): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/tr): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs/cat1): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs/cat7): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs/cat8): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: unlink(/var/cache/man/cs/index.db): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs/cat5): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: unlink(/var/cache/man/cs/CACHEDIR.TAG): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: utimensat(/var/cache/man/cs): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/cs): Read-only file system
Feb 16 20:02:37 emonpi systemd-tmpfiles[6924]: rmdir(/var/cache/man/ru/cat1): Read-only file system

Did you make /var/log/journal a persisent directory in /home/pi/data as I suggested? If so then it makes perfect sense that it is quite large and larger than the /var/log filesystem, since it is in the /home/pi/data filesystem. Mine is 258M so 65M seems quite small :slight_smile: You still have 3.4G to fill!

You have an older system image than mine (Oct2018) but it seems strange that there are various separate filesystems for /var/lib/* and that they are tmpfs especially since /var/lib subdirectories are supposed to be for persistent data, not temporary. Even /var/tmp is supposed to be persistent. So are those changes you have made or did your system come that way?

/var/cache should not be read-only. It seems it is part of the root filesystem and that seems to occupy quite a large space for read-only parts of the system.

In short, your system seems a bit messed up. I’d be tempted to start again with a new image to get to a known state. I believe you can backup and restore your data before and after.

edit: Oh, I don’t think that the inability to clear /var/cache will cause problems.

There is a sticky post on the RaspberryPi forum re SSDs and incompatibility with UAS for some devices although this is for Pi4. I’ve had issues with SSD/HDD but I seem to have tracked that down to me running DietPi. I’ve got a test setup using Raspbian and an SSD and it is working fine so far.

https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=245931

A couple of hours ago I saw this, which made me think of the issue you’re having.
After watching it, I figured you’d have it up and runnin’ again. thumbsup

Yes, I think it is an issue with DietPi. It is quick with an SSD!

A rough guide…

[email protected]:~ $ sudo hdparm -t /dev/sda
/dev/sda:
 Timing buffered disk reads: 882 MB in  3.00 seconds = 293.98 MB/sec

SD card

[email protected]:~ $ sudo hdparm -t /dev/mmcblk0
/dev/mmcblk0:
 HDIO_DRIVE_CMD(identify) failed: Invalid argument
 Timing buffered disk reads: 132 MB in  3.04 seconds =  43.46 MB/sec

SWEET! beer_cheer

1 Like