Old thread, but I have experienced this twice past month. I have been using emoncms on an RPI3b+ without any issues whatsoever from January to November 2019. I think it has been rebooted once, maybe. Then I swithced to a new Emonpi (bought in 2019). After about one or two weeks something suddenly crashed during the night, making it impossible to log into it (some overflow error appeared). I rebooted it by using the button on the emonpi to shut it down, then cycle power and everything looked good when it rebooted. Now, one week and a few hours after the first incident, it happened again yesterday morning. Same procedure with shutdown/reboot/power cycle and now ok.
During the problem period, data becomes garbled. The CT data are restored, since they are logged using Iotawatt, which will hold the data and fill up the emoncms db once up and running again. But - the calculated data becomes garbled for the period between the crash and reboot. For example. power calculations become very high and this obviously disturbs the statistics.
My primary concern is however that the entire setup is unstable. I am away in periods, and cannot rely on physically be there to recycle power every 7 days…
Is there any way to track down the root cause (I am a real newbie to linux/RPI)? Is this error known?
Administration | App v2.0.9 | Backup v2.1.7 | EmonHub Config v2.0.4 | Dashboard v2.0.5 | Device v2.0.3 | EventProcesses | Feed | Graph v2.0.6 | Input | Postprocess v2.1.2 | CoreProcess | Schedule | Network Setup v1.0.0 | sync | Time | User | Visualisation | WiFi v2.0.2
emonpi | emonpi | (xx.xx.xx.xx)
2019-12-02 09:30:40 CET
09:30:40 up 14:54, 1 user, load average: 0.35, 0.17, 0.12
Apache/2.4.25 (Raspbian) HTTP/1.1 CGI/1.1 80
2019-12-02 09:30:40 (UTC 01:00)
Uptime: 53746 Threads: 2 Questions: 55840 Slow queries: 0 Opens: 27 Flush tables: 1 Open tables: 21 Queries per second avg: 1.038
427 keys (824.08K)Flush
Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)
51.54°C - 51.5°C
There have been odd reports of this - there was an issue with the logs filling up, but that should be fixed.
The best way to move forward, I suggest, is to upgrade the system to the latest EmonSD image (export and import your data) - that then removes any possibility of it being an problem with historic SW issues.
I’d also start with a new SD Card so that; a) you have a roll back position, and b) removes any possibility the card is failing (though I doubt that).
Thanks, will do that. I would like to increase the SD card size at the same time. Would a 32GB card be ok?
Yes should be. I would say though, that volume of data is pretty slow to backup, so you might want to consider a system using something with more reliable storage such as an SSD.
Ok, then I wonder - would an USB connected SSD be faster and/or more reliable as a permanent data storage device? Maybe even recommended?
It really depends on how much data you actually have, and how important it is.
EmonCMS is really efficient in terms of data storage, a 16GB card has about 10GB of data storage. If you think that is not enough, you may need to look at alternatives. If you check, I anticipate your data storage is less than you expect.
Yeah, I know. Right now I have spent about 1.6% of the 10GB, and with the current write rate, it will take years to fill it. But I already ordered a 32GB card (easily available, fast and quite cheap), and so I will be using that.
If an SSD were connected via USB3, it would be faster than a µSD card.
More reliable? Without a doubt, yes. The larger capacity of a SSD means the wear leveling
“mechanism” can spread the “wear” over a much greater amount of flash memory.
In a nutshell, it’s one of the reasons SSDs have lifetimes measured in Terabytes written (TBW).
Yes., that’s Terabytes, as in 1012 bytes. Not likely to get that type of performance
from an SD card. Here’s a couple of scenario examples:
A typical TBW figure for a 250 GB SSD lies between 60 and 150 terabytes written. That means: To get over a guaranteed TBW of 70, a user would have to write 190(!) GB daily over a period of one year (In other words, to fill two thirds of the SSD with new data every day).
A normal office user writes approximately between 10 and 35 GB on a normal day. Even if one raises this amount up to 40 GB, it means that they could write (and only write) more than almost 5 years until they reach the 70 TBW limit.
Ref: SSD Lifespan: How Long do SSDs Really Last?
Recommended? Yes, definitely!
Good to know. I was slightly unsure if this was true or not. Writing would be no problem - it would be reading large amounts I wondered about (for graphing etc.).
I should have been a bit more specific. i.e. a drive connected via USB 3.1 will be faster.
But delving into it a little deeper shows that it’s not by much. i.e. some of today’s media can hit
fairly high speeds, and it looks like that may change. Here’s some more info:
From the article at the link below:
SD Express , a joint effort with PCI-SIG to create SD cards that are compatible with PCIe 3.0/NVMe v1.3 protocols and will offer peak theoretical speeds of 985MB/s.
As with most speed tests, coming up with a definitive answer isn’t easy.
@Bill.Thomson, how does SSD v eMMC v SD Crad fare in terms of reliability and longevity?
I’m trying to get a Pi4 running off an SSD (orHDD) reliably (booting from SD rootfs moved to drive) but the USB comms seem to keep failing. I’m therefore looking at alternatives and one is a unit with eMMC.
SSD longeveity is measured in TBW. (TeraBytes Written)
From the link in post #7 in this thread:
A typical TBW figure for a 250 GB SSD lies between 60 and 150 terabytes written. That means: To get over a guaranteed TBW of 70, a user would have to write 190(!) GB daily over a period of one year (In other words, to fill two thirds of the SSD with new data every day). In a consumer environment this is highly unlikely.
The good news: These manufacturer figures are even lower than the real TBWs detected in a long-term test conducted by Germany´s most respected IT and Computer magazine c´t and the Heise publishing company. In the magazine´s test, they bought two SSDs each of the 12 most popular products available in 2016 and tested those for one year until the end of June 2017. The SSDs that were tested were OCZ TR150, Crucial BX 200, Samsung 750 Evo, Samsung 850 Pro, SanDisk Extreme Pro, and SanDisk Ultra II.
The outcome of the tests conducted were astonishing: All of the drives tested were able to write more data than what was promised by the producer. Even cheaper drives were able to write more data than promised: The Crucial BX 200 drives were able to write 187 TB and 280 TB – that is more than 2.5 times the figure promised.
One of the Samsung SSD 850 PRO drives achieved a figure of 9.1 petabytes of data written! That’s 60 times the TBW figure Samsung promises on their data sheets. The other Samsung product – the Samsung SSD 750 Evo – was able to write 1.2 petabytes of data, which equals (in theory) to more than 80 years of constant writing. However, the pro models showed why their price is higher: None of them did write less than 2.2 Petabyte of data.
Given we’ve seen SD card failures on RPis running emonCMS, it would appear to be no contest
when it comes to which one is going to last the longest.
I’ve almost no experience with eMMC. It’s quite a bit faster than SD, but that’s about all I know of it.
Do you have more than one drive attached to the Pi you’re having the USB issues with?
Yes I saw that, I was wondering how the technology used in an SSD related to eMMC. Is eMMC closer to SD or SSD?
Seems the underlying ‘NAND’ technology is basically the same in all of them but it is the controller that adds the life.
No. I have tried both an SSD and an HDD. I’m beginning to think it is either a Pi4 issue or a Buster issue. I’ve moved the HDD back onto the Pi3 it was on for a long time (with Stretch) to see what happens. If it falls over - I’m suspecting Buster, if it doesn’t then I’m looking at the Pi4 as the culprit.
I’ve been looking at alternative platforms and these MiniPCs. There are others with just eMMC so I wondered on the longevity of the storage.
Fell over - I think it is a Buster issue. Currently DietPi, so need to load a Raspbian image, move rootfs to the SSD and then see if it is DietPi Buster or all Buster!
A cpouple of years ago, I wanted something that had a bit more “horsepower” than a Pi3.
I managed to find one of these for 100 bucks:
What I like about it, aside from the fact that it’s a complete system, the power consumption is
only a little more than a Pi - about 6 Watts during normal day-to-day operation, and 8 to 10 Watts
when doing anything more than just running InfluxDB and Grafana. The 500 GB HDD is nice too.
No worries about running out of space any time soon and plenty of oomph in the CPU department.
I don’t think that helps the OP. The emonpi is a pi 3B+ AFAIK and that doesn’t have any USB 3 ports. You’d need a pi 4 for that.
Hence my qualifying it with an if.
Long time since I visited, interesting discussion here! Regarding my initial question - the emonpi became stable after updating to emonSD-17Oct19.
The second thing, however, is not solved. When updating the emonSD, the original 16GB microSD card was replaced by a faster 32GB. But the /var/opt/emoncms size was not increased, still is about 10GB. How should I go about to allocate the extra space?
I use the Oct19 image and 32GB SDHC’s. This is what I did from an ssh terminal (copied from my Build Notes) …
Do sudo parted
Then do print
This will show #3 as the Ext2 data partition which is to be expanded.
Then do resizepart and respond with 3
Respond Yes to the question about the partition already being mounted
At the prompt End? (15.5GB)? … enter the new ending point in MB …
I used 31600
Verify the change has been made by doing print
Finish by doing quit
Finally expand the Ext2 file system to use its newly enlarged partition …
Do sudo resize2fs /dev/mmcblk0p3
Verify all is OK by doing df -hT
Hope this helps