EmonPI Lockup and Failure

Hi

Something caused my emonPI to lock up tonight. The LCD display had frozen, no response to pressing the button and no response to the network.

So I had to power cycle it.

It appears to have come back to life, but I can see one issue.

On cycling through the LCD display, Power1 & Power2 both indicate “—”. However, both of these feeds are reading and logging values which appear to be correct.

Both CTs along with the Voltage Adapter are shown as detected on the LCD Display during the startup.

Would this be a hardware or software issue I’m looking at?

In the event that the unit has a fault, are replacement EmonPI units still available?

At the moment I’m making a backup of what’s there and I do have a backup which was made last week.

Just to add to those. It’s only the Power1 & Power2 values on the LCD which seem to be the problem. All the other data shown on the LCD such as wifi status, vrms, appear to be fine.

Cheers

Mike

How old is the system / SD Card. Most likely it is an SD Card failure.

https://docs.openenergymonitor.org/emoncms/import.html#update-sd-card-and-import-using-an-usb-sd-card-reader

Hi,

The SD Card is less than a year old, it was replaced back in January this year when the system was showing some corrupted data in the feeds. The emonPI was four years old in August.

I could flash an image onto a spare card to see how that affects the values on the LCD Display.

The emonPI had several reboots last night and the hardware powered off completely to ensure a full reset. The only thing I see is that the Power1 & Power2 values are still indicating “

so, I might leave it till the weekend if it’s still functioning and only the local LCD display values are affected. Is it possible that the power value indications are unrelated to the lockup and that these values on the LCD have been faulty for some time? I don’t visit the electrical cupboard very often to press that button.

But I wouldn’t mind understanding what corruption/failure has taken place to stop only the power values indicated on the display. Should I be looking at any particular log file?

In my mind also, is that the system has grown from only two CTs and a couple of EmonTH’s. To have 245 Inputs, 270 feeds, 10 dashboards and collect/shares feeds and inputs with Home Assistant. It is four years old so I have thought that since the home relies on it so much, I should perhaps consider upgrading to the new version which I see splits off the power monitoring from the database/webserver which means I’m really only looking at a Raspberry PI being the culprit of a Total Failure of the system which to me seems easier to repair.

It is possible you are pushing it a bit, but I’d be surprised if that caused the failure.

Can you access it via SSH?

[edit]
@glyn.hudson @TrystanLea I don’t actually know how the LCD gets it’s data.

Hi, yes I can ssh and ftp into the PI. I had a look though almost every logfile I could last night.

Mike

Can you post the output (use Ctl-E before posting) from df -h, please?

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.8G  2.4G  3.2G  43% /
devtmpfs        333M     0  333M   0% /dev
tmpfs           462M     0  462M   0% /dev/shm
tmpfs           185M  9.1M  176M   5% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs            30M     0   30M   0% /tmp
tmpfs           1.0M  4.0K 1020K   1% /var/lib/php/sessions
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p1  255M   50M  206M  20% /boot
/dev/mmcblk0p3  9.7G  2.8G  6.4G  30% /var/opt/emoncms
log2ram          50M   13M   38M  26% /var/log
tmpfs            93M     0   93M   0% /run/user/1000

Thanks, checking it isn’t full logs.

Right now, is everything working apart from the LCD?

Seems to be only two values on the LCD. Looking at the values from the feeds they seem to be reading correctly for the house usage and solar generation today.

I’m sure someone will ask for this. I’ll respond to other replies later as I have to go out for a few hours.

Server Information
-----------------------

Services
	emonhub :	 Active Running                      
	emoncms_mqtt :	 Active Running                      
	feedwriter :	 Active Running - sleep 300s 1418 feed points pending write
	service-runner :	 Active Running                      
	emonPiLCD :	 Active Running                      
	redis-server :	 Active Running                      
	mosquitto :	 Active Running                      
	demandshaper :	 Not found or not installed                                  
Emoncms
	Version :	 low-write 11.2.11
	Git :	 
		URL :	 https://github.com/emoncms/emoncms.git
		Branch :	 * stable
		Describe :	 11.2.11
	Components :	 Emoncms Core v11.2.11 | App v2.6.8 | EmonHub Config v2.1.5 | Dashboard v2.3.3 | Device v2.2.2 | Graph v2.2.3 | Network Setup v1.0.2 | WiFi v2.1.1 | Backup v2.3.2 | Postprocess v2.2.7 | Sync v2.1.4 | Usefulscripts v2.3.10 | Avrdude-rpi v1.0.1 | Emonhub v2.5.2 | EmonPi v2.9.5 | EmonScripts v1.5.14 | RFM2Pi v1.4.1

Server
	CPU :	 1 Threads(s) | 4 Core(s) | 1 Sockets(s) | Cortex-A53 | 89.60MIPS | 
	OS :	 Linux 5.15.76-v7+
	Host :	 emonpi | emonpi | (192.168.1.212)
	Date :	 2023-09-01 09:56:19 BST
	Uptime :	 09:56:19 up  8:07,  1 user,  load average: 0.27, 0.19, 0.19

Memory
	RAM :	 Used: 25.16%
		Total :	 922.14 MB
		Used :	 232 MB
		Free :	 690.14 MB
	Swap :	 Used: 0.00%
		Total :	 100 MB
		Used :	 0 B
		Free :	 100 MB

Disk
	 :	 - / :	 Used: 40.45%
		Total :	 5.78 GB
		Used :	 2.34 GB
		Free :	 3.13 GB
		Read Load :	 1.6 KB/s
		Write Load :	 1.17 KB/s
		Load Time :	 8 hours 1 mins
	/boot :	 Used: 19.52%
		Total :	 254.99 MB
		Used :	 49.78 MB
		Free :	 205.21 MB
		Read Load :	 0 B/s
		Write Load :	 0.02 B/s
		Load Time :	 8 hours 1 mins
	/var/opt/emoncms :	 Used: 28.31%
		Total :	 9.61 GB
		Used :	 2.72 GB
		Free :	 6.4 GB
		Read Load :	 68.95 B/s
		Write Load :	 1.2 KB/s
		Load Time :	 8 hours 1 mins
	/var/log :	 Used: 25.41%
		Total :	 50 MB
		Used :	 12.7 MB
		Free :	 37.3 MB
		Read Load :	 n/a
		Write Load :	 n/a
		Load Time :	 n/a

HTTP
	Server :	 Apache/2.4.54 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL
	Version :	 10.5.15-MariaDB-0+deb11u1
	Host :	 127.0.0.1 (127.0.0.1)
	Date :	 2023-09-01 09:56:18 (UTC 01:00‌​)
	Stats :	 Uptime: 29253  Threads: 9  Questions: 63666  Slow queries: 0  Opens: 46  Open tables: 39  Queries per second avg: 2.176

Redis
	Version :	 
		Redis Server :	 6.0.16
		PHP Redis :	 6.0.0-dev
	Host :	 localhost:6379
	Size :	 1227 keys (948.68K)
	Uptime :	 0 days

MQTT Server
	Version :	 Mosquitto 2.0.11
	Host :	 localhost:1883 (127.0.0.1)

PHP
	Version :	 8.1.12 (Zend Version 4.1.12)
	Run user :	 User: www-data Group: www-data video Script Owner: pi
	Modules :	 apache2handler calendar Core ctype curl date dom v20031129exif FFI fileinfo filter ftp gd gettext hash iconv json libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 8.1.12openssl pcre PDO pdo_mysql Phar posix readline redis v6.0.0-devReflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer xml xmlreader xmlwriter xsl Zend OPcache zlib 
Pi
	Model :	 Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)
	Serial num. :	 76AB6BF1
	CPU Temperature :	 53.69°C
	GPU Temperature :	 N/A (to show GPU temp execute this command from the console "sudo usermod -G video www-data" )
	emonpiRelease :	 emonSD-10Nov22
	File-system :	 read-write

Client Information
-----------------------

HTTP
	Browser :	 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
	Language :	 en-GB,en-US;q=0.9,en;q=0.8

Window
	Size :	 1709 x 829

Screen
	Resolution :	 1920 x 1080

Better to use Copy as MarkDown button and post without further formatting (looks like this)

Server Information

Server Information

Services

  • emonhub :- Active Running

  • emoncms_mqtt :- Active Running

  • feedwriter :- Active Running - sleep 300s 102 feed points pending write

  • service-runner :- Active Running

  • emonPiLCD :- Active Running

  • redis-server :- Active Running

  • mosquitto :- Active Running

  • demandshaper :- Not found or not installed

Emoncms

Server

  • CPU :- 1 Threads(s) | 4 Core(s) | 1 Sockets(s) | Cortex-A53 | 89.60MIPS |
  • OS :- Linux 5.15.84-v7+
  • Host :- emonpi | emonpi | (192.168.7.243)
  • Date :- 2023-09-01 10:01:31 BST
  • Uptime :- 10:01:31 up 24 days, 1:45, 1 user, load average: 0.24, 0.27, 0.32

Memory

  • RAM :- Used: 30.05%
    • Total :- 922.14 MB
    • Used :- 277.13 MB
    • Free :- 645 MB
  • Swap :- Used: 0.00%
    • Total :- 100 MB
    • Used :- 0 B
    • Free :- 100 MB

Disk

  • **** :- - / :- Used: 42.58%
    • Total :- 5.78 GB
    • Used :- 2.46 GB
    • Free :- 3.01 GB
    • Read Load :- 3.61 B/s
    • Write Load :- 190.87 B/s
    • Load Time :- 13 days 22 hours 25 mins
  • /boot :- Used: 19.54%
    • Total :- 254.99 MB
    • Used :- 49.82 MB
    • Free :- 205.17 MB
    • Read Load :- 0 B/s
    • Write Load :- 0 B/s
    • Load Time :- 13 days 22 hours 25 mins
  • /var/opt/emoncms :- Used: 8.03%
    • Total :- 8.07 GB
    • Used :- 663.85 MB
    • Free :- 7.01 GB
    • Read Load :- 0 B/s
    • Write Load :- 101.97 B/s
    • Load Time :- 13 days 22 hours 25 mins
  • /var/log :- Used: 100.00%
    • Total :- 50 MB
    • Used :- 50 MB
    • Free :- 0 B
    • Read Load :- n/a
    • Write Load :- n/a
    • Load Time :- n/a

HTTP

  • Server :- Apache/2.4.54 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 10.5.15-MariaDB-0+deb11u1
  • Host :- 127.0.0.1 (127.0.0.1)
  • Date :- 2023-09-01 10:01:31 (UTC 01:00‌​)
  • Stats :- Uptime: 2079910 Threads: 6 Questions: 671904 Slow queries: 0 Opens: 46 Open tables: 39 Queries per second avg: 0.323

Redis

  • Version :-
    • Redis Server :- 6.0.16
    • PHP Redis :- 6.0.0-dev
  • Host :- localhost:6379
  • Size :- 316 keys (797.42K)
  • Uptime :- 24 days

MQTT Server

  • Version :- Mosquitto 2.0.11
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 8.1.15 (Zend Version 4.1.15)
  • Run user :- User: www-data Group: www-data video Script Owner: pi
  • Modules :- apache2handler calendar Core ctype curl date dom v20031129exif FFI fileinfo filter ftp gd gettext hash iconv json libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 8.1.15openssl pcre PDO pdo_mysql Phar posix readline redis v6.0.0-devReflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

  • Model :- Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)

  • Serial num. :- 71604C19

  • CPU Temperature :- 55.84°C

  • GPU Temperature :- N/A (to show GPU temp execute this command from the console “sudo usermod -G video www-data” )

  • emonpiRelease :- emonSD-10Nov22

  • File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36
  • Language :- en-GB,en-US;q=0.9,en;q=0.8

Window

  • Size :- 2069 x 1104

Screen

  • Resolution :- 2560 x 1440

If you think you are pushing it too much, the key bit to watch is this. As long as load average is less than one, the CPU is idling at times. Also need to watch memory usage.

The one thing I have seen is that if reading too many MQTT topics, the system can miss some data. Better to send multiple inputs as a single JSON data structure on a single MQTT topic. If you get the structure right, it just replaces existing inputs so you don’t need to redo all the input processing.

Server Information

Server Information

Services

  • emonhub :- Active Running

  • emoncms_mqtt :- Active Running

  • feedwriter :- Active Running - sleep 300s 1418 feed points pending write

  • service-runner :- Active Running

  • emonPiLCD :- Active Running

  • redis-server :- Active Running

  • mosquitto :- Active Running

  • demandshaper :- Not found or not installed

Emoncms

Server

  • CPU :- 1 Threads(s) | 4 Core(s) | 1 Sockets(s) | Cortex-A53 | 89.60MIPS |
  • OS :- Linux 5.15.76-v7+
  • Host :- emonpi | emonpi | (192.168.1.212)
  • Date :- 2023-09-01 09:56:19 BST
  • Uptime :- 09:56:19 up 8:07, 1 user, load average: 0.27, 0.19, 0.19

Memory

  • RAM :- Used: 25.16%
    • Total :- 922.14 MB
    • Used :- 232 MB
    • Free :- 690.14 MB
  • Swap :- Used: 0.00%
    • Total :- 100 MB
    • Used :- 0 B
    • Free :- 100 MB

Disk

  • **** :- - / :- Used: 40.45%
    • Total :- 5.78 GB
    • Used :- 2.34 GB
    • Free :- 3.13 GB
    • Read Load :- 1.6 KB/s
    • Write Load :- 1.17 KB/s
    • Load Time :- 8 hours 1 mins
  • /boot :- Used: 19.52%
    • Total :- 254.99 MB
    • Used :- 49.78 MB
    • Free :- 205.21 MB
    • Read Load :- 0 B/s
    • Write Load :- 0.02 B/s
    • Load Time :- 8 hours 1 mins
  • /var/opt/emoncms :- Used: 28.31%
    • Total :- 9.61 GB
    • Used :- 2.72 GB
    • Free :- 6.4 GB
    • Read Load :- 68.95 B/s
    • Write Load :- 1.2 KB/s
    • Load Time :- 8 hours 1 mins
  • /var/log :- Used: 25.41%
    • Total :- 50 MB
    • Used :- 12.7 MB
    • Free :- 37.3 MB
    • Read Load :- n/a
    • Write Load :- n/a
    • Load Time :- n/a

HTTP

  • Server :- Apache/2.4.54 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 10.5.15-MariaDB-0+deb11u1
  • Host :- 127.0.0.1 (127.0.0.1)
  • Date :- 2023-09-01 09:56:18 (UTC 01:00‌​)
  • Stats :- Uptime: 29253 Threads: 9 Questions: 63666 Slow queries: 0 Opens: 46 Open tables: 39 Queries per second avg: 2.176

Redis

  • Version :-
    • Redis Server :- 6.0.16
    • PHP Redis :- 6.0.0-dev
  • Host :- localhost:6379
  • Size :- 1227 keys (948.68K)
  • Uptime :- 0 days

MQTT Server

  • Version :- Mosquitto 2.0.11
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 8.1.12 (Zend Version 4.1.12)
  • Run user :- User: www-data Group: www-data video Script Owner: pi
  • Modules :- apache2handler calendar Core ctype curl date dom v20031129exif FFI fileinfo filter ftp gd gettext hash iconv json libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 8.1.12openssl pcre PDO pdo_mysql Phar posix readline redis v6.0.0-devReflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

  • Model :- Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)

  • Serial num. :- 76AB6BF1

  • CPU Temperature :- 53.69°C

  • GPU Temperature :- N/A (to show GPU temp execute this command from the console “sudo usermod -G video www-data” )

  • emonpiRelease :- emonSD-10Nov22

  • File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
  • Language :- en-GB,en-US;q=0.9,en;q=0.8

Window

  • Size :- 1709 x 829

Screen

  • Resolution :- 1920 x 1080

Cheers for the hint on the Markdown/copy :slight_smile:

1 Like

Load average and memory use look fine, so I don’t think it is a fundamental problem with the system.

What do you run HA on?

HA runs on its own hardware, another PI in a nice metal case hidden away in the IT cupboard:-) The emonPI I like to leave standard which makes supporting it and getting support far easier.

If I can’t make the changes to its configuration via the web interface I’m usually very reluctant to make changes to the Linux operating system as that takes it away from the standard install which may cause me issues in the future.

The problem I have is that in a previous life part of my duties was as the administrator of the OSISoft PI Server for a Power Station. So that was collecting roughly 45000 tags every say 1 second. So I tend to treat the emonPI as if it can do the same thing and forget that adding several hundred tags could kill it:-) I do occasionally check to see how the PI is performing, and that’s partially a reason I’d like to upgrade to a separate Database/webserver from the Power Measurement.

I’m used to creating batches of thousands of tags and not blinking an eye:-)

But hey, comparing it with an industrial data logging system with a database that supports a hundred client requests for data and tens of thousands of tags, I’m always impressed by what the emonPI can do for its price:-)

But hey, today’s puzzle is these two LCD indications:-)

Via MQTT?

Never by mqtt.

Modbus over TCP/IP, Modbus Serial, OPC, and a couple of proprietary interfaces on a Siemens and Wonderware-based DCS. In the same way, we grouped the Modbus tags to collect around 100 consecutive addresses at a time, and the OPC were in groups of 800.

You create the tags as a list in a Spreadsheet with all their attributes and then import them into the system. I think the biggest single import of tag configuration I ever did was adding just over 14,000 binary and 3,500 floating Point tags when some new Gas Turbines were added, and they all just started reading.

Interesting, I do see that an MQTT software interface is available for that DCS as well now:-) But everything seemed to be heading OPC, or we would insist on Modbus over TCP/IP as 2nd option.

On another note, yesterday I made the decision to upgrade the EmonPI to the newer system with separate EmonTX V4 and emonCMS Base Station. There is so much in the house now that relies on it, and that PI has run now for 4 years I suddenly realised how exposed the house is to what would be a single point of failure.

I think that with the new system, it is easier to replace the base station, and should the emonTX V4 fail, then I don’t have the issue of the other nodes losing data. So that’s going to be installed through the week sometime. I think I’ll run the two in parallel till I see all the feeds updating correctly.

But still, I’ll bottom out the problem with the EmonPI once it’s released from service and hold it as a backup should I ever need it. So I need to understand how the LCD gets its values.

Mike

I know it won’t help much, but…
At startup, the very first message on the LCD comes from the ATMega 328P, because this starts up a lot faster than the Pi. Thereafter, the Pi takes over except when you’ve asked for current and power factor to aid calibration (c command), which come direct from the front end again.
Beyond this, I can’t help you, because I don’t know my way around inside emonHub & emonCMS.

That will be why then :slight_smile: The EmonPi does have an emonhub ModbusTCP interfacer, but it is a little rudimentary.

If you are worried about reliability, an emonTX with a Wi-Fi module hooked into emoncms running on an LXC under PVE on an old laptop is about as robust as it gets (IMHO).