Thanks for all the thoughts, everybody. The system is low-write 9.9.8 (yes, I know). The pi has a battery backup so shouldn’t be suffering from any brownouts. No other recent changes. /var/log is 3% full. I have logging set to warn & errors so most of the time emoncms.log sits there with nothing much happening (for months). I’ll try to check emonhub.log before rebooting if the problem recurs.
I don’t think I run log2ram (there’s no process with that name) because on my system /var/log is a tmpfs and log management is good enough that there’s never a problem (because I’ve controlled the emoncms.log bad boy).
I rebooted via ssh, rather than hardware, and as I reported the system was still working apart from the RF link. The emoncms admin display claimed emonhub was Active running.
I did look at the journal (which is persistent). Here’s the relevant bit for -u emonhub
Jan 16 16:17:44 emonpi sudo[1131]: pam_unix(sudo:session): session closed for user root
Jan 16 16:17:44 emonpi emonhub[1103]: Starting OpenEnergyMonitor emonHub: emonhub has been started ok.
Jan 16 16:17:44 emonpi systemd[1]: Started LSB: Start/stop emonHub.
Jan 18 09:56:07 emonpi systemd[1]: Stopping LSB: Start/stop emonHub...
Jan 18 09:56:10 emonpi sudo[27334]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mkdir -p /var/log/emonhub
Jan 18 09:56:10 emonpi sudo[27334]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 18 09:56:10 emonpi sudo[27334]: pam_unix(sudo:session): session closed for user root
As I say, the data stopped at 21:02 yesterday so there’s nothing in the log except my sudo reboot and the system restarting. Here’s the full journal around the time of the failure:
Jan 17 21:00:03 emonpi CRON[9048]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 17 21:00:03 emonpi CRON[9046]: pam_unix(cron:session): session opened for user pi by (uid=0)
Jan 17 21:00:03 emonpi CRON[9047]: pam_unix(cron:session): session opened for user pi by (uid=0)
Jan 17 21:00:03 emonpi CRON[9060]: (pi) CMD (/home/pi/additional-scripts-djh/tplink-to-emoncms.pl)
Jan 17 21:00:03 emonpi CRON[9059]: (root) CMD (/home/pi/additional-scripts-djh/mk-var-log-djh-scripts)
Jan 17 21:00:03 emonpi CRON[9061]: (pi) CMD (/home/pi/additional-scripts-djh/enphase-to-emoncms.pl)
Jan 17 21:00:03 emonpi CRON[9048]: (CRON) info (No MTA installed, discarding output)
Jan 17 21:00:03 emonpi CRON[9048]: pam_unix(cron:session): session closed for user root
Jan 17 21:00:04 emonpi CRON[9047]: pam_unix(cron:session): session closed for user pi
Jan 17 21:00:05 emonpi CRON[9046]: pam_unix(cron:session): session closed for user pi
Jan 17 21:01:01 emonpi CRON[9093]: pam_unix(cron:session): session opened for user pi by (uid=0)
Jan 17 21:01:01 emonpi CRON[9097]: (pi) CMD (/home/pi/additional-scripts-djh/onehouse-weather-to-emoncms.pl)
Jan 17 21:01:35 emonpi CRON[9093]: pam_unix(cron:session): session closed for user pi
Jan 17 21:02:02 emonpi CRON[9115]: pam_unix(cron:session): session opened for user pi by (uid=0)
Jan 17 21:02:02 emonpi CRON[9119]: (pi) CMD (/home/pi/additional-scripts-djh/wattisham-weather-to-emoncms.pl)
Jan 17 21:02:07 emonpi CRON[9115]: pam_unix(cron:session): session closed for user pi
Jan 17 21:03:00 emonpi CRON[8993]: pam_unix(cron:session): session closed for user pi
Jan 17 21:03:31 emonpi CRON[8991]: pam_unix(cron:session): session closed for user pi
Jan 17 21:05:01 emonpi CRON[9170]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 17 21:05:01 emonpi CRON[9169]: pam_unix(cron:session): session opened for user pi by (uid=0)
Jan 17 21:05:01 emonpi CRON[9179]: (pi) CMD (/home/pi/additional-scripts-djh/tplink-to-emoncms.pl)
Jan 17 21:05:01 emonpi CRON[9180]: (root) CMD (/home/pi/additional-scripts-djh/mk-var-log-djh-scripts)
Jan 17 21:05:02 emonpi CRON[9170]: (CRON) info (No MTA installed, discarding output)
Jan 17 21:05:02 emonpi CRON[9170]: pam_unix(cron:session): session closed for user root
Jan 17 21:05:03 emonpi CRON[9169]: pam_unix(cron:session): session closed for user pi
I’m not sure this sounds much like the other problems people have had.