No Inputs and /var/log full within 1h of boot

OK, thanks for the heads-up. Post deleted.

Thanks Paul,
I did notice that emonhub logs were 10Mb, it was syslog and daemon.log that appeared largest around 22Mb.

I didn’t know about du -h however, I’ve done a power cycle, so not full “yet”.
Gives me:

4.0K	/var/log/supervisor
8.0K	/var/log/mosquitto
0	/var/log/logrotate
0	/var/log/openhab
4.0K	/var/log/mysql
156K	/var/log/apache2
8.0K	/var/log/redis
44K	/var/log/emonhub
4.0K	/var/log/emonpilcd
17M	/var/log

/var/log is now at 20M in the few minutes it’s taken me to write this.
Doing sudo du -h /var/log/* gives me:

196K	/var/log/apache2
16K	/var/log/auth.log
28K	/var/log/boot.log
0	/var/log/btmp
9.8M	/var/log/daemon.log
4.0K	/var/log/dataplicity.log
4.0K	/var/log/emoncms.log
56K	/var/log/emonhub
4.0K	/var/log/emonpilcd
240K	/var/log/kern.log
0	/var/log/logrotate
240K	/var/log/messages
8.0K	/var/log/mosquitto
0	/var/log/mqtt_input.log
4.0K	/var/log/mysql
0	/var/log/mysql.log
0	/var/log/ntp_update.log
0	/var/log/openhab
8.0K	/var/log/redis
4.0K	/var/log/service-runner.log
4.0K	/var/log/supervisor
10M	/var/log/syslog
4.0K	/var/log/ufw.log
4.0K	/var/log/wtmp

emonHub has only been running 10-15 minutes at most.

Inputs view still empty with spinning loader

Thanks for pointing that out though Bill, have changed level anyway, can’t do any harm. WIll try to remember when I need to debug.

You’re not the first to report this, but I’m afraid I’m not sure what the answer is. but now you know where the log entries are piling up, syslog and daemon.log, is there any clues when you check those log files?

Actually, something I hadn’t noticed earlier is the tail on both logs is all node-red.
I’ve stopped node-red and that seems to have stopped them filling up so rapidly.

Trying to run node-red upgrade script, see if that helps, then best see what settings are for logs there if I can work it out.

Should probably update system too, one step at a time!

@waako trying to trace the error line numbers you posted above, the first in input_model.php corresponds to the input clean method, which seems strange to have been triggered from a mqtt_input reset as there are no calls to this method from the script. Are you running the emoncms stable branch on a standard emonSD setup?

For the inputs spinning issue: can you post the result of emoncms/input/list.json?

Thanks @TrystanLea ,

Yes, I am running stable branch with emonSD setup, and emoncms/input/list.json gives me:
Fatal error: Call to a member function fetch_object() on boolean in /var/www/emoncms/Modules/input/input_model.php on line 377

I notice this error came up in Emoncms Apps: Call to a member function fetch_object() on a non-object? But I’ve tried running database updates, but they don’t stick and have this issue on the page:
Notice: Trying to get property of non-object in /var/www/emoncms/Lib/dbschemasetup.php on line 41

The logs were indeed being filled by Node Red’s telegram node, removed it and logs issue fixed, I’d already moved to using chatbot anyway.
I’m guessing it had something to do with node-red-contrib-telegrambot wanting NodeJS version 5.x and I’m on 4.8. Although unsure why it suddenly became a problem.

Thanks @waako, sorry line 377 does make sense, its the line where emoncms loads the inputs from mysql into redis: https://github.com/emoncms/emoncms/blob/stable/Modules/input/input_model.php#L377

It sounds like something has gone a bit amiss in the inputs table in the emoncms mysql database, perhaps the table has crashed somehow, as it appears that emoncms is trying to recreate all fields!

Would you be able to login to mysql on the pi locally and check the inputs table? try:

mysql -u emoncms -p emoncms

password: emonpiemoncmsmysql2016

SELECT * FROM inputs;

Then you could try (I’ve only ever had to do this once 5+ years ago so not sure if it will sort it):

REPAIR TABLE inputs;

So, this would explain the issue of inputs not loading :confused:

mysql> SELECT * FROM inputs;
ERROR 1146 (42S02): Table 'emoncms.inputs' doesn't exist

Try “input” rather than “inputs

Actually there’s an input table however (sorry trying to respond and cook dinner is probably not the most sensible for either tasks):

mysql> show tables;
+-------------------+
| Tables_in_emoncms |
+-------------------+
| app_config        |
| dashboard         |
| feeds             |
| graph             |
| input             |
| multigraph        |
| rememberme        |
| schedule          |
| setup             |
| users             |
+-------------------+
10 rows in set (0.00 sec)

However, that gives me:

mysql> SELECT * FROM input;
ERROR 130 (HY000): Incorrect file format 'input'

Tried to repair anyway:

mysql> REPAIR TABLE input;
+---------------+--------+----------+-------------------------------+
| Table         | Op     | Msg_type | Msg_text                      |
+---------------+--------+----------+-------------------------------+
| emoncms.input | repair | Error    | Incorrect file format 'input' |
| emoncms.input | repair | error    | Corrupt                       |
+---------------+--------+----------+-------------------------------+
2 rows in set (0.00 sec)

I ran mysqlcheck and it seems to only be the input table which has problems.

using use_frm repaired the table

mysql> REPAIR TABLE input use_frm;
+---------------+--------+----------+--------------------------------------+
| Table         | Op     | Msg_type | Msg_text                             |
+---------------+--------+----------+--------------------------------------+
| emoncms.input | repair | warning  | Number of rows changed from 0 to 114 |
| emoncms.input | repair | status   | OK                                   |
+---------------+--------+----------+--------------------------------------+
2 rows in set (0.02 sec)

I now have inputs listed on /emoncms/input/view, although they all have n/a value.
Thanks for all the guidance and help finding the issues so far @TrystanLea and @pb66

May be worth me restarting everything

Great reboot has fixed the input list view.

That leaves with nodered that won’t start: /usr/bin/env: node-red-pi: No such file or directory
I know that’s a minefield of fun with how best to (re)install nodejs or nodejs-legacy and which versions to run etc.

Also have some unmet dependencies, g++ : Depends: g++-4.9 and libgcc-4.9-dev : Depends: libatomic1 but these last days whenever I try to run sudo apt-get -f install everything just crashes, so ignoring that for now and hope doesn’t cause other issues.

Thank you all for your help, I’m slowly getting there.

Good to hear its partially working again.
Did you have an extensive node-red setup? or was it just stock emonSD examples?

I have a lot of stuff in nodered, boiler power/state, burner state, oil tank levels and heating.
But that shouldn’t be a problem, I’ve backed up node-red folder just in case.

It seems I’m blocked on that anyway due to the system unmet dependencies, sudo apt-get -f install seems to die at 75% and crash the pi, or at least ssh and all the services.
So that’s the main challenge now…

I ran the node red upgrade script and that seems to have fixed node red for now.

I’ll leave it running with nodeJS 4.8.

I’m all sorted for now thanks!

Great good to hear!

I experienced the same with some emonbase stations not starting again after a reboot and I’m quite sure the problem is the power source. After the failed reboots the red led of the Pi was flashing which indicates a problem with the power source, because IMHO this LED is directly connected to the power. So flashing is definitely not good. I run some emonTx with emonbase in remote locations so I replaced the delivered USB-power source with another one with more ampere which is working reliable for me so far.

Completely a power supply issue you are right, I’ve changed my power supply and PWR LED stays fixed, however the harm has already been done and the SD has gone awol.
Put a new SD card with fresh emonSD and it all runs smoothly, albeit a blank slate.

I’m doing my best to try and get all the data off it and onto a new card, but having great difficulty with partitions mounted on mac with osxfuse and ext4fuse, as soon as I start copying a decent amount of data the mount fails.

Then I’ll need a decent way of moving the data onto new SD card, looking at VirtualBox with a Ubuntu VM, but no time at the moment to fiddle with mounting USB (given up on mounting SD Card, seems not possible since Mac OS Mountain Lion).

So, at the moment emonhub is just providing live input values and no feeds, until I find the time to fix all the problems, or decide to start from scratch :frowning: