No Inputs and /var/log full within 1h of boot

Hardware emonPi
1

If I got to /emoncms/input/view I just get a loading icon in front of a blank page.
Feeds haven’t updated in 20 hours.

I noticed logs were really out of date yesterday, and a df -h showed why, /var/logs/ is 100% full with syslog and daemon.log both taking up around 22MB.

If I do a sudo reboot the Pi doesn’t always reboot all that well, and I need to pull the power, wait a bit then plug it back in again. From the start /var/log sits at 10%, but within the hour it’s at 100%.

MQTT is running, and I’ve restarted mqtt_input but I notice when doing status it shows:

Sep 25 12:04:44 emonpi systemd[1]: Started Emoncms MQTT Input Script.
Sep 25 12:04:45 emonpi mqtt_input[14725]: PHP Fatal error:  Call to a member function fetch_object() on boolean in /var/www/emoncms/Modules/input/input_model.php on line 377
Sep 25 12:04:45 emonpi mqtt_input[14725]: Fatal error: Call to a member function fetch_object() on boolean in /var/www/emoncms/Modules/input/input_model.php on line 377

I did run an emonPi and emonBase Update yesterday, when I go to update dabatase I get the following:
Notice: Trying to get property of non-object in /var/www/emoncms/Lib/dbschemasetup.php on line 41

Along with the following updates, which never apply:

Database schema
ALTER TABLE `input` ADD `id` int(11) NOT NULL;
ALTER TABLE `input` ADD `userid` int(11) NOT NULL;
ALTER TABLE `input` ADD `name` text NOT NULL;
ALTER TABLE `input` ADD `description` text NOT NULL DEFAULT '';
ALTER TABLE `input` ADD `nodeid` text NOT NULL;
ALTER TABLE `input` ADD `processList` text NOT NULL;
ALTER TABLE `input` ADD `time` int(10) NOT NULL;
ALTER TABLE `input` ADD `value` float NOT NULL;

I did notice in the emonhub logs it had difficulty connecting to MQTT at first, then it was fine but after publishing values, RFM2PI talks of unreliable content:

2017-09-25 07:24:53,145 DEBUG    MQTT       Publishing: emon/emonth6/temperature 18.1
2017-09-25 07:24:53,147 DEBUG    MQTT       Publishing: emon/emonth6/external temperature 0
2017-09-25 07:24:53,148 DEBUG    MQTT       Publishing: emon/emonth6/humidity 77.3
2017-09-25 07:24:53,150 DEBUG    MQTT       Publishing: emon/emonth6/battery 2.4
2017-09-25 07:24:53,151 DEBUG    MQTT       Publishing: emon/emonth6/pulsecount 1
2017-09-25 07:24:53,152 INFO     MQTT       Publishing: emon/emonth6/rssi -65
2017-09-25 07:24:53,154 INFO     MQTT       Publishing: emonhub/rx/24/values 18.1,0,77.3,2.4,1
2017-09-25 07:24:53,156 INFO     MQTT       Publishing: emonhub/rx/24/rssi -65
2017-09-25 07:24:54,294 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 2 219 97 245 76 219 132 105 195 181 129 60 197 184 217 61 7 171 79 98 159 (-96)
2017-09-25 07:24:54,502 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 0 174 23 155 220 196 161 203 36 185 199 55 208 89 200 132 131 14 122 56 230 (-87)
2017-09-25 07:24:54,911 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 26 68 251 129 189 4 140 96 189 39 149 185 218 134 114 118 184 245 214 255 81 (-97)
2017-09-25 07:24:55,425 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 19 71 67 121 153 22 140 3 12 212 91 250 240 200 191 68 27 227 150 241 24 (-96)
2017-09-25 07:24:56,246 DEBUG    RFM2Pi     11 NEW FRAME : OK 8 0 0 0 0 0 0 0 0 49 103 184 11 184 11 184 11 184 11 184 11 184 11 205 229 1 0 (-66)
2017-09-25 07:24:56,249 DEBUG    RFM2Pi     11 Timestamp : 1506317096.25
2017-09-25 07:24:56,250 DEBUG    RFM2Pi     11 From Node : 8
2017-09-25 07:24:56,253 DEBUG    RFM2Pi     11    Values : [0, 0, 0, 0, 264.17, 300, 300, 300, 300, 300, 300, 124365]
2017-09-25 07:24:56,257 DEBUG    RFM2Pi     11      RSSI : -66
2017-09-25 07:24:56,258 DEBUG    RFM2Pi     11 Sent to channel(start)' : ToEmonCMS
2017-09-25 07:24:56,259 DEBUG    RFM2Pi     11 Sent to channel(end)' : ToEmonCMS

I’m at a complete loss on how to move forward, or what the underlying issue(s) might be, any help would be gratefully appreciated!

3

Actually Bill that isn’t shouldn’t be possible since emonhub caps it’s own logfile size to 5Mb and rotates a max of 2 files so emonhub can only take up a maximum of 10Mb. (Or at least that’s how it should be working, there’s been a lot changed in the emonpi variant)

@waako - Can you try using

sudo du -h /var/log

to see what is hogging the diskspace?

4

OK, thanks for the heads-up. Post deleted.

5

Thanks Paul,
I did notice that emonhub logs were 10Mb, it was syslog and daemon.log that appeared largest around 22Mb.

I didn’t know about du -h however, I’ve done a power cycle, so not full “yet”.
Gives me:

4.0K	/var/log/supervisor
8.0K	/var/log/mosquitto
0	/var/log/logrotate
0	/var/log/openhab
4.0K	/var/log/mysql
156K	/var/log/apache2
8.0K	/var/log/redis
44K	/var/log/emonhub
4.0K	/var/log/emonpilcd
17M	/var/log

/var/log is now at 20M in the few minutes it’s taken me to write this.
Doing sudo du -h /var/log/* gives me:

196K	/var/log/apache2
16K	/var/log/auth.log
28K	/var/log/boot.log
0	/var/log/btmp
9.8M	/var/log/daemon.log
4.0K	/var/log/dataplicity.log
4.0K	/var/log/emoncms.log
56K	/var/log/emonhub
4.0K	/var/log/emonpilcd
240K	/var/log/kern.log
0	/var/log/logrotate
240K	/var/log/messages
8.0K	/var/log/mosquitto
0	/var/log/mqtt_input.log
4.0K	/var/log/mysql
0	/var/log/mysql.log
0	/var/log/ntp_update.log
0	/var/log/openhab
8.0K	/var/log/redis
4.0K	/var/log/service-runner.log
4.0K	/var/log/supervisor
10M	/var/log/syslog
4.0K	/var/log/ufw.log
4.0K	/var/log/wtmp

emonHub has only been running 10-15 minutes at most.

Inputs view still empty with spinning loader

6

Thanks for pointing that out though Bill, have changed level anyway, can’t do any harm. WIll try to remember when I need to debug.

7

You’re not the first to report this, but I’m afraid I’m not sure what the answer is. but now you know where the log entries are piling up, syslog and daemon.log, is there any clues when you check those log files?

8

Actually, something I hadn’t noticed earlier is the tail on both logs is all node-red.
I’ve stopped node-red and that seems to have stopped them filling up so rapidly.

Trying to run node-red upgrade script, see if that helps, then best see what settings are for logs there if I can work it out.

Should probably update system too, one step at a time!

9

@waako trying to trace the error line numbers you posted above, the first in input_model.php corresponds to the input clean method, which seems strange to have been triggered from a mqtt_input reset as there are no calls to this method from the script. Are you running the emoncms stable branch on a standard emonSD setup?

For the inputs spinning issue: can you post the result of emoncms/input/list.json?

10

Thanks @TrystanLea ,

Yes, I am running stable branch with emonSD setup, and emoncms/input/list.json gives me:
Fatal error: Call to a member function fetch_object() on boolean in /var/www/emoncms/Modules/input/input_model.php on line 377

I notice this error came up in Emoncms Apps: Call to a member function fetch_object() on a non-object? But I’ve tried running database updates, but they don’t stick and have this issue on the page:
Notice: Trying to get property of non-object in /var/www/emoncms/Lib/dbschemasetup.php on line 41

The logs were indeed being filled by Node Red’s telegram node, removed it and logs issue fixed, I’d already moved to using chatbot anyway.
I’m guessing it had something to do with node-red-contrib-telegrambot wanting NodeJS version 5.x and I’m on 4.8. Although unsure why it suddenly became a problem.

11

Thanks @waako, sorry line 377 does make sense, its the line where emoncms loads the inputs from mysql into redis: https://github.com/emoncms/emoncms/blob/stable/Modules/input/input_model.php#L377

It sounds like something has gone a bit amiss in the inputs table in the emoncms mysql database, perhaps the table has crashed somehow, as it appears that emoncms is trying to recreate all fields!

Would you be able to login to mysql on the pi locally and check the inputs table? try:

mysql -u emoncms -p emoncms

password: emonpiemoncmsmysql2016

SELECT * FROM inputs;

Then you could try (I’ve only ever had to do this once 5+ years ago so not sure if it will sort it):

REPAIR TABLE inputs;
12

So, this would explain the issue of inputs not loading :confused:

mysql> SELECT * FROM inputs;
ERROR 1146 (42S02): Table 'emoncms.inputs' doesn't exist
13

Try “input” rather than “inputs”

14

Actually there’s an input table however (sorry trying to respond and cook dinner is probably not the most sensible for either tasks):

mysql> show tables;
+-------------------+
| Tables_in_emoncms |
+-------------------+
| app_config        |
| dashboard         |
| feeds             |
| graph             |
| input             |
| multigraph        |
| rememberme        |
| schedule          |
| setup             |
| users             |
+-------------------+
10 rows in set (0.00 sec)

However, that gives me:

mysql> SELECT * FROM input;
ERROR 130 (HY000): Incorrect file format 'input'
15

Tried to repair anyway:

mysql> REPAIR TABLE input;
+---------------+--------+----------+-------------------------------+
| Table         | Op     | Msg_type | Msg_text                      |
+---------------+--------+----------+-------------------------------+
| emoncms.input | repair | Error    | Incorrect file format 'input' |
| emoncms.input | repair | error    | Corrupt                       |
+---------------+--------+----------+-------------------------------+
2 rows in set (0.00 sec)
16

I ran mysqlcheck and it seems to only be the input table which has problems.

using use_frm repaired the table

mysql> REPAIR TABLE input use_frm;
+---------------+--------+----------+--------------------------------------+
| Table         | Op     | Msg_type | Msg_text                             |
+---------------+--------+----------+--------------------------------------+
| emoncms.input | repair | warning  | Number of rows changed from 0 to 114 |
| emoncms.input | repair | status   | OK                                   |
+---------------+--------+----------+--------------------------------------+
2 rows in set (0.02 sec)

I now have inputs listed on /emoncms/input/view, although they all have n/a value.
Thanks for all the guidance and help finding the issues so far @TrystanLea and @pb66

May be worth me restarting everything

17

Great reboot has fixed the input list view.

That leaves with nodered that won’t start: /usr/bin/env: node-red-pi: No such file or directory
I know that’s a minefield of fun with how best to (re)install nodejs or nodejs-legacy and which versions to run etc.

Also have some unmet dependencies, g++ : Depends: g++-4.9 and libgcc-4.9-dev : Depends: libatomic1 but these last days whenever I try to run sudo apt-get -f install everything just crashes, so ignoring that for now and hope doesn’t cause other issues.

Thank you all for your help, I’m slowly getting there.

18

Good to hear its partially working again.
Did you have an extensive node-red setup? or was it just stock emonSD examples?

19

I have a lot of stuff in nodered, boiler power/state, burner state, oil tank levels and heating.
But that shouldn’t be a problem, I’ve backed up node-red folder just in case.

It seems I’m blocked on that anyway due to the system unmet dependencies, sudo apt-get -f install seems to die at 75% and crash the pi, or at least ssh and all the services.
So that’s the main challenge now…

20

I ran the node red upgrade script and that seems to have fixed node red for now.

I’ll leave it running with nodeJS 4.8.

I’m all sorted for now thanks!

21

Great good to hear!