Mqtt_input service down: 'activating (auto-restart)' state

Hi guys,

Some help from your expertise :slight_smile:

In admin dashboard, mqtt_input service is in ‘activating (auto-restart)’ state

Checking from shell, i can see service status:

    pi@emonpi(rw):emoncms$ service mqtt_input status
    ● mqtt_input.service - Emoncms MQTT Input Script
       Loaded: loaded (/etc/systemd/system/mqtt_input.service; enabled)
       Active: activating (auto-restart) (Result: exit-code) since Sat 2019-11-23 16:59:36 CET; 7s ago
      Process: 27734 ExecStart=/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php (code=exited, status=255)
     Main PID: 27734 (code=exited, status=255)

And from logfile: /var/log/daemon.log part of logfile shows Fatal error with ‘Lib/EmonLogger.php’

    Nov 23 16:16:48 emonpi systemd[1]: Stopping Emoncms MQTT Input Script...
    Nov 23 16:16:48 emonpi systemd[1]: Starting Emoncms MQTT Input Script...
    Nov 23 16:16:48 emonpi systemd[1]: Started Emoncms MQTT Input Script.
    Nov 23 16:16:48 emonpi systemd[1]: Stopping LSB: feedwriter script daemon...
    Nov 23 16:16:48 emonpi feedwriter[6060]: Log is turned off
    Nov 23 16:16:48 emonpi php[6050]: PHP Warning:  require(Lib/EmonLogger.php): failed to open stream: No such file or directory in /var/www/emoncms/scripts/phpmqtt_input.php on line 47
    Nov 23 16:16:48 emonpi php[6050]: PHP Fatal error:  require(): Failed opening required 'Lib/EmonLogger.php' (include_path='.:/usr/share/php:/usr/share/pear') in /var/www/emoncms/scripts/phpmqtt_input.php on line 47
    Nov 23 16:16:48 emonpi systemd[1]: mqtt_input.service: main process exited, code=exited, status=255/n/a
    Nov 23 16:16:48 emonpi systemd[1]: Unit mqtt_input.service entered failed state.

But Library file exist:

    pi@emonpi(rw):emoncms$ ls -l /var/www/emoncms/Lib/EmonLogger.php
    -rw-r--r-- 1 pi pi 2371 Nov 21 17:37 /var/www/emoncms/Lib/EmonLogger.php

I have updated emoncms to last version with ‘stable’ branch:

    pi@emonpi(rw):emoncms$ git branch
      dev-mosquitto-php
      master
    * stable
      symlinked_modules

    pi@emonpi(rw):emoncms$ git pull
    Updating 25e8437..90f3915
    Fast-forward
     Modules/feed/feed_model.php | 10 ++++------
     default-settings.ini        |  6 +++---
     version.txt                 |  2 +-
     3 files changed, 8 insertions(+), 10 deletions(-)

     pi@emonpi(rw):emoncms$ git status
    On branch stable
    Your branch is up-to-date with 'origin/stable'.
    nothing to commit, working directory clean

Emonscms version is 10.1.13. After reboot, service is in the same state. No idea how solve it, any suggestions? Thanks in advance.

Hi Dan,

Just wondering how old the SD card is? Odd errors like this can sometimes be the first sign of a card failing.

I’d take a backup before doing anything else.

@TrystanLea any ideas?

Hy Borpin,

Really thanks for your reply. Checking my Daily backups running under emonpi (/home/pi/backup/emoncms-export.sh), i can see that files under /var/www/emoncms) are not backed!!

But i have an older backup (from 2019 March) and i have recovered ‘EmonLogger.php’ file.

After check repository status:

pi@emonpi(rw):emoncms$ git status
On branch stable
Your branch is up-to-date with 'origin/stable'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   Lib/EmonLogger.php

no changes added to commit (use "git add" and/or "git commit -a")
pi@emonpi(rw):emoncms$ git add Lib/EmonLogger.php
pi@emonpi(rw):emoncms$ git pull
Already up-to-date.
pi@emonpi(rw):emoncms$ git status
On branch stable
Your branch is up-to-date with 'origin/stable'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   Lib/EmonLogger.php

pi@emonpi(rw):emoncms$

And update repo, i have restarted all services, but mqtt_input service is in the same state, sniff, sniff :cry:

● mqtt_input.service - Emoncms MQTT Input Script
   Loaded: loaded (/etc/systemd/system/mqtt_input.service; enabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2019-11-26 17:50:13 CET; 24s ago
  Process: 16096 ExecStart=/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php (code=exited, status=255)
 Main PID: 16096 (code=exited, status=255)
   CGroup: /system.slice/mqtt_input.service

Nov 26 17:50:13 emonpi systemd[1]: mqtt_input.service: main process exited, code=exited, status=255/n/a
Nov 26 17:50:13 emonpi systemd[1]: Unit mqtt_input.service entered failed state.

Then it’s a good idea go to Factory Reset?

$ sudo su
$ /home/pi/emonpi/factoryreset

No they will not as they are system files not data/settings files.

Nooooooo

Do you know you have edited Lib/EmonLogger.php file?

If not then restore it by

git checkout -- Lib/EmonLogger.php

Can you run an update from the emoncms admin page? That will modify the installed service and that may be the issue as the service file has changed.

The method of updating you are using, does not update all the underlying modules so is not recommended.

You are running an older setup. My advice is to

  • get a new SD Card
  • flash a new EmonSD image
  • take a backup
  • install the new card
  • wait, wait, wait, make a cup of coffee, wait, wait, wait… (lots of updates etc)
  • import the backup

Hi Borpin,

Thanks for your advices. Some comments regarding your notes:

If not then restore it by

git checkout -- Lib/EmonLogger.php

After run checkout,

pi@emonpi(rw):emoncms$ git checkout -- Lib/EmonLogger.php
pi@emonpi(rw):emoncms$ git status
On branch stable
Your branch is up-to-date with 'origin/stable'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   Lib/EmonLogger.php

mqtt_input service seems in the same state.

● mqtt_input.service - Emoncms MQTT Input Script
   Loaded: loaded (/etc/systemd/system/mqtt_input.service; enabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2019-11-28 10:46:33 CET; 21s ago
  Process: 24079 ExecStart=/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php (code=exited, status=255)
 Main PID: 24079 (code=exited, status=255)
   CGroup: /system.slice/mqtt_input.service

Also i have tryed some workaround without success:

    pi@emonpi(rw):emoncms$ php -f /var/www/emoncms/scripts/phpmqtt_input.php
    PHP Warning:  fopen(/var/lock/emoncms_mqtt.lock): failed to open stream: Permission denied in /var/www/emoncms/scripts/phpmqtt_input.php on line 43
    PHP Warning:  flock() expects parameter 1 to be resource, boolean given in /var/www/emoncms/scripts/phpmqtt_input.php on line 44
    Already running

    pi@emonpi(rw):emoncms$ sudo systemctl stop mqtt_input.service

    pi@emonpi(rw):emoncms$ sudo rm -f /var/lock/emoncms_mqtt.lock

About your recomendations about update from the emoncms admin page, the problem is that not works and not perform any action… Pushing over ‘Full Update’ button, or sub buttons, a quick message is showed: “service-runner trigger sent”

image

And after a while message dissapear and nothing happens!

Then if no exist any alternative, i will procedure with your notes:

  • get a new SD Card
  • flash a new EmonSD image
  • take a backup
  • install the new card
  • wait, wait, wait, make a cup of coffee, wait, wait, wait… (lots of updates etc)
  • import the backup

The problem is in this raspberry pi, i have some other applications running (nagios, nagiosgraph, etc) and some Filesystem customizations. Is some complicated reinstall all stuff in a new SD card… :pensive:

Is this a recent problem? Has it worked previously?

Highly likely the service-runner service is not running either.

I decided long ago not to run multiple systems on one device/machine (real or virtual) for just this reason.

3 posts were split to a new topic: Mqtt_input service down: ‘activating (auto-restart)’