Input's updates, the feeds do not

this is a SELFHOSTED install of emoncms

so the pi has been running for almost a week now and everything seems fine

now suddenly the feeds stopped updating in the middle of the night, inputs still works fine, but the feeds do not get updated

if keeping in mind that things worked last night and without touching anything feed just stopped updating, what would the first troubleshoot step be?

just checked date and time

Sun 5 May 09:31:02 UTC 2019

i have a pi sending data to emoncms with this script

https://github.com/boelle/Rpi-Heating-Control/blob/master/heatcontrol.py

and the time on that one is: Sun 5 May 10:32:03 BST 2019

Server Information
Services
emonhub Active Running
emoncms_mqtt Active Running
feedwriter Active Running - sleep 60s
service-runner Active Running
emonPiLCD Active Exited
redis-server Active Running
mosquitto Active Running
Emoncms Version low-write 9.9.8
Modules Administration : App v1.2.1 : Backup v1.2.0 : EmonHub Config v1.1.0 : Dashboard v1.3.3 : Device v1.2.1 : EventProcesses : Feed : Graph v1.2.3 : Input : Postprocess v1.0.0 : CoreProcess : Schedule : Network Setup v1.0.0 : sync : Time : User : Visualisation : WiFi v1.3.1
Git URL: GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data : Branch: * stable : Describe: v9.5.1-1775-gd0db7a5
Server OS Linux 4.9.35+
Host emonpi : emonpi : (192.168.1.180)
Date 2019-05-05 09:35:20 UTC
Uptime 09:35:21 up 30 min, 1 user, load average: 0.43, 0.40, 0.57
HTTP Server Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80
MySQL Version 5.5.62-0+deb8u1
Host 127.0.0.1 (127.0.0.1)
Date 2019-05-05 09:35:19 (UTC 00:00‌​)
Stats Uptime: 1780 Threads: 3 Questions: 2249 Slow queries: 0 Opens: 61 Flush tables: 1 Open tables: 54 Queries per second avg: 1.263
Redis Version 2.8.17
Host localhost:6379 (127.0.0.1)
Size
Uptime 0 days
MQTT Server Version Mosquitto 1.5.3
Host localhost:1883 (127.0.0.1)
Pi Model Raspberry Pi Model B+ Rev 1.0 - 512 MB (Sony UK)
SoC Broadcom BCM2835
Serial num. B7977A6B
Temperature CPU: 35.78°C - GPU: 35.8’C
Release emonSD-26Oct17
File-system Current: read-only - Set root file-system temporarily to read-write, (default read-only)
Memory RAM Used: 49.42% Total: 481.69 MB Used: 238.07 MB Free: 243.63 MB
Disk Mount Stats
/ Used: 63.25% Total: 3.33 GB Used: 2.11 GB Free: 1.06 GB
/boot Used: 36.36% Total: 59.95 MB Used: 21.8 MB Free: 38.15 MB
/home/pi/data Used: 4.13% Total: 968.33 MB Used: 40.02 MB Free: 878.33 MB
PHP Version 5.6.38-0+deb8u1 (Zend Version 2.6.0)
Modules apache2handler : bcmath : bz2 : calendar : Core v5.6.38-0+deb8u1 : ctype : curl : date v5.6.38-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 $Id: cad29b729548e4206f0697710cc9e177f26fdff3 $ : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.38-0+deb8u1 : redis v2.2.7 : Reflection v$Id: 5f15287237d5f78d75b19c26915aa7bd83dee8b8 $ : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.38-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0
Client Information
HTTP Browser Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36
Screen Resolution 1280 x 1024
Window Size 1263 x 881

LOG

LAST ENTRIES ON THE LOG FILE
2019-05-05 09:06:32.420|ERROR|feedwriter.php|Starting feedwriter script
2019-05-05 09:06:34.632|WARN|emoncms_mqtt.php|Not connected, retrying connection
2019-05-05 09:06:34.855|WARN|emoncms_mqtt.php|Connecting to MQTT server: Connection Accepted.: code: 0

update log

Starting update via service-runner-update.sh (v2.0) >
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
- emonSD version: emonSD-26Oct17
emonSD base image check passed...continue update
git pull /home/pi/emonpi
* master
  wifiap
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	1

nothing added to commit but untracked files present (use "git add" to track)
Already up-to-date.
-------------------------------------------------------------
Main Update Script
-------------------------------------------------------------
Date: Mon 29 Apr 19:54:23 UTC 2019
EUID: 1000
usrdir: /home/pi
type: all
firmware: rfm69pi
Checking environment:
- pi user found

EUID: 1000
Hardware detected: rfm2pi
Stopping emonPiLCD service
Display update message on LCD
I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
-------------------------------------------------------------
Hit http://ppa.launchpad.net xenial InRelease
Hit http://mirrordirector.raspbian.org jessie InRelease
Hit http://repo.mosquitto.org jessie InRelease
Ign http://dl.bintray.com stable InRelease
Hit http://archive.raspberrypi.org jessie InRelease
Get:1 http://dl.bintray.com stable Release.gpg [821 B]
Get:2 http://dl.bintray.com stable Release [6,051 B]
Hit http://ppa.launchpad.net xenial/main Sources
Hit http://mirrordirector.raspbian.org jessie/main armhf Packages
Hit http://ppa.launchpad.net xenial/main armhf Packages
Hit http://ppa.launchpad.net xenial/main Translation-en
Hit http://mirrordirector.raspbian.org jessie/contrib armhf Packages
Hit http://mirrordirector.raspbian.org jessie/non-free armhf Packages
Hit http://mirrordirector.raspbian.org jessie/rpi armhf Packages
Hit http://repo.mosquitto.org jessie/main armhf Packages
Ign http://repo.mosquitto.org jessie/main Translation-en_GB
Ign http://repo.mosquitto.org jessie/main Translation-en
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en_GB
Hit http://archive.raspberrypi.org jessie/main armhf Packages
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en
Hit http://archive.raspberrypi.org jessie/ui armhf Packages
Ign http://mirrordirector.raspbian.org jessie/main Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/main Translation-en
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en
Ign http://archive.raspberrypi.org jessie/main Translation-en_GB
Ign http://archive.raspberrypi.org jessie/main Translation-en
Ign http://archive.raspberrypi.org jessie/ui Translation-en_GB
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Get:3 http://dl.bintray.com stable/main armhf Packages [77.8 kB]
Ign http://dl.bintray.com stable/main Translation-en_GB
Ign http://dl.bintray.com stable/main Translation-en
Fetched 84.6 kB in 1min 1s (1,369 B/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
rng-tools is already the newest version.
python-gpiozero is already the newest version.
python-rpi.gpio is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 99 not upgraded.
Requirement already up-to-date: paho-mqtt in /usr/local/lib/python2.7/dist-packages (1.4.0)
You are using pip version 18.1, however version 19.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
-------------------------------------------------------------
git pull /home/pi/RFM2Pi
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.

git pull /home/pi/usefulscripts
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.

git pull /home/pi/huawei-hilink-status
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.

git pull /home/pi/oem_openHab
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.

git pull /home/pi/oem_node-red
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.

-------------------------------------------------------------
RFM69Pi Firmware Update
-------------------------------------------------------------
Getting latest RFM69Pi release info from github
Latest RFM69Pi firmware: V1.4.0
downloading latest RFM69Pi firmware from github releases:
https://github.com/openenergymonitor/RFM2Pi/releases/download/1.4.0/firmware.hex
Saving to /home/pi/data/firmware/rfm69pi-1.4.0.hex

Flashing RFM69Pi with V 1.4.0


avrdude-original: Version 6.1, compiled on Jul  7 2015 at 10:29:47
                  Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
                  Copyright (c) 2007-2014 Joerg Wunsch

                  System wide configuration file is "/etc/avrdude.conf"
                  User configuration file is "/root/.avrduderc"
                  User configuration file does not exist or is not a regular file, skipping

                  Using Port                    : /dev/ttyAMA0
                  Using Programmer              : arduino
                  Overriding Baud Rate          : 38400
avrdude-original: Using autoreset DTR on GPIO Pin 7
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 1 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 2 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 3 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 4 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 5 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 6 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 7 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 8 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 9 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 10 of 10: not in sync: resp=0x00

avrdude-original done.  Thank you.

strace: |autoreset: Broken pipe
Flashing RFM69Pi with V 1.4.0  done
Start emonhub update script:
-------------------------------------------------------------
emonHub update started
-------------------------------------------------------------
- usr directory: /home/pi
git pull /home/pi/emonhub
  K0den-wibeee_interface
* emon-pi
  hmm01i-syslogging
On branch emon-pi
Your branch is up-to-date with 'origin/emon-pi'.
nothing to commit, working directory clean
Already up-to-date.
emonhub.service already installed

Running emonhub automatic node addition script
[[5]]
Node 5 already present
[[6]]
Node 6 already present
[[7]]
Node 7 already present
[[8]]
Node 8 already present
[[9]]
Node 9 already present
[[10]]
Node 10 already present
[[11]]
Node 11 already present
[[12]]
Node 12 already present
[[13]]
Node 13 already present
[[14]]
Node 14 already present
[[19]]
Node 19 already present
[[20]]
Node 20 already present
[[21]]
Node 21 already present
[[22]]
Node 22 already present
[[23]]
Node 23 already present
[[24]]
Node 24 already present
[[25]]
Node 25 already present
[[26]]
Node 26 already present

Start emoncms update:

-------------------------------------------------------------
Emoncms update started
Emoncms update script V1.3 (26th March 2019)
-------------------------------------------------------------
- usr directory: /home/pi
- emoncms directory: /var/www/emoncms

current settings.php md5: 468b110786bf0ab60a856fa3c0036f0f
Default settings.php md5: 468b110786bf0ab60a856fa3c0036f0f
settings.php has NOT been user modifed

Checking status of /var/www/emoncms git repository
- git branch: stable
- no local changes
- running: git pull origin stable

From https://github.com/emoncms/emoncms
 * branch            stable     -> FETCH_HEAD
Already up-to-date.

NEW default settings.php md5: 468b110786bf0ab60a856fa3c0036f0f
settings.php not updated

-------------------------------------------------------------
Updating Emoncms Modules
-------------------------------------------------------------

------------------------------------------
Updating /var/www/emoncms/Modules/app module
------------------------------------------
- git branch: stable
- git tags: 1.2.0-21-g8edcc30
- no local changes
- running: git pull origin stable

From https://github.com/emoncms/app
 * branch            stable     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Updating /var/www/emoncms/Modules/config module
------------------------------------------
- git branch: stable
- git tags: 1.0.0-8-g5e96633
- no local changes
- running: git pull origin stable

From https://github.com/emoncms/config
 * branch            stable     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Updating /var/www/emoncms/Modules/dashboard module
------------------------------------------
- git branch: stable
- git tags: 1.3.1-36-g8be171b
- no local changes
- running: git pull origin stable

From https://github.com/emoncms/dashboard
 * branch            stable     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Updating /var/www/emoncms/Modules/device module
------------------------------------------
- git branch: master
- git tags: 1.1.1-35-gda7fa4e
- no local changes
- running: git pull origin master

From https://github.com/emoncms/device
 * branch            master     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Updating /var/www/emoncms/Modules/graph module
------------------------------------------
- git branch: stable
- git tags: 1.2.1-66-g91d4db5
- no local changes
- running: git pull origin stable

From https://github.com/emoncms/graph
 * branch            stable     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Updating /var/www/emoncms/Modules/wifi module
------------------------------------------
- git branch: stable
- git tags: 1.2.0-33-g2f37618
- no local changes
- running: git pull origin stable

From https://github.com/emoncms/wifi
 * branch            stable     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Updating postprocess module
------------------------------------------
- git branch: emonpi
fatal: No names found, cannot describe anything.
- git tags: 
- no local changes
- running: git pull origin emonpi

From https://github.com/emoncms/postprocess
 * branch            emonpi     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Updating sync module
------------------------------------------
- git branch: master
- git tags: 1.0.0-7-gc391cd2
- no local changes
- running: git pull origin master

From https://github.com/emoncms/sync
 * branch            master     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Updating backup module
------------------------------------------
- git branch: stable
- git tags: 1.1.5-37-gf55d616
- no local changes
- running: git pull origin stable

From https://github.com/emoncms/backup
 * branch            stable     -> FETCH_HEAD
Already up-to-date.

------------------------------------------
Auto Installation of Emoncms Modules
------------------------------------------
- graph module already installed
- device module already installed
- postprocess module already installed
- sync module already installed
- backup module already installed

www-data : www-data video systemd-journal

------------------------------------------
SERVICES
------------------------------------------
emoncms_mqtt.service already installed
feedwriter.service already installed
service-runner.service already installed
------------------------------------------
Update Emoncms database
[]

Restarting Services...
- sudo systemctl restart feedwriter.service
--- ActiveState=active ---
- sudo systemctl restart emoncms_mqtt.service
--- ActiveState=active ---
- sudo systemctl restart emonhub.service
--- ActiveState=active ---
- sudo /etc/init.d/openhab restart
Restarting openhab (via systemctl): openhab.service.
● openhab.service - Starts and stops the openHAB Home Automation Bus
   Loaded: loaded (/usr/lib/systemd/system/openhab.service; enabled)
   Active: active (running) since Mon 2019-04-29 20:00:14 UTC; 1s ago
     Docs: http://www.openhab.org
  Process: 18964 ExecStartPre=/usr/share/openhab/bin/setpermissions.sh (code=exited, status=0/SUCCESS)
 Main PID: 19017 (openhab.sh)
   CGroup: /system.slice/openhab.service
           ├─19017 /bin/sh /usr/share/openhab/bin/openhab.sh -o
           └─19045 /usr/bin/java -Dlogback.configurationFile=/etc/openhab/logback.xml -Dosgi.clean=true -Declipse.ignoreApp=true -Dosgi.noShutdown=true -Djetty.port=8080 -Dopenhab.configfile=/etc/openhab/configurations/openhab.cfg -Dopenhab.configdir=/etc/openhab/configurations -Dopenhab.logdir=/var/log/openhab -Dsmarthome.userdata=/var/lib/openhab -Djetty.home=/usr/share/openhab -Djetty.port.ssl=8443 -Djetty.config=/etc/openhab/jetty -Djetty.logs=/var/log/openhab -Djetty.rundir=/usr/share/openhab -Dfelix.fileinstall.dir=/usr/share/openhab/addons -Dfelix.fileinstall.filter=.*\.jar -Djava.library.path=/usr/share/openhab/lib -Djava.security.auth.login.config=/etc/openhab/login.conf -Dorg.quartz.properties=/etc/openhab/quartz.properties -Dequinox.ds.block_timeout=240000 -Dequinox.scr.waitTimeOnBlock=60000 -Dfelix.fileinstall.active.level=4 -Djava.awt.headless=true -jar /usr/share/openhab/server/plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar -configuration /var/lib/openhab/workspace -data /var/lib/openhab/workspace -console 5555

Apr 29 20:00:12 emonpi setpermissions.sh[18964]: setting owner openhab:openhab recursively for /var/lib/openhab
Apr 29 20:00:12 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab recursively for /var/log/openhab
Apr 29 20:00:12 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab recursively for /etc/openhab/configurations
Apr 29 20:00:14 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab for /usr/share/openhab/webapps/static
Apr 29 20:00:14 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab for /etc/openhab/jetty/etc/keystore
Apr 29 20:00:14 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab for /etc/openhab/configurations/users.cfg
Apr 29 20:00:14 emonpi systemd[1]: Started Starts and stops the openHAB Home Automation Bus.
Installing emoncms logrotate...

Now setting up Logrotate...
Backing up old logrotate configuration...
Linked to new logrotate configuration...
Backing up old logrotate cron job...
Linked to new logrotate cron job...

Completed

setup logrotate state & logfile in /var/log/logrotate
mkdir: cannot create directory ‘/var/log/logrotate’: File exists

Completed

Running logrotate...

set log rotate config owner to root
chown: changing ownership of ‘/etc/logrotate.conf’: Read-only file system

------------------------------------------
Emoncms update script complete
------------------------------------------


Starting emonPi LCD service..

Filesystem is locked - Read Only access
type ' rpi-rw ' to unlock

-------------------------------------------------------------
emonPi update done: Mon 29 Apr 20:00:44 UTC 2019
-------------------------------------------------------------
restarting service-runner

timezone in my account (inside emoncms) and the one in raspi config match up

and this is the low write sd card image if i forgot to mention that

Almost certainly /log/var filling up. Daft suggestion as it is self hosted… :man_facepalming:

and how do i confirm if that is the case?

i thought it cleared things like that on it’s own

that one suggest that it has been fixed

and then again my input works fine and that is not what the link is about

so i’m still puzzled,

@TrystanLea @glyn.hudson if something is filling up should the user not get a warning and simple instructions on how to fix it?

df -h

So it should if configured correctly. I have been saying this is a problem for ages with little traction…

The fix, as it stands, doesn’t work IMNSHO.

I believe this fix does on an OCT18 SD image (and provides some additional log retention).

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.4G  2.2G  1.1G  67% /
devtmpfs        237M     0  237M   0% /dev
tmpfs           241M     0  241M   0% /dev/shm
tmpfs           241M  4.5M  237M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           241M     0  241M   0% /sys/fs/cgroup
tmpfs            40M   28K   40M   1% /var/lib/openhab
tmpfs           1.0M  4.0K 1020K   1% /var/lib/dhcpcd5
tmpfs           1.0M     0  1.0M   0% /var/lib/dhcp
tmpfs            50M  520K   50M   2% /var/log
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            30M   40K   30M   1% /tmp
/dev/mmcblk0p1   60M   22M   39M  37% /boot
/dev/mmcblk0p3  969M   41M  879M   5% /home/pi/data

does not look full to me, or am i wrong?

Nope you are correct.

I see you are posting by API, is there anything in the log in the admin screen?

You can change the log level in settings.php to INFO so you will see an entry each time you post to emoncms via the API.

What do you get as your responses to the GET?

i will have to check, put i assume the GET is ok since INPUTS are updated

so if i do the copy log to clipboard i only get this

LAST ENTRIES ON THE LOG FILE
2019-05-05 10:35:41.007|ERROR|feedwriter.php|Starting feedwriter script
2019-05-05 10:35:43.240|WARN|emoncms_mqtt.php|Not connected, retrying connection
2019-05-05 10:35:43.450|WARN|emoncms_mqtt.php|Connecting to MQTT server: Connection Accepted.: code: 0

ohhhh
ERROR

missed that

Yes good point. Change the logging level and see if anything additional that will help.

Can you post output from…

systemctl status feedwriter.service

It isn’t actually an error. The log should be as INFO for that message.

pi@emonpi(ro):~$ systemctl status feedwriter.service
● feedwriter.service - Emoncms feedwriter script
Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; disabled)
Active: active (running) since Sun 2019-05-05 10:35:33 UTC; 4min 11s ago
Docs: emoncms/install-service-feedwriter.md at master · emoncms/emoncms · GitHub
Main PID: 2262 (php)
CGroup: /system.slice/feedwriter.service
└─2262 /usr/bin/php /var/www/emoncms/scripts/feedwriter.php
pi@emonpi(ro):~$

That looks OK,

On the admin page, under the services

image

How many points pending write?

0

feedwriter Active Running - sleep 60s 0 feed points pending write