this is a SELFHOSTED install of emoncms
so the pi has been running for almost a week now and everything seems fine
now suddenly the feeds stopped updating in the middle of the night, inputs still works fine, but the feeds do not get updated
if keeping in mind that things worked last night and without touching anything feed just stopped updating, what would the first troubleshoot step be?
just checked date and time
Sun 5 May 09:31:02 UTC 2019
i have a pi sending data to emoncms with this script
https://github.com/boelle/Rpi-Heating-Control/blob/master/heatcontrol.py
and the time on that one is: Sun 5 May 10:32:03 BST 2019
| Services | ||
| emonhub | Active Running | |
| emoncms_mqtt | Active Running | |
| feedwriter | Active Running - sleep 60s | |
| service-runner | Active Running | |
| emonPiLCD | Active Exited | |
| redis-server | Active Running | |
| mosquitto | Active Running | |
| Emoncms | Version | low-write 9.9.8 |
| Modules | Administration : App v1.2.1 : Backup v1.2.0 : EmonHub Config v1.1.0 : Dashboard v1.3.3 : Device v1.2.1 : EventProcesses : Feed : Graph v1.2.3 : Input : Postprocess v1.0.0 : CoreProcess : Schedule : Network Setup v1.0.0 : sync : Time : User : Visualisation : WiFi v1.3.1 | |
| Git | URL: GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data : Branch: * stable : Describe: v9.5.1-1775-gd0db7a5 | |
| Server | OS | Linux 4.9.35+ |
| Host | emonpi : emonpi : (192.168.1.180) | |
| Date | 2019-05-05 09:35:20 UTC | |
| Uptime | 09:35:21 up 30 min, 1 user, load average: 0.43, 0.40, 0.57 | |
| HTTP | Server | Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80 |
| MySQL | Version | 5.5.62-0+deb8u1 |
| Host | 127.0.0.1 (127.0.0.1) | |
| Date | 2019-05-05 09:35:19 (UTC 00:00) | |
| Stats | Uptime: 1780 Threads: 3 Questions: 2249 Slow queries: 0 Opens: 61 Flush tables: 1 Open tables: 54 Queries per second avg: 1.263 | |
| Redis | Version | 2.8.17 |
| Host | localhost:6379 (127.0.0.1) | |
| Size | ||
| Uptime | 0 days | |
| MQTT Server | Version | Mosquitto 1.5.3 |
| Host | localhost:1883 (127.0.0.1) | |
| Pi | Model | Raspberry Pi Model B+ Rev 1.0 - 512 MB (Sony UK) |
| SoC | Broadcom BCM2835 | |
| Serial num. | B7977A6B | |
| Temperature | CPU: 35.78°C - GPU: 35.8’C | |
| Release | emonSD-26Oct17 | |
| File-system | Current: read-only - Set root file-system temporarily to read-write, (default read-only) | |
| Memory | RAM | Used: 49.42% Total: 481.69 MB Used: 238.07 MB Free: 243.63 MB |
| Disk | Mount | Stats |
| / | Used: 63.25% Total: 3.33 GB Used: 2.11 GB Free: 1.06 GB | |
| /boot | Used: 36.36% Total: 59.95 MB Used: 21.8 MB Free: 38.15 MB | |
| /home/pi/data | Used: 4.13% Total: 968.33 MB Used: 40.02 MB Free: 878.33 MB | |
| PHP | Version | 5.6.38-0+deb8u1 (Zend Version 2.6.0) |
| Modules | apache2handler : bcmath : bz2 : calendar : Core v5.6.38-0+deb8u1 : ctype : curl : date v5.6.38-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 $Id: cad29b729548e4206f0697710cc9e177f26fdff3 $ : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.38-0+deb8u1 : redis v2.2.7 : Reflection v$Id: 5f15287237d5f78d75b19c26915aa7bd83dee8b8 $ : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.38-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 |
| HTTP | Browser | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36 |
| Screen | Resolution | 1280 x 1024 |
| Window | Size | 1263 x 881 |
LOG
update log
Starting update via service-runner-update.sh (v2.0) >
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
- emonSD version: emonSD-26Oct17
emonSD base image check passed...continue update
git pull /home/pi/emonpi
* master
wifiap
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
1
nothing added to commit but untracked files present (use "git add" to track)
Already up-to-date.
-------------------------------------------------------------
Main Update Script
-------------------------------------------------------------
Date: Mon 29 Apr 19:54:23 UTC 2019
EUID: 1000
usrdir: /home/pi
type: all
firmware: rfm69pi
Checking environment:
- pi user found
EUID: 1000
Hardware detected: rfm2pi
Stopping emonPiLCD service
Display update message on LCD
I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
-------------------------------------------------------------
Hit http://ppa.launchpad.net xenial InRelease
Hit http://mirrordirector.raspbian.org jessie InRelease
Hit http://repo.mosquitto.org jessie InRelease
Ign http://dl.bintray.com stable InRelease
Hit http://archive.raspberrypi.org jessie InRelease
Get:1 http://dl.bintray.com stable Release.gpg [821 B]
Get:2 http://dl.bintray.com stable Release [6,051 B]
Hit http://ppa.launchpad.net xenial/main Sources
Hit http://mirrordirector.raspbian.org jessie/main armhf Packages
Hit http://ppa.launchpad.net xenial/main armhf Packages
Hit http://ppa.launchpad.net xenial/main Translation-en
Hit http://mirrordirector.raspbian.org jessie/contrib armhf Packages
Hit http://mirrordirector.raspbian.org jessie/non-free armhf Packages
Hit http://mirrordirector.raspbian.org jessie/rpi armhf Packages
Hit http://repo.mosquitto.org jessie/main armhf Packages
Ign http://repo.mosquitto.org jessie/main Translation-en_GB
Ign http://repo.mosquitto.org jessie/main Translation-en
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en_GB
Hit http://archive.raspberrypi.org jessie/main armhf Packages
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en
Hit http://archive.raspberrypi.org jessie/ui armhf Packages
Ign http://mirrordirector.raspbian.org jessie/main Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/main Translation-en
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en
Ign http://archive.raspberrypi.org jessie/main Translation-en_GB
Ign http://archive.raspberrypi.org jessie/main Translation-en
Ign http://archive.raspberrypi.org jessie/ui Translation-en_GB
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Get:3 http://dl.bintray.com stable/main armhf Packages [77.8 kB]
Ign http://dl.bintray.com stable/main Translation-en_GB
Ign http://dl.bintray.com stable/main Translation-en
Fetched 84.6 kB in 1min 1s (1,369 B/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
rng-tools is already the newest version.
python-gpiozero is already the newest version.
python-rpi.gpio is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 99 not upgraded.
Requirement already up-to-date: paho-mqtt in /usr/local/lib/python2.7/dist-packages (1.4.0)
You are using pip version 18.1, however version 19.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
-------------------------------------------------------------
git pull /home/pi/RFM2Pi
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/usefulscripts
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/huawei-hilink-status
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/oem_openHab
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/oem_node-red
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
-------------------------------------------------------------
RFM69Pi Firmware Update
-------------------------------------------------------------
Getting latest RFM69Pi release info from github
Latest RFM69Pi firmware: V1.4.0
downloading latest RFM69Pi firmware from github releases:
https://github.com/openenergymonitor/RFM2Pi/releases/download/1.4.0/firmware.hex
Saving to /home/pi/data/firmware/rfm69pi-1.4.0.hex
Flashing RFM69Pi with V 1.4.0
avrdude-original: Version 6.1, compiled on Jul 7 2015 at 10:29:47
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch
System wide configuration file is "/etc/avrdude.conf"
User configuration file is "/root/.avrduderc"
User configuration file does not exist or is not a regular file, skipping
Using Port : /dev/ttyAMA0
Using Programmer : arduino
Overriding Baud Rate : 38400
avrdude-original: Using autoreset DTR on GPIO Pin 7
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 1 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 2 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 3 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 4 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 5 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 6 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 7 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 8 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 9 of 10: not in sync: resp=0x00
avrdude-original: stk500_recv(): programmer is not responding
avrdude-original: stk500_getsync() attempt 10 of 10: not in sync: resp=0x00
avrdude-original done. Thank you.
strace: |autoreset: Broken pipe
Flashing RFM69Pi with V 1.4.0 done
Start emonhub update script:
-------------------------------------------------------------
emonHub update started
-------------------------------------------------------------
- usr directory: /home/pi
git pull /home/pi/emonhub
K0den-wibeee_interface
* emon-pi
hmm01i-syslogging
On branch emon-pi
Your branch is up-to-date with 'origin/emon-pi'.
nothing to commit, working directory clean
Already up-to-date.
emonhub.service already installed
Running emonhub automatic node addition script
[[5]]
Node 5 already present
[[6]]
Node 6 already present
[[7]]
Node 7 already present
[[8]]
Node 8 already present
[[9]]
Node 9 already present
[[10]]
Node 10 already present
[[11]]
Node 11 already present
[[12]]
Node 12 already present
[[13]]
Node 13 already present
[[14]]
Node 14 already present
[[19]]
Node 19 already present
[[20]]
Node 20 already present
[[21]]
Node 21 already present
[[22]]
Node 22 already present
[[23]]
Node 23 already present
[[24]]
Node 24 already present
[[25]]
Node 25 already present
[[26]]
Node 26 already present
Start emoncms update:
-------------------------------------------------------------
Emoncms update started
Emoncms update script V1.3 (26th March 2019)
-------------------------------------------------------------
- usr directory: /home/pi
- emoncms directory: /var/www/emoncms
current settings.php md5: 468b110786bf0ab60a856fa3c0036f0f
Default settings.php md5: 468b110786bf0ab60a856fa3c0036f0f
settings.php has NOT been user modifed
Checking status of /var/www/emoncms git repository
- git branch: stable
- no local changes
- running: git pull origin stable
From https://github.com/emoncms/emoncms
* branch stable -> FETCH_HEAD
Already up-to-date.
NEW default settings.php md5: 468b110786bf0ab60a856fa3c0036f0f
settings.php not updated
-------------------------------------------------------------
Updating Emoncms Modules
-------------------------------------------------------------
------------------------------------------
Updating /var/www/emoncms/Modules/app module
------------------------------------------
- git branch: stable
- git tags: 1.2.0-21-g8edcc30
- no local changes
- running: git pull origin stable
From https://github.com/emoncms/app
* branch stable -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating /var/www/emoncms/Modules/config module
------------------------------------------
- git branch: stable
- git tags: 1.0.0-8-g5e96633
- no local changes
- running: git pull origin stable
From https://github.com/emoncms/config
* branch stable -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating /var/www/emoncms/Modules/dashboard module
------------------------------------------
- git branch: stable
- git tags: 1.3.1-36-g8be171b
- no local changes
- running: git pull origin stable
From https://github.com/emoncms/dashboard
* branch stable -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating /var/www/emoncms/Modules/device module
------------------------------------------
- git branch: master
- git tags: 1.1.1-35-gda7fa4e
- no local changes
- running: git pull origin master
From https://github.com/emoncms/device
* branch master -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating /var/www/emoncms/Modules/graph module
------------------------------------------
- git branch: stable
- git tags: 1.2.1-66-g91d4db5
- no local changes
- running: git pull origin stable
From https://github.com/emoncms/graph
* branch stable -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating /var/www/emoncms/Modules/wifi module
------------------------------------------
- git branch: stable
- git tags: 1.2.0-33-g2f37618
- no local changes
- running: git pull origin stable
From https://github.com/emoncms/wifi
* branch stable -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating postprocess module
------------------------------------------
- git branch: emonpi
fatal: No names found, cannot describe anything.
- git tags:
- no local changes
- running: git pull origin emonpi
From https://github.com/emoncms/postprocess
* branch emonpi -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating sync module
------------------------------------------
- git branch: master
- git tags: 1.0.0-7-gc391cd2
- no local changes
- running: git pull origin master
From https://github.com/emoncms/sync
* branch master -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating backup module
------------------------------------------
- git branch: stable
- git tags: 1.1.5-37-gf55d616
- no local changes
- running: git pull origin stable
From https://github.com/emoncms/backup
* branch stable -> FETCH_HEAD
Already up-to-date.
------------------------------------------
Auto Installation of Emoncms Modules
------------------------------------------
- graph module already installed
- device module already installed
- postprocess module already installed
- sync module already installed
- backup module already installed
www-data : www-data video systemd-journal
------------------------------------------
SERVICES
------------------------------------------
emoncms_mqtt.service already installed
feedwriter.service already installed
service-runner.service already installed
------------------------------------------
Update Emoncms database
[]
Restarting Services...
- sudo systemctl restart feedwriter.service
--- ActiveState=active ---
- sudo systemctl restart emoncms_mqtt.service
--- ActiveState=active ---
- sudo systemctl restart emonhub.service
--- ActiveState=active ---
- sudo /etc/init.d/openhab restart
Restarting openhab (via systemctl): openhab.service.
● openhab.service - Starts and stops the openHAB Home Automation Bus
Loaded: loaded (/usr/lib/systemd/system/openhab.service; enabled)
Active: active (running) since Mon 2019-04-29 20:00:14 UTC; 1s ago
Docs: http://www.openhab.org
Process: 18964 ExecStartPre=/usr/share/openhab/bin/setpermissions.sh (code=exited, status=0/SUCCESS)
Main PID: 19017 (openhab.sh)
CGroup: /system.slice/openhab.service
├─19017 /bin/sh /usr/share/openhab/bin/openhab.sh -o
└─19045 /usr/bin/java -Dlogback.configurationFile=/etc/openhab/logback.xml -Dosgi.clean=true -Declipse.ignoreApp=true -Dosgi.noShutdown=true -Djetty.port=8080 -Dopenhab.configfile=/etc/openhab/configurations/openhab.cfg -Dopenhab.configdir=/etc/openhab/configurations -Dopenhab.logdir=/var/log/openhab -Dsmarthome.userdata=/var/lib/openhab -Djetty.home=/usr/share/openhab -Djetty.port.ssl=8443 -Djetty.config=/etc/openhab/jetty -Djetty.logs=/var/log/openhab -Djetty.rundir=/usr/share/openhab -Dfelix.fileinstall.dir=/usr/share/openhab/addons -Dfelix.fileinstall.filter=.*\.jar -Djava.library.path=/usr/share/openhab/lib -Djava.security.auth.login.config=/etc/openhab/login.conf -Dorg.quartz.properties=/etc/openhab/quartz.properties -Dequinox.ds.block_timeout=240000 -Dequinox.scr.waitTimeOnBlock=60000 -Dfelix.fileinstall.active.level=4 -Djava.awt.headless=true -jar /usr/share/openhab/server/plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar -configuration /var/lib/openhab/workspace -data /var/lib/openhab/workspace -console 5555
Apr 29 20:00:12 emonpi setpermissions.sh[18964]: setting owner openhab:openhab recursively for /var/lib/openhab
Apr 29 20:00:12 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab recursively for /var/log/openhab
Apr 29 20:00:12 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab recursively for /etc/openhab/configurations
Apr 29 20:00:14 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab for /usr/share/openhab/webapps/static
Apr 29 20:00:14 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab for /etc/openhab/jetty/etc/keystore
Apr 29 20:00:14 emonpi setpermissions.sh[18964]: setting permissions and owner openhab:openhab for /etc/openhab/configurations/users.cfg
Apr 29 20:00:14 emonpi systemd[1]: Started Starts and stops the openHAB Home Automation Bus.
Installing emoncms logrotate...
Now setting up Logrotate...
Backing up old logrotate configuration...
Linked to new logrotate configuration...
Backing up old logrotate cron job...
Linked to new logrotate cron job...
Completed
setup logrotate state & logfile in /var/log/logrotate
mkdir: cannot create directory ‘/var/log/logrotate’: File exists
Completed
Running logrotate...
set log rotate config owner to root
chown: changing ownership of ‘/etc/logrotate.conf’: Read-only file system
------------------------------------------
Emoncms update script complete
------------------------------------------
Starting emonPi LCD service..
Filesystem is locked - Read Only access
type ' rpi-rw ' to unlock
-------------------------------------------------------------
emonPi update done: Mon 29 Apr 20:00:44 UTC 2019
-------------------------------------------------------------
restarting service-runnertimezone in my account (inside emoncms) and the one in raspi config match up
and this is the low write sd card image if i forgot to mention that
Almost certainly Daft suggestion as it is self hosted… /log/var filling up.
and how do i confirm if that is the case?
i thought it cleared things like that on it’s own
that one suggest that it has been fixed
and then again my input works fine and that is not what the link is about
so i’m still puzzled,
@TrystanLea @glyn.hudson if something is filling up should the user not get a warning and simple instructions on how to fix it?
df -h
So it should if configured correctly. I have been saying this is a problem for ages with little traction…
The fix, as it stands, doesn’t work IMNSHO.
I believe this fix does on an OCT18 SD image (and provides some additional log retention).
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.4G 2.2G 1.1G 67% /
devtmpfs 237M 0 237M 0% /dev
tmpfs 241M 0 241M 0% /dev/shm
tmpfs 241M 4.5M 237M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 241M 0 241M 0% /sys/fs/cgroup
tmpfs 40M 28K 40M 1% /var/lib/openhab
tmpfs 1.0M 4.0K 1020K 1% /var/lib/dhcpcd5
tmpfs 1.0M 0 1.0M 0% /var/lib/dhcp
tmpfs 50M 520K 50M 2% /var/log
tmpfs 1.0M 0 1.0M 0% /var/tmp
tmpfs 30M 40K 30M 1% /tmp
/dev/mmcblk0p1 60M 22M 39M 37% /boot
/dev/mmcblk0p3 969M 41M 879M 5% /home/pi/data
does not look full to me, or am i wrong?
Nope you are correct.
I see you are posting by API, is there anything in the log in the admin screen?
You can change the log level in settings.php to INFO so you will see an entry each time you post to emoncms via the API.
What do you get as your responses to the GET?
i will have to check, put i assume the GET is ok since INPUTS are updated
so if i do the copy log to clipboard i only get this
ohhhh
ERROR
missed that
Yes good point. Change the logging level and see if anything additional that will help.
Can you post output from…
systemctl status feedwriter.service
It isn’t actually an error. The log should be as INFO for that message.
pi@emonpi(ro):~$ systemctl status feedwriter.service
● feedwriter.service - Emoncms feedwriter script
Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; disabled)
Active: active (running) since Sun 2019-05-05 10:35:33 UTC; 4min 11s ago
Docs: emoncms/install-service-feedwriter.md at master · emoncms/emoncms · GitHub
Main PID: 2262 (php)
CGroup: /system.slice/feedwriter.service
└─2262 /usr/bin/php /var/www/emoncms/scripts/feedwriter.php
pi@emonpi(ro):~$
That looks OK,
On the admin page, under the services
How many points pending write?
0
feedwriter Active Running - sleep 60s 0 feed points pending write