System working fine for months, all of a sudden, emonhub won't start

[wainsley64 (Warren)]

Hi can someone please help me with this?
Have emonpi working great for many months now, then 2 days ago I notice none of my feeds work. Looked closer and noticed that the emonhub service won’t start even after reboots.

didn’t update, system didn’t reboot,
it just stopped for no apparent reason.

other weird thing I noticed is that my emonhub.conf file has a whole bunch of additional nodes in it that I’m quite sure weren’t there before. eg. ( emonTx_4, emonTx_1, emonTx_2, 3phase2, etc.) and some lines I’ve never seen before (such as… #Firmware =<V1.4 (un-comment))

I’ve been on low-write 10.2.0 since sometime in January.
Seems like there’s nothing in the logs…

Thanks much.

[borpin (Brian Orpin)]

Instead of posting the image of the status, could you paste the actual text please.

first do

sudo systemctl restart emonhub

before running the status command.

[edit]

please also click this button and paste the result into a post with no further formatting please.

[wainsley64 (Warren)]

@borpin

sudo systemctl restart emonhub

ok thanks.

● emonhub.service - emonHub service description
   Loaded: loaded (/opt/openenergymonitor/emonhub/service/emonhub.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2020-03-15 15:22:16 EST; 4min 52s ago
  Process: 15361 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 15362 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 15363 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 15364 ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
 Main PID: 15364 (code=exited, status=1/FAILURE)

Mar 15 15:22:16 emonpi systemd[1]: emonhub.service: Service RestartSec=100ms expired, scheduling restart.
Mar 15 15:22:16 emonpi systemd[1]: emonhub.service: Scheduled restart job, restart counter is at 5.
Mar 15 15:22:16 emonpi systemd[1]: Stopped emonHub service description.
Mar 15 15:22:16 emonpi systemd[1]: emonhub.service: Start request repeated too quickly.
Mar 15 15:22:16 emonpi systemd[1]: emonhub.service: Failed with result 'exit-code'.
Mar 15 15:22:16 emonpi systemd[1]: Failed to start emonHub service description.
~
~
~
~
~
~
~
~

pi@emonpi:~ $

.
.
<details><summary>Server Information</summary>

# Server Information
## Services
 
- **emonhub** :- **Failed** Failed
- **emoncms_mqtt** :- **Active** Running
- **feedwriter** :- **Active** Running - sleep 300s 0 feed points pending write
- **service-runner** :- **Active** Running
- **emonPiLCD** :- **Inactive** Dead
- **redis-server** :- **Active** Running
- **mosquitto** :- **Active** Running
- **demandshaper** :- **Active** Running

## Emoncms
 
- **Version** :- low-write 10.2.0
- **Modules** :- Administration | App v2.1.0 | Backup v2.2.1 | EmonHub Config v2.0.4 | Dashboard v2.0.6 | DemandShaper v1.2.2 | Device v2.0.4 | EventProcesses | Feed | Graph v2.0.8 | Input | Postprocess v2.1.2 | CoreProcess | Schedule | Network Setup v1.0.0 | sync | Time | User | Visualisation | WiFi v2.0.2
- **Git** :- 
     - **URL** :- https://github.com/emoncms/emoncms.git
     - **Branch** :- * stable
     - **Describe** :- 10.2.0

## Server
 
- **OS** :- Linux 4.19.75-v7+
- **Host** :- emonpi | emonpi | (192.168.15.209)
- **Date** :- 2020-03-15 16:24:57 EDT
- **Uptime** :- 15:24:57 up  8:46,  1 user,  load average: 0.01, 0.02, 0.00

## Memory
 
- **RAM** :- Used: 23.75%
    - **Total** :- 975.62 MB
     - **Used** :- 231.69 MB
     - **Free** :- 743.93 MB
- **Swap** :- Used: 0.00%
    - **Total** :- 100 MB
     - **Used** :- 0 B
     - **Free** :- 100 MB
Write Load Period
 
## Disk
- **/** :- Used: 55.08%
    - **Total** :- 3.92 GB
     - **Used** :- 2.16 GB
     - **Free** :- 1.57 GB
     - **Write Load** :- 31.86 B/s (3 hours 6 mins)
- **/var/opt/emoncms** :- Used: 3.05%
    - **Total** :- 9.98 GB
     - **Used** :- 312.1 MB
     - **Free** :- 9.17 GB
     - **Write Load** :- 0 B/s (3 hours 6 mins)
- **/boot** :- Used: 20.55%
    - **Total** :- 252.05 MB
     - **Used** :- 51.79 MB
     - **Free** :- 200.25 MB
     - **Write Load** :- 0 B/s (3 hours 6 mins)
- **/var/log** :- Used: 100.00%
    - **Total** :- 50 MB
     - **Used** :- 50 MB
     - **Free** :- 0 B
     - **Write Load** :- n/a

## HTTP
 
- **Server** :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

## MySQL
 
- **Version** :- 5.5.5-10.3.17-MariaDB-0+deb10u1
- **Host** :- localhost:6379 (127.0.0.1)
- **Date** :- 2020-03-15 15:24:57 (UTC -05:00‌​)
- **Stats** :- Uptime: 31589  Threads: 14  Questions: 65092  Slow queries: 0  Opens: 46  Flush tables: 1  Open tables: 40  Queries per second avg: 2.060

## Redis
 
- **Version** :- 
     - **Redis Server** :- 5.0.3
     - **PHP Redis** :- 5.0.2
- **Host** :- localhost:6379
- **Size** :- 218 keys (815.05K)
- **Uptime** :- 0 days
 
## MQTT Server
 
- **Version** :- Mosquitto 1.5.7
- **Host** :- localhost:1883 (127.0.0.1)

## PHP
 
- **Version** :- 7.3.9-1~deb10u1 (Zend Version 3.3.9)
- **Modules** :- apache2handler | calendar v7.3.9-1~deb10u1 | Core v7.3.9-1~deb10u1 | ctype v7.3.9-1~deb10u1 | curl v7.3.9-1~deb10u1 | date v7.3.9-1~deb10u1 | dom v20031129 | exif v7.3.9-1~deb10u1 | fileinfo v7.3.9-1~deb10u1 | filter v7.3.9-1~deb10u1 | ftp v7.3.9-1~deb10u1 | gd v7.3.9-1~deb10u1 | gettext v7.3.9-1~deb10u1 | hash v7.3.9-1~deb10u1 | iconv v7.3.9-1~deb10u1 | json v1.7.0 | libxml v7.3.9-1~deb10u1 | mbstring v7.3.9-1~deb10u1 | mosquitto v0.4.0 | mysqli v7.3.9-1~deb10u1 | mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $ | openssl v7.3.9-1~deb10u1 | pcre v7.3.9-1~deb10u1 | PDO v7.3.9-1~deb10u1 | pdo_mysql v7.3.9-1~deb10u1 | Phar v7.3.9-1~deb10u1 | posix v7.3.9-1~deb10u1 | readline v7.3.9-1~deb10u1 | redis v5.0.2 | Reflection v7.3.9-1~deb10u1 | session v7.3.9-1~deb10u1 | shmop v7.3.9-1~deb10u1 | SimpleXML v7.3.9-1~deb10u1 | sockets v7.3.9-1~deb10u1 | sodium v7.3.9-1~deb10u1 | SPL v7.3.9-1~deb10u1 | standard v7.3.9-1~deb10u1 | sysvmsg v7.3.9-1~deb10u1 | sysvsem v7.3.9-1~deb10u1 | sysvshm v7.3.9-1~deb10u1 | tokenizer v7.3.9-1~deb10u1 | wddx v7.3.9-1~deb10u1 | xml v7.3.9-1~deb10u1 | xmlreader v7.3.9-1~deb10u1 | xmlwriter v7.3.9-1~deb10u1 | xsl v7.3.9-1~deb10u1 | Zend OPcache v7.3.9-1~deb10u1 | zlib v7.3.9-1~deb10u1

## Pi
 
- **Model** :- Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)
 
- **Serial num.** :- 248B5069
- **Temperature** :- 52.08°C - 52.1°C
- **emonpiRelease** :- emonSD-17Oct19
- **File-system** :- read-write
</details>

<details><summary>Client Information</summary>

# Client Information
## HTTP
 
- **Browser** :- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
- **Language** :- en-US,en;q=0.5


## Window
 
- **Size** :- 1263 x 687


## Screen
 
- **Resolution** :- 1280 x 800
</details>

[borpin (Brian Orpin)]

You need to make the SSH window wider so the whole message is captured.

Please also post using 3 backticks (top left of keyboard). If you look at the edit on the post above you will see how.

[borpin (Brian Orpin)]

The issue is /var/log/ is full. You need to remove the big files.

However before you do, please run this command and post it so we can see what the issue was.

du -a /var/log | sort -n -r | head -n 20

To make the files smaller (deleting them can be a bad idea) use

truncate -s 0 /var/log/<log file>

[wainsley64 (Warren)]

ok thanks.
i edited the original SSH window post above.

~
~

pi@emonpi:~ $ du -a /var/log | sort -n -r | head -n 20
du: cannot read directory '/var/log/private': Permission denied
51200   /var/log
35508   /var/log/emonhub/emonhub.log.1
35508   /var/log/emonhub
14132   /var/log/logrotate/logrotate.log
14132   /var/log/logrotate
484     /var/log/daemon.log
348     /var/log/wtmp
328     /var/log/auth.log
288     /var/log/lastlog
28      /var/log/apt/eipp.log.xz
28      /var/log/apt
24      /var/log/faillog
20      /var/log/emonpilcd/emonpilcd.log
20      /var/log/emonpilcd
16      /var/log/syslog
16      /var/log/messages
8       /var/log/wifiAP.log
0       /var/log/user.log
0       /var/log/rotated_logs
0       /var/log/redis/redis-server.log
pi@emonpi:~ $

[Bill.Thomson]

@wainsley64

Permission denied

Try:

sudo du -a /var/log | sort -n -r | head -n 20

[wainsley64 (Warren)]

@borpin

du -a /var/log | sort -n -r | head -n 20

pi@emonpi:~ $
pi@emonpi:~ $
pi@emonpi:~ $
pi@emonpi:~ $ sudo du -a /var/log | sort -n -r | head -n 20
51200   /var/log
35508   /var/log/emonhub/emonhub.log.1
35508   /var/log/emonhub
14132   /var/log/logrotate/logrotate.log
14132   /var/log/logrotate
484     /var/log/daemon.log
348     /var/log/wtmp
328     /var/log/auth.log
288     /var/log/lastlog
28      /var/log/apt/eipp.log.xz
28      /var/log/apt
24      /var/log/faillog
20      /var/log/emonpilcd/emonpilcd.log
20      /var/log/emonpilcd
16      /var/log/syslog
16      /var/log/messages
8       /var/log/wifiAP.log
0       /var/log/user.log
0       /var/log/rotated_logs
0       /var/log/redis/redis-server.log
pi@emonpi:~ $

[borpin (Brian Orpin)]

Sorry, missed a wildcard

sudo du -a /var/log* | sort -n -r | head -n 20

However, you can delete this file

/var/log/emonhub/emonhub.log.1

[wainsley64 (Warren)]

pi@emonpi:~ $
pi@emonpi:~ $
pi@emonpi:~ $ sudo du -a /var/log* | sort -n -r | head -n 20
51200   /var/log
50972   /var/log.bak
35512   /var/log.bak/emonhub
35508   /var/log/emonhub/emonhub.log.1
35508   /var/log/emonhub
35508   /var/log.bak/emonhub/emonhub.log.1
15136   /var/log.old
14536   /var/log.old/emonhub
14144   /var/log.bak/logrotate
14140   /var/log.bak/logrotate/logrotate.log
14132   /var/log/logrotate/logrotate.log
14132   /var/log/logrotate
7316    /var/log.old/emonhub/emonhub.log.3.gz
7208    /var/log.old/emonhub/emonhub.log.4.gz
488     /var/log.bak/daemon.log
484     /var/log/daemon.log
352     /var/log.bak/wtmp
348     /var/log/wtmp
332     /var/log.bak/auth.log
328     /var/log/auth.log
pi@emonpi:~ $

[borpin (Brian Orpin)]

@wainsley64

/var/log/emonhub/emonhub.log.1

delete this file and you should be fine.

For others (@pb66), where has `/var/log/log.bak’ come from?

[wainsley64 (Warren)]

so do I manually go through and delete the duplicate nodes that seems to have appeared out of nowhere?

LAST ENTRIES ON THE LOG FILE

Starting update via service-runner-update.sh (v3.0) >
- emonSD version: emonSD-17Oct19
emonSD base image check passed...continue update
git pull /opt/openenergymonitor/EmonScripts
master
* stable
On branch stable
Your branch is up to date with 'origin/stable'.

nothing to commit, working tree clean
Already up to date.
-------------------------------------------------------------
Main Update Script
-------------------------------------------------------------
Date: Sun 15 Mar 16:23:09 EST 2020
EUID: 1000
openenergymonitor_dir: /opt/openenergymonitor
type: all
firmware:
Hardware detected: rfm2pi
git pull /opt/openenergymonitor/emonpi
* master
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
Fetching origin
Already up to date.
git pull /opt/openenergymonitor/RFM2Pi
* master
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
Fetching origin
Already up to date.
Start emonhub update script:
-------------------------------------------------------------
emonHub update
-------------------------------------------------------------
git pull /opt/openenergymonitor/emonhub
* emon-pi
On branch emon-pi
Your branch is up to date with 'origin/emon-pi'.

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

modified: src/interfacers/__init__.py

Untracked files:
(use "git add ..." to include in what will be committed)

src/interfacers/EmonHubVEDirectInterfacer2.py
src/interfacers/EmonHubVEDirectInterfacer3.py

no changes added to commit (use "git add" and/or "git commit -a")
Already up to date.
Failed to restart .service.service: Unit .service.service not found.
- Service
Running emonhub automatic node addition script
[[5]]
[[5]]
[[5]]
Node 5 already present
[[6]]
[[6]]
[[6]]
Node 6 already present
[[7]]
[[7]]
[[7]]
Node 7 already present
[[8]]
[[8]]
[[8]]
Node 8 already present
[[9]]
[[9]]
[[9]]
Node 9 already present
[[10]]
[[10]]
[[10]]
Node 10 already present
[[11]]
[[11]]
[[11]]
Node 11 already present
[[12]]
[[12]]
[[12]]
Node 12 already present
[[13]]
[[13]]
[[13]]
Node 13 already present
[[14]]
[[14]]
[[14]]
Node 14 already present
[[15]]
[[15]]
[[15]]
Node 15 already present
[[16]]
[[16]]
[[16]]
Node 16 already present
[[17]]
Node 17 already present
[[19]]
[[19]]
[[19]]
Node 19 already present
[[20]]
[[20]]
[[20]]
Node 20 already present
[[21]]
[[21]]
[[21]]
Node 21 already present
[[22]]
[[22]]
[[22]]
Node 22 already present
[[23]]
[[23]]
[[23]]
Node 23 already present
[[24]]
[[24]]
[[24]]
Node 24 already present
[[25]]
[[25]]
[[25]]
Node 25 already present
[[26]]
[[26]]
[[26]]
Node 26 already present

Start emoncms update:
-------------------------------------------------------------
Update Emoncms Core
-------------------------------------------------------------


Checking status of /var/www/emoncms git repository
- git branch: stable
- no local changes
- running: git pull origin stable

Fetching origin
From https://github.com/emoncms/emoncms
* branch stable -> FETCH_HEAD
Already up to date.

Update Emoncms database
[]

-------------------------------------------------------------
Update Emoncms Services
-------------------------------------------------------------
emoncms_mqtt.service already installed
feedwriter.service already installed
service-runner.service already installed

Reloading systemctl deamon
Restarting Services...
- sudo systemctl restart feedwriter.service
--- ActiveState=active ---
- sudo systemctl restart emoncms_mqtt.service
--- ActiveState=active ---
- sudo systemctl restart emonhub.service
--- ActiveState=active ---
/opt/openenergymonitor/EmonScripts/sudoers.d/emoncms-rebootbutton: parsed OK
emonPi emoncms admin reboot button sudoers updated

-------------------------------------------------------------
Update Emoncms Modules
-------------------------------------------------------------
------------------------------------------
Updating /var/www/emoncms/Modules/app module
------------------------------------------
- git branch: stable
- git tags: 2.1.0
- no local changes
- running: git pull origin stable

Fetching origin
Already on 'stable'
Your branch is up to date with 'origin/stable'.
Already up to date.

------------------------------------------
Updating /var/www/emoncms/Modules/config module
------------------------------------------
- git branch: stable
- git tags: 2.0.2-9-g1bba25a
- no local changes
- running: git pull origin stable

Fetching origin
Already on 'stable'
Your branch is up to date with 'origin/stable'.
Already up to date.

------------------------------------------
Updating /var/www/emoncms/Modules/dashboard module
------------------------------------------
- git branch: stable
- git tags: 2.0.5
- no local changes
- running: git pull origin stable

Fetching origin
Already on 'stable'
Your branch is up to date with 'origin/stable'.
Already up to date.

------------------------------------------
Updating /var/www/emoncms/Modules/device module
------------------------------------------
- git branch: stable
- git tags: 2.0.4
- no local changes
- running: git pull origin stable

Fetching origin
Already on 'stable'
Your branch is up to date with 'origin/stable'.
Already up to date.

------------------------------------------
Updating /var/www/emoncms/Modules/graph module
------------------------------------------
- git branch: stable
- git tags: 2.0.8-2-g7df046d
- no local changes
- running: git pull origin stable

Fetching origin
Already on 'stable'
Your branch is up to date with 'origin/stable'.
Already up to date.

------------------------------------------
Updating /var/www/emoncms/Modules/setup module
------------------------------------------
- git branch: stable
fatal: No names found, cannot describe anything.
- git tags:
- no local changes
- running: git pull origin stable

Fetching origin
Already on 'stable'
Your branch is up to date with 'origin/stable'.
Already up to date.

------------------------------------------
Updating /var/www/emoncms/Modules/wifi module
------------------------------------------
- git branch: stable
- git tags: 2.0.0-29-g274bcd5
- no local changes
- running: git pull origin stable

Fetching origin
Already on 'stable'
Your branch is up to date with 'origin/stable'.
Already up to date.

/opt/openenergymonitor/EmonScripts/sudoers.d/wifi-sudoers: parsed OK
wifi sudoers entry updated
------------------------------------------
Updating /opt/emoncms/modules/backup module
------------------------------------------
- git branch: stable
- git tags: 2.2.1
- no local changes
- running: git pull origin stable

Already up to date.
Already on 'stable'
Your branch is up to date with 'origin/stable'.

------------------------------------------
Updating /opt/emoncms/modules/demandshaper module
------------------------------------------
- git branch: stable
- git tags: 1.2.0-27-gfdfbf3d
- no local changes
- running: git pull origin stable

Already up to date.
Already on 'stable'
Your branch is up to date with 'origin/stable'.

------------------------------------------
Updating /opt/emoncms/modules/postprocess module
------------------------------------------
- git branch: stable
- git tags: 2.1.0-21-g0e2d78f
- no local changes
- running: git pull origin stable

Already up to date.
Already on 'stable'
Your branch is up to date with 'origin/stable'.

------------------------------------------
Updating /opt/emoncms/modules/sync module
------------------------------------------
- git branch: stable
- git tags: 2.0.3
- no local changes
- running: git pull origin stable

Already up to date.
Already on 'stable'
Your branch is up to date with 'origin/stable'.

------------------------------------------
Updating /opt/emoncms/modules/usefulscripts module
------------------------------------------
- git branch: stable
fatal: No names found, cannot describe anything.
- git tags:
- no local changes
- running: git pull origin stable

Already up to date.
Already on 'stable'
Your branch is up to date with 'origin/stable'.

--------------------------------------------
Backup module installation and update script
--------------------------------------------
- Loading EmonScripts config.ini
- Copying default.config.cfg to config.cfg
- Setting config.cfg settings
- Backup module symlink already exists
- PHP Version: 7.3
- Creating /etc/php/7.3/mods-available/emoncmsbackup.ini
post_max_size = 3G
upload_max_filesize = 3G
upload_tmp_dir = /var/opt/emoncms/backup/uploads
- phpenmod emoncmsbackup
- /var/opt/emoncms/backup already exists
- /var/opt/emoncms/backup/uploads already exists
- restarting apache

Update Emoncms database
[]


-------------------------------------------------------------
emonPi update done: Sun 15 Mar 16:24:00 EST 2020
-------------------------------------------------------------
restarting service-runner

[pb66 (Paul)]

@borpin

For others (@pb66), where has `/var/log/log.bak’ come from?

I don’t see a /var/log/log.bak but I do see a /var/log.bak which is the persisted copy (aka backup hence the .bak) of the /var/log partition in ram.

I think the output is confused due to listing /var/log* rather than /var/log/* since I would expect you are only interested in the files filling the 50Mb ram partition, which IMO is way too small for a Pi with >= 1Gb of ram.

I’d be more interested to know how the emonhub log reaches 35Mb when the inbuilt rotation should happen at 5Mb and there is also logrotate working with log2ram, that shouldn’t happen! maybe there is a clue in the 14Mb of logrotate log? And where is the log2ram log?

Seeing the output from a ls -la /var/log/ command would be useful to see what dates are on the files. It looks like logrotate maybe isn’t working due to an error.

Is this image a new style image or is this an older image before full adoptation of log2ram? I’ve lost track of the image dates.

[pb66 (Paul)]

Check out this portion of the update log

@wainsley64

-------------------------------------------------------------
emonHub update
-------------------------------------------------------------
git pull /opt/openenergymonitor/emonhub
* emon-pi
On branch emon-pi
Your branch is up to date with 'origin/emon-pi'.

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

modified: src/interfacers/__init__.py

Untracked files:
(use "git add ..." to include in what will be committed)

src/interfacers/EmonHubVEDirectInterfacer2.py
src/interfacers/EmonHubVEDirectInterfacer3.py

no changes added to commit (use "git add" and/or "git commit -a")
Already up to date.
Failed to restart .service.service: Unit .service.service not found.
- Service
Running emonhub automatic node addition script
[[5]]
[[5]]
[[5]]
Node 5 already present
[[6]]
[[6]]
[[6]]
Node 6 already present
[[7]]
[[7]]
[[7]]
Node 7 already present
[[8]]
[[8]]
[[8]]
Node 8 already present
[[9]]
[[9]]
[[9]]
Node 9 already present
[[10]]
[[10]]
[[10]]
Node 10 already present
[[11]]
[[11]]
[[11]]
Node 11 already present
[[12]]
[[12]]
[[12]]
Node 12 already present
[[13]]
[[13]]
[[13]]
Node 13 already present
[[14]]
[[14]]
[[14]]
Node 14 already present
[[15]]
[[15]]
[[15]]
Node 15 already present
[[16]]
[[16]]
[[16]]
Node 16 already present
[[17]]
Node 17 already present
[[19]]
[[19]]
[[19]]
Node 19 already present
[[20]]
[[20]]
[[20]]
Node 20 already present
[[21]]
[[21]]
[[21]]
Node 21 already present
[[22]]
[[22]]
[[22]]
Node 22 already present
[[23]]
[[23]]
[[23]]
Node 23 already present
[[24]]
[[24]]
[[24]]
Node 24 already present
[[25]]
[[25]]
[[25]]
Node 25 already present
[[26]]
[[26]]
[[26]]
Node 26 already present

It explains where the additional nodes came from (an earlier run of the update), points out there are some local changes to emonhub and indicates that emonhub couldn’t be restarted after the update as the service unit couldn’t be found, although it also says that it is up to date with origin, but is origin still the OEM repo ? The local changes may have caused the previous update to fail (despite no changes in origin) and the service unit wasn’t properly installed as a result?? I’m reading between the lines a bit and filling in the gaps with a touch of guesswork I’m afraid

[wainsley64 (Warren)]

@pb66

ls -la /var/log/

ok… I have pasted the output below.
I’ve been using this since the Nov2016 image and I think I updated to the oct19 image sometime in December.

Thanks everyone for your help so far.

pi@emonpi:~ $
pi@emonpi:~ $ ls -la /var/log/
total 1200
drwxr-xr-x 12 root      root       580 Mar 15 17:17 .
drwxr-xr-x 14 root      root      4096 Oct 22 22:53 ..
-rw-r--r--  1 root      root         0 Nov  1 00:00 alternatives.log
drwxr-x---  2 root      adm        100 Oct 24 00:00 apache2
drwxr-xr-x  2 root      root       100 Nov  1 00:00 apt
-rw-r-----  1 root      adm     347323 Mar 15 17:50 auth.log
-rw-r--r--  1 root      root         0 Sep 25 19:24 bootstrap.log
-rw-rw----  1 root      utmp         0 Mar  1 00:17 btmp
-rw-r-----  1 root      adm       4486 Mar 15 17:50 daemon.log
-rw-r-----  1 root      adm       1064 Mar 15 16:22 debug
-rw-r--r--  1 root      root         0 Nov  1 00:00 dpkg.log
drwxr-xr-x  2 pi        root       180 Mar  8 00:17 emoncms
drwxrwxr-x  2 root      emonhub     80 Mar 12 20:18 emonhub
drwxr-xr-x  2 pi        root        60 Oct 17 04:35 emonpilcd
-rw-r--r--  1 root      root     24000 Oct 17 04:33 faillog
-rw-r-----  1 root      adm      28883 Mar 15 16:22 kern.log
-rw-rw-r--  1 root      utmp    292292 Mar 15 17:50 lastlog
-rw-r--r--  1 root      root       528 Mar 15 17:17 log2ram.log
drwxr-xr-x  2 root      adm         80 Mar 15 17:17 logrotate
-rw-r-----  1 root      adm      44473 Mar 15 17:17 messages
drwxr-xr-x  2 mosquitto root        60 Mar 15 16:23 mosquitto
drwxr-s---  2 mysql     adm         60 Mar 15 16:23 mysql
drwx------  2 root      root        40 Sep 25 19:24 private
drwxr-s---  2 redis     adm         60 Mar 15 16:23 redis
lrwxrwxrwx  1 root      root        12 Oct 17 04:42 rotated_logs -> /var/log.old
-rw-r-----  1 root      adm      90368 Mar 15 17:50 syslog
-rw-r-----  1 root      adm        982 Mar 15 16:22 user.log
-rw-r--r--  1 root      root      6340 Mar 15 16:24 wifiAP.log
-rw-rw-r--  1 root      utmp    355968 Mar 15 17:50 wtmp
pi@emonpi:~ $
pi@emonpi:~ $

[borpin (Brian Orpin)]

@pb66

I don’t see a /var/log/log.bak but I do see a /var/log.bak

Doh!

@pb66

I think the output is confused due to listing /var/log* rather than /var/log/*

You are correct… should be sudo du -a /var/log/* | sort -n -r | head -n 30 to list biggest files.

@pb66

I’d be more interested to know how the emonhub log reaches 35Mb when the inbuilt rotation should happen at 5Mb

Yes that was what I wondered.

@pb66

maybe there is a clue in the 14Mb of logrotate log?

I was going to get to that once the system was running . @pb66 I’m sure I pointed this out before, but the logrotate.log does not appear to be rotated which is why it grows! I also note that the log2ram log doesn’t have any timestamps.

@wainsley64, could you paste a portion of the /var/log/logrotate/logrotate.log file please. I suggest you use the truncate command above and wait an hour until logrotate has run again then you will have a single rotation log.

[wainsley64 (Warren)]

Thanks borpin,
I just copied the the tail section of the logfile… seem as if it just ran so I tried to capture a previous run also.
please have a look below, thanks.

Now: 2020-03-16 04:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)

rotating pattern: /var/log/syslog
 after 1 days (7 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/syslog
  Now: 2020-03-16 04:17
  Last rotated at 2020-03-16 00:17
  log does not need rotating (log has been rotated at 2020-3-16 0:17, that is not day ago yet)

rotating pattern: /var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
 weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mail.info
  log /var/log/mail.info does not exist -- skipping
considering log /var/log/mail.warn
  log /var/log/mail.warn does not exist -- skipping
considering log /var/log/mail.err
  log /var/log/mail.err does not exist -- skipping
considering log /var/log/mail.log
  log /var/log/mail.log does not exist -- skipping
considering log /var/log/daemon.log
  Now: 2020-03-16 04:17
  Last rotated at 2020-03-15 17:17
  log does not need rotating (log has been rotated at 2020-3-15 17:17, that is not week ago yet)
considering log /var/log/kern.log
  Now: 2020-03-16 04:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/auth.log
  Now: 2020-03-16 04:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/user.log
  Now: 2020-03-16 04:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/lpr.log
  log /var/log/lpr.log does not exist -- skipping
considering log /var/log/cron.log
  log /var/log/cron.log does not exist -- skipping
considering log /var/log/debug
  Now: 2020-03-16 04:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/messages
  Now: 2020-03-16 04:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/ufw.log
 weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/ufw.log
  log /var/log/ufw.log does not exist -- skipping
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/wtmp  monthly (1 rotations)
olddir is /var/log.old, empty log files are rotated, only log files >= 1048576 bytes are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/wtmp
  Now: 2020-03-16 04:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not month ago yet)
  log does not need rotating ('minsize' directive is used and the log size is smaller than the minsize value)
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file 00_defaults
reading config file 00_olddir
olddir is now /var/log.old
reading config file alternatives
reading config file apache2
reading config file apt
reading config file btmp
reading config file dpkg
reading config file emoncms
olddir is now /var/log.old/emoncms
reading config file emonhub
olddir is now /var/log.old/emonhub
reading config file log2ram
reading config file mosquitto
reading config file mysql-server
reading config file nodered
reading config file redis-server
reading config file rsyslog
reading config file ufw
reading config file wtmp
Reading state from file: /var/log/logrotate/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state

Handling 17 logs

rotating pattern: /var/log/alternatives.log  monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/alternatives.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not month ago yet)

rotating pattern: /var/log/apache2/*.log  after 1 days (14 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apache2/access.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log is empty)
considering log /var/log/apache2/error.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log is empty)
considering log /var/log/apache2/other_vhosts_access.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log is empty)
not running prerotate script, since no logs will be rotated
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/apt/term.log  monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apt/term.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not month ago yet)

rotating pattern: /var/log/apt/history.log  monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apt/history.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not month ago yet)

rotating pattern: /var/log/btmp  monthly (1 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/btmp
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not month ago yet)

rotating pattern: /var/log/dpkg.log  monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/dpkg.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not month ago yet)

rotating pattern: /var/log/emoncms/*.log  weekly (4 rotations)
olddir is /var/log.old/emoncms, empty log files are rotated, log files >= 3145728 are rotated earlier, old logs are removed
considering log /var/log/emoncms/apache2-error.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/emoncms/emoncms.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/emoncms/emonpiupdate.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/emoncms/emonupdate.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/emoncms/exportbackup.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/emoncms/importbackup.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/emoncms/wificheck.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)

rotating pattern: /var/log/emonhub/emonhub.log  weekly (4 rotations)
olddir is /var/log.old/emonhub, empty log files are rotated, log files >= 3145728 are rotated earlier, old logs are removed
considering log /var/log/emonhub/emonhub.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)

rotating pattern: /var/log/log2ram.log
 after 1 days (7 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/log2ram.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-16 01:17
  log does not need rotating (log has been rotated at 2020-3-16 1:17, that is not day ago yet)

rotating pattern: /var/log/mosquitto/mosquitto.log  102400 bytes (7 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mosquitto/mosquitto.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log size is below the 'size' threshold)

rotating pattern: /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log  after 1 days (7 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mysql/mysql.log
  log /var/log/mysql/mysql.log does not exist -- skipping
considering log /var/log/mysql/mysql-slow.log
  log /var/log/mysql/mysql-slow.log does not exist -- skipping
considering log /var/log/mysql/mariadb-slow.log
  log /var/log/mysql/mariadb-slow.log does not exist -- skipping
considering log /var/log/mysql/error.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-16 00:17
  log does not need rotating (log has been rotated at 2020-3-16 0:17, that is not day ago yet)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/nodered-install.log
 weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/nodered-install.log
  log /var/log/nodered-install.log does not exist -- skipping

rotating pattern: /var/log/redis/redis-server*.log  weekly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/redis/redis-server.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)

rotating pattern: /var/log/syslog
 after 1 days (7 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/syslog
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-16 00:17
  log does not need rotating (log has been rotated at 2020-3-16 0:17, that is not day ago yet)

rotating pattern: /var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
 weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mail.info
  log /var/log/mail.info does not exist -- skipping
considering log /var/log/mail.warn
  log /var/log/mail.warn does not exist -- skipping
considering log /var/log/mail.err
  log /var/log/mail.err does not exist -- skipping
considering log /var/log/mail.log
  log /var/log/mail.log does not exist -- skipping
considering log /var/log/daemon.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:17
  log does not need rotating (log has been rotated at 2020-3-15 17:17, that is not week ago yet)
considering log /var/log/kern.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/auth.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/user.log
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/lpr.log
  log /var/log/lpr.log does not exist -- skipping
considering log /var/log/cron.log
  log /var/log/cron.log does not exist -- skipping
considering log /var/log/debug
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
considering log /var/log/messages
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not week ago yet)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/ufw.log
 weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/ufw.log
  log /var/log/ufw.log does not exist -- skipping
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/wtmp  monthly (1 rotations)
olddir is /var/log.old, empty log files are rotated, only log files >= 1048576 bytes are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/wtmp
  Now: 2020-03-16 05:17
  Last rotated at 2020-03-15 17:00
  log does not need rotating (log has been rotated at 2020-3-15 17:0, that is not month ago yet)
  log does not need rotating ('minsize' directive is used and the log size is smaller than the minsize value)

[borpin (Brian Orpin)]

@wainsley64

I just copied the the tail section of the logfile

As I thought, seems to be fine and not throwing any errors. Something odd happened but hopefully it is back working again?

[wainsley64 (Warren)]

Thanks borpin,

No, still not back up as yet.

I figure all I need to do is to delete the duplicated nodes in the conf file, but I haven’t done that yet because I wasn’t sure.

Can you let me know if I should go ahead and try this?
much appreciated.

[TrystanLea]

@wainsley64 did you see @pb66’s post starting:

@pb66

Check out this portion of the update log

This shows local changes to emonhub that have resulted in a failed update.

I would recommend removing your edit to:

emonhub/src/interfacers/__init__.py

clearing the duplicate entries in emonhub.conf and then running the Update process via the Admin UI again.

You can then reapply the modification to: emonhub/src/interfacers/init.py if needed?