Questions about emoncms 10.0.2 upgrade

It is an Emonpi, and it was updated early this year, so the base system is now emonSD-30Oct18.

It is working normally again (apart from the missing Backup link), so the issue appears to have been the incorrect links in the emonhub.service file.

Thanks @julesb yes there has been a change to the emonhub service config but the update script should handle that update process correctly - at least it did when I last tested. Perhaps something tripped it up during the update. Is sounds like you have it sorted. If anyone else experiences this please let me know.

The backup module is installed at /home/pi/backup and the emoncms UI part of it /home/pi/backup/backup-module should be symlinked to /var/www/emoncms/Modules/backup. To update the module manually you could try:

cd /home/pi/backup
git pull
ln -sf /home/pi/backup/backup-module /var/www/emoncms/Modules/backup

I will double check the update process, this should all be handled automatically, at least it did last time I tested.

Do you by any chance have a copy of your emonpiupdate.log for the last update @julesb? Its available from the administration page.

I did an install 3 times. The first one is the one that stopped Emonhub from running, and it had similar issues with avrdude. The first log was overwritten then I attempted the update again. This got stuck on ATMega328 upating and was truncated when I shutdown the device, although I had copied the log lines before rebooting. That one is shown below.

<details><summary>LAST ENTRIES ON THE LOG FILE</summary><br />
#############################################################<br />
Starting emonPi Update ><br />
via service-runner-update.sh<br />
Service Runner update script V1.1.1<br />
EUID: 1000<br />
Argument: all<br />
Tue 18 Jun 11:34:49 UTC 2019<br />
#############################################################<br />
<br />
emonSD version: emonSD-30Oct18<br />
<br />
emonSD base image check passed...continue update<br />
<br />
#############################################################<br />
I2C LCD DETECTED Ox3f<br />
git pull /home/pi/emonpi<br />
  enable-ssh<br />
  enable-ssh-button<br />
  gpiozero-enable-ssh<br />
  greeebs-patch-1<br />
* master<br />
  python-systemd-servicerunner<br />
On branch master<br />
Your branch is behind 'origin/master' by 145 commits, and can be fast-forwarded.<br />
  (use "git pull" to update your local branch)<br />
Changes not staged for commit:<br />
  (use "git add ..." to update what will be committed)<br />
  (use "git checkout -- ..." to discard changes in working directory)<br />
<br />
	modified:   lcd/emonPiLCD.cfg<br />
<br />
no changes added to commit (use "git add" and/or "git commit -a")<br />
error: Your local changes to the following files would be overwritten by merge:<br />
	lcd/emonPiLCD.cfg<br />
Please commit your changes or stash them before you merge.<br />
Aborting<br />
Updating bc7ebd1..0a0c6ba<br />
git pull /home/pi/RFM2Pi<br />
* master<br />
On branch master<br />
Your branch is up-to-date with 'origin/master'.<br />
nothing to commit, working tree clean<br />
Already up-to-date.<br />
git pull /home/pi/emonhub<br />
* emon-pi<br />
On branch emon-pi<br />
Your branch is up-to-date with 'origin/emon-pi'.<br />
Changes not staged for commit:<br />
  (use "git add ..." to update what will be committed)<br />
  (use "git checkout -- ..." to discard changes in working directory)<br />
<br />
	modified:   conf/default/emonhub<br />
	modified:   service/emonhub.service<br />
<br />
no changes added to commit (use "git add" and/or "git commit -a")<br />
Already up-to-date.<br />
git pull /home/pi/usefulscripts<br />
* master<br />
On branch master<br />
Your branch is up-to-date with 'origin/master'.<br />
nothing to commit, working tree clean<br />
Already up-to-date.<br />
git pull /home/pi/huawei-hilink-status<br />
* master<br />
On branch master<br />
Your branch is up-to-date with 'origin/master'.<br />
nothing to commit, working tree clean<br />
Already up-to-date.<br />
<br />
Start emonPi Atmega328 firmware update:<br />
<br />
=================================<br />
EmonPi update started<br />
=================================<br />
<br />
EUID: 1000<br />
<br />
Requirement already up-to-date: paho-mqtt in /usr/local/lib/python2.7/dist-packages<br />
Hit:1 http://archive.raspberrypi.org/debian stretch InRelease<br />
Get:2 http://raspbian.raspberrypi.org/raspbian stretch InRelease [15.0 kB]<br />
Fetched 15.0 kB in 1s (9,090 B/s)<br />
Reading package lists...<br />
Reading package lists...<br />
Building dependency tree...<br />
Reading state information...<br />
rng-tools is already the newest version (2-unofficial-mt.14-1).<br />
python-gpiozero is already the newest version (1.5.0).<br />
python-rpi.gpio is already the newest version (0.6.5~stretch-1).<br />
0 upgraded, 0 newly installed, 0 to remove and 118 not upgraded.<br />
Start ATmega328 serial upload using avrdude with latest.hex<br />
Discrete Sampling<br />
avrdude -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex<br />
avrdude-original: Using autoreset DTR on GPIO Pin 7<br />
<br />
avrdude-original: AVR device initialized and ready to accept instructions<br />
<br />
Reading | ################################################## | 100% 0.00s<br />
<br />
avrdude-original: Device signature = 0x1e950f (probably m328p)<br />
avrdude-original: NOTE: "flash" memory has been specified, an erase cycle will be performed<br />
                  To disable this feature, specify the -D option.<br />
avrdude-original: erasing chip<br />
avrdude-original: reading input file "/home/pi/emonpi/firmware/compiled/latest.hex"<br />
avrdude-original: input file /home/pi/emonpi/firmware/compiled/latest.hex auto detected as Intel Hex<br />
avrdude-original: writing flash (17886 bytes):<br />
<br />
Writing | avrdude-original: stk500_loadaddr(): (a) protocol error, expect=0x14, resp=0x1c<br />
<br />
avrdude-original: stk500_paged_write(): (a) protocol error, expect=0x14, resp=0xe0<br />
avrdude-original: stk500_cmd(): programmer is out of sync<br />
avrdude-original: stk500_cmd(): programmer is out of sync<br />
avrdude-original: stk500_cmd(): programmer is out of sync<br />
[snipped a further 100 lines]
avrdude-original: stk500_recv(): programmer is not responding<br />
avrdude-original: stk500_recv(): programmer is not responding<br />
avrdude-original: stk500_recv(): programmer is not responding<br />
[snipped a further 261 lines which stop when device was shutdown]

The after a reboot I updated again and this appeared to be more successful, but did not resolve the issues with Emonhub.

> #############################################################
> Starting emonPi Update >
> via service-runner-update.sh
> Service Runner update script V1.1.1
> EUID: 1000
> Argument: emonhub
> Tue 18 Jun 12:10:07 UTC 2019
> #############################################################
> 
> emonSD version: emonSD-30Oct18
> 
> emonSD base image check passed...continue update
> 
> #############################################################
> I2C LCD DETECTED Ox3f
> git pull /home/pi/emonpi
>   enable-ssh
>   enable-ssh-button
>   gpiozero-enable-ssh
>   greeebs-patch-1
> * master
>   python-systemd-servicerunner
> On branch master
> Your branch is behind 'origin/master' by 145 commits, and can be fast-forwarded.
>   (use "git pull" to update your local branch)
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git checkout -- <file>..." to discard changes in working directory)
> 
> 	modified:   lcd/emonPiLCD.cfg
> 
> no changes added to commit (use "git add" and/or "git commit -a")
> error: Your local changes to the following files would be overwritten by merge:
> 	lcd/emonPiLCD.cfg
> Please commit your changes or stash them before you merge.
> Aborting
> Updating bc7ebd1..0a0c6ba
> git pull /home/pi/RFM2Pi
> * master
> On branch master
> Your branch is up-to-date with 'origin/master'.
> nothing to commit, working tree clean
> Already up-to-date.
> git pull /home/pi/emonhub
> * emon-pi
> On branch emon-pi
> Your branch is up-to-date with 'origin/emon-pi'.
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git checkout -- <file>..." to discard changes in working directory)
> 
> 	modified:   conf/default/emonhub
> 	modified:   service/emonhub.service
> 
> no changes added to commit (use "git add" and/or "git commit -a")
> Already up-to-date.
> git pull /home/pi/usefulscripts
> * master
> On branch master
> Your branch is up-to-date with 'origin/master'.
> nothing to commit, working tree clean
> Already up-to-date.
> git pull /home/pi/huawei-hilink-status
> * master
> On branch master
> Your branch is up-to-date with 'origin/master'.
> nothing to commit, working tree clean
> Already up-to-date.
> 
> Start emonPi Atmega328 firmware update:
> 
> =================================
> EmonPi update started
> =================================
> 
> EUID: 1000
> 
> Requirement already up-to-date: paho-mqtt in /usr/local/lib/python2.7/dist-packages
> Hit:1 http://archive.raspberrypi.org/debian stretch InRelease
> Hit:2 http://raspbian.raspberrypi.org/raspbian stretch InRelease
> Reading package lists...
> Reading package lists...
> Building dependency tree...
> Reading state information...
> rng-tools is already the newest version (2-unofficial-mt.14-1).
> python-gpiozero is already the newest version (1.5.0).
> python-rpi.gpio is already the newest version (0.6.5~stretch-1).
> 0 upgraded, 0 newly installed, 0 to remove and 118 not upgraded.
> Start ATmega328 serial upload using avrdude with latest.hex
> Discrete Sampling
> avrdude -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex
> avrdude-original: Using autoreset DTR on GPIO Pin 7
> 
> avrdude-original: AVR device initialized and ready to accept instructions
> 
> Reading | ################################################## | 100% 0.00s
> 
> avrdude-original: Device signature = 0x1e950f (probably m328p)
> avrdude-original: NOTE: "flash" memory has been specified, an erase cycle will be performed
>                   To disable this feature, specify the -D option.
> avrdude-original: erasing chip
> avrdude-original: reading input file "/home/pi/emonpi/firmware/compiled/latest.hex"
> avrdude-original: input file /home/pi/emonpi/firmware/compiled/latest.hex auto detected as Intel Hex
> avrdude-original: writing flash (17886 bytes):
> 
> Writing | ################################################## | 100% 2.48s
> 
> avrdude-original: 17886 bytes of flash written
> avrdude-original: verifying flash memory against /home/pi/emonpi/firmware/compiled/latest.hex:
> avrdude-original: load data flash data from input file /home/pi/emonpi/firmware/compiled/latest.hex:
> avrdude-original: input file /home/pi/emonpi/firmware/compiled/latest.hex auto detected as Intel Hex
> avrdude-original: input file /home/pi/emonpi/firmware/compiled/latest.hex contains 17886 bytes
> avrdude-original: reading on-chip flash data:
> 
> Reading | ################################################## | 100% 1.86s
> 
> avrdude-original: verifying ...
> avrdude-original: 17886 bytes of flash verified
> strace: |autoreset: Broken pipe
> strace: |autoreset: Broken pipe
> strace: |autoreset: Broken pipe
> strace: |autoreset: Broken pipe
> strace: |autoreset: Broken pipe
> 
> avrdude-original done.  Thank you.
> 
> strace: |autoreset: Broken pipe
> 
> 
> Start emonhub update script:
> 
> =================================
> emonHub update started
> =================================
> emonhub.service already installed
> 
> Running emonhub automatic node addition script
> EUID: 1000
> [[5]]
> Node 5 already present
> [[6]]
> Node 6 already present
> [[7]]
> Node 7 already present
> [[8]]
> Node 8 already present
> [[9]]
> Node 9 already present
> [[10]]
> Node 10 already present
> [[11]]
> Node 11 already present
> [[12]]
> Node 12 already present
> [[13]]
> Node 13 already present
> [[14]]
> Node 14 already present
> [[19]]
> Node 19 already present
> [[20]]
> Node 20 already present
> [[21]]
> Node 21 already present
> [[22]]
> Node 22 already present
> [[23]]
> Node 23 already present
> [[24]]
> Node 24 already present
> [[25]]
> Node 25 already present
> [[26]]
> Node 26 already present
> 
> Start emoncms update:
> 
> ===========================================
> Emoncms update started
> Emoncms update script V1.2.1 (6th Feb 2019)
> Tue 18 Jun 12:10:39 UTC 2019
> ===========================================
> 
> username: pi
> 
> Checking environment:
> - User directory /home/pi found
> - pi user and pi user directory found
> - emonSD version: emonSD-30Oct18
> - emoncms directory: /var/www/emoncms
> 
> EUID: 1000
> /home/pi/emonpi/emonhub-sudoers: parsed OK
> 
> emonhub service control sudoers entry installed
> www-data : www-data video systemd-journal
> 
> current settings.php md5: 0e4803a641b7784ed23edf87506ce761
> Default settings.php md5: 0e4803a641b7784ed23edf87506ce761
> 
> Checking status of /var/www/emoncms git repository
> - git branch: stable
> - no local changes
> - running: git pull origin stable
> 
> From https://github.com/emoncms/emoncms
>  * branch              stable     -> FETCH_HEAD
> Already up-to-date.
> 
> NEW default settings.php md5: 0e4803a641b7784ed23edf87506ce761
> settings.php has NOT been user modifed
> settings.php not updated
> 
> ==========================================
> UPDATING EMONCMS MODULES
> ==========================================
> 
> ------------------------------------------
> Updating /var/www/emoncms/Modules/app module
> ------------------------------------------
> - git branch: stable
> - git tags: 1.2.0-108-g6e5016a
> - no local changes
> - running: git pull origin stable
> 
> From https://github.com/emoncms/app
>  * branch            stable     -> FETCH_HEAD
> Already up-to-date.
> 
> ------------------------------------------
> Updating /var/www/emoncms/Modules/config module
> ------------------------------------------
> - git branch: stable
> - git tags: 1.0.0-63-g1d9238e
> - no local changes
> - running: git pull origin stable
> 
> From https://github.com/emoncms/config
>  * branch            stable     -> FETCH_HEAD
> Already up-to-date.
> 
> ------------------------------------------
> Updating /var/www/emoncms/Modules/dashboard module
> ------------------------------------------
> - git branch: stable
> - git tags: 1.3.1-115-gabd1a0b
> - no local changes
> - running: git pull origin stable
> 
> From https://github.com/emoncms/dashboard
>  * branch            stable     -> FETCH_HEAD
> Already up-to-date.
> 
> ------------------------------------------
> Updating /var/www/emoncms/Modules/device module
> ------------------------------------------
> - git branch: master
> - git tags: 1.1.1-48-g2ea6e74
> - no local changes
> - running: git pull origin master
> 
> From https://github.com/emoncms/device
>  * branch            master     -> FETCH_HEAD
> Already up-to-date.
> 
> ------------------------------------------
> Updating /var/www/emoncms/Modules/graph module
> ------------------------------------------
> - git branch: stable
> - git tags: 1.2.1-151-gc7f4ea9
> - no local changes
> - running: git pull origin stable
> 
> From https://github.com/emoncms/graph
>  * branch            stable     -> FETCH_HEAD
> Already up-to-date.
> 
> ------------------------------------------
> Updating /var/www/emoncms/Modules/wifi module
> ------------------------------------------
> - git branch: stable
> - git tags: 1.2.0-39-gb67783d
> - no local changes
> - running: git pull origin stable
> 
> From https://github.com/emoncms/wifi
>  * branch            stable     -> FETCH_HEAD
> Already up-to-date.
> 
> ------------------------------------------
> Updating postprocess module
> ------------------------------------------
> - git branch: emonpi
> fatal: No names found, cannot describe anything.
> - git tags: 
> - no local changes
> - running: git pull origin emonpi
> 
> From https://github.com/emoncms/postprocess
>  * branch            emonpi     -> FETCH_HEAD
> Already up-to-date.
> 
> ------------------------------------------
> Updating sync module
> ------------------------------------------
> - git branch: master
> - git tags: 1.0.0-23-gc5eb931
> - no local changes
> - running: git pull origin master
> 
> From https://github.com/emoncms/sync
>  * branch            master     -> FETCH_HEAD
> Already up-to-date.
> 
> ------------------------------------------
> Updating backup module
> ------------------------------------------
> - git branch: stable
> - git tags: 1.1.5-54-g67a1f1d
> - no local changes
> - running: git pull origin stable
> 
> From https://github.com/emoncms/backup
>  * branch            stable     -> FETCH_HEAD
> Already up-to-date.
> 
> ==========================================
> AUTO INSTALL EMONCMS MODULES
> ==========================================
> - graph module already installed
> - device module already installed
> - postprocess module already installed
> - sync module already installed
> - backup module already installed
> 
> 
> ==========================================
> SERVICES
> ==========================================
> emoncms_mqtt.service already installed
> feedwriter.service already installed
> service-runner.service already installed
> ------------------------------------------
> Update Emoncms database
> []
> 
> Restarting Services...
> - sudo systemctl restart feedwriter.service
> --- ActiveState=active ---
> - sudo systemctl restart emoncms_mqtt.service
> --- ActiveState=active ---
> - sudo systemctl restart emonhub.service
> --- ActiveState=active ---
> Installing emoncms logrotate...
> 
> Now setting up Logrotate...
> Backing up old logrotate configuration...
> Linked to new logrotate configuration...
> Backing up old logrotate cron job...
> mv: cannot stat '/etc/cron.daily/logrotate': No such file or directory
> Linked to new logrotate cron job...
> 
> Completed
> 
> 
> Completed
> 
> Running logrotate...
> 
> set log rotate config owner to root
> 
> ------------------------------------------
> Emoncms update script complete
> ------------------------------------------
> 
> 
> Starting emonPi LCD service..
> 
> Tue 18 Jun 12:11:01 UTC 2019
> 
> 
> ...................
> emonPi update done
> restarting service-runner
> service-runner: no process found

Hello @julesb the update shows local changes to both the emonpi repo:

modified:   lcd/emonPiLCD.cfg

and the emonhub repo:

modified:   conf/default/emonhub
modified:   service/emonhub.service

Did you make these changes? If you didnt make the changes, or dont need to keep them you can clear them manually like this:

cd /home/pi/emonpi
git checkout .

cd /home/pi/emonhub
git checkout .

If you then run the update again and repost the log it would be interesting to see if that fixes it.

I will have a look later in the week when I have more time. The changes to the emonhub repo are probably mine, but I did not make any changes to the emonPiLCD.cfg file.

1 Like

This put backup back in my emoncms menu.

1 Like

Hi

Running:

cd /home/pi/backup git pull ln -sf /home/pi/backup/backup-module /var/www/emoncms/Modules/backup

Recovered backup menu but is showing this:

Error: missing backup config.cfg

Any suggestion how to recover this features as 9.9.5 version=

Thanks

Missatge de Sam via Community [email protected] del dia dc., 26 de juny 2019 a les 13:25:

Hi

Also detected if you reboot emoncms then ln has dissapeared and backup module not showed, and also since latest upgrade emoncms.log has dissapperared, if you create folder and log, works, but after a reboot is lost this configuration and manual regenerate is required

Thanks

Missatge de klxout [email protected] del dia ds., 6 de jul. 2019 a les 22:06:

Yes, I just actually tried backup and same issue:

**Error:** missing backup config.cfg

Has happened to me now @TrystanLea ā€¦

Donā€™t suppose this is an easy fix for this - havenā€™t backed up for a while now? Not super keen to do a full update if I donā€™t have to given issues so far post-update.

Hi Sam, itā€™s not clear (to me anyway) what issue(s) you have, are you now saying the only issue you have is " Error: missing backup config.cfg"?

Your previous post

Quotes 2 passages about emonhub failing, is emonhub now ok?

And your post prior to that

whilst it quotes **Error:** missing backup config.cfg it follows a post by klx (replying to you) saying that the fix to the symlinks isnā€™t sticking and after a reboot the backup link disappears again. Do you also see this problem?

If it is just a simple case of the config.cfg file not found I think @TrystanLea would probably suggest you just re-run the update, the config.cfg file is created if missing by this part of the update scripts

if you are not comfortable with that approach you can manually create the file using default.config.cfg as a starting point and edit the paths accordingly using the code above as a guide. That is how itā€™s done when manually installing the backup-module (GitHub - emoncms/backup: emoncms backup module). I donā€™t have any experience with the backup module so I can only offer limited help, but if you arenā€™t able to create the config.cfg file yourself Iā€™m sure we can work it out or someone else might offer up a copy of theirs?

Iā€™m pretty sure $usrdir is /home/pi on all emonSD images thus far so most if not all the paths and settings can be read from the update code above.

If you do have other issues can you please give us more details?

[edit] There is also the question of why this happened in the first place if only to predict or head off it happening when you next update. Can you attach a copy of your full update log? The likelihood is there maybe a reason it didnā€™t originally complete successfully and if so, we should try and fix the cause rather than just the results to avoid further problems.

1 Like

Sorry Paul, I probably should have done a summary and not been so quote post lazy!

  • RE: emonhub issues, I got that working again from your advice
  • I lost the original 2 x updates logs i ran when all the issues started moving to new updates
  • Realised I had no ā€œBackupā€ option in the dropdown cog
  • Ran @TrystanLeaā€™s suggestion (cd /home/pi/backup git pull ln -sf /home/pi/backup/backup-module /var/www/emoncms/Modules/backup) restored the menu item
  • But when I click ā€œBackupā€ I get that config error - Error: missing backup config.cfg

I just tried re-running the Emoncms update from Admin page, log:
emonpiupdate.txt (6.7 KB)

I havenā€™t read through the whole log yet but right at the top there is

git pull /home/pi/emonpi
  enable-ssh
  enable-ssh-button
  gpiozero-enable-ssh
  greeebs-patch-1
* master
  python-systemd-servicerunner
On branch master
Your branch is behind 'origin/master' by 79 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   lcd/emonPiLCD.cfg

no changes added to commit (use "git add" and/or "git commit -a")
error: Your local changes to the following files would be overwritten by merge:
	lcd/emonPiLCD.cfg
Please commit your changes or stash them before you merge.
Aborting

and the emonpi repo is the one with all the updater code so itā€™s the foundation for a successful update (or not!).

Do you recall what changes you made to emonpi/lcd/emonPiLCD.cfg ? Chances are it was the backlight timeout, this error has popped up a few times since this commit

anyone that manually edited that value prior to this commit will have had all updates to the emonpi repo blocked since that commit was merged.

to fix you can run

git -C /home/pi/emonpi checkout -f -B master

(or any of many alternative commands) to reset the repo changes. Thatā€™s assuming you are ok with losing any manual changes youā€™ve made, obviously the timeout will be 300 after the update but you can change it again if you so choose.

Then run the updater again. Then retry the backup link :crossed_fingers:

1 Like

Nailed it. Working as expected. Yes had updated LCD to ensure LCD wasnā€™t perma-onā€¦

1 Like