EmonPi stuck in Raspberry Pi booting

Elbow · 8 December 2019 07:08

OK - thanks. - Turned out easier than expected.

I made a new card with the Oct2019 version.

Restoring my backup brought back my feeds and graphs and what not. I had duplication of devices so I cleaned up.

My backup tarball only had a database dump and a few files - didn’t include the phpfine and phptimeseries directories.

So I copied them from /home/pi/data I think it was and guessed that they needed to go into the new /var/opt/emoncms.

Booting up after that there was a quite long time when the display showed Updating, but once done I seem to be back in business with my history intact.

Thanks for the help.

For others like me who haven’t dug into the detailed working - it seems that the ordinary update process in the web interface just updates the emoncms code itself and doesn’t update the underlying raspbian? Is that right? If so, what is the “proper” way to keep the OS up to date?

Thanks.
Steve

My backup did not include the I then shut down and copied the php… directories onto the 3rd partition. On booting up there was

borpin · 8 December 2019 11:38

We do try.

Which is odd.

Which is what one would expect - when you update an application on Windows, does that update the Windows installation as well?

Google is your friend Raspberry Pi Documentation - Raspberry Pi OS

Did you take a fresh backup?

Bill.Thomson · 8 December 2019 11:40

I recall reading a thread some time back where updating the OS caused some things to break.
Is that still the case, or has that become a non-issue? i.e was that a problem with Jessie but
no longer an issue with Stretch or Buster?

Bill.Thomson · 8 December 2019 11:53

borpin · 8 December 2019 12:14

I have a feeling there was an issue with Stretch and MQTT at one point, but I’m reasonably sure it is fine now.

Bill.Thomson · 8 December 2019 12:17

Elbow · 14 January 2020 13:27

Well - in case the perhaps naive expectation of a user/customer is relevant - I bought it as an piece-of-tin appliance so mentally did not distinguish the different parts of the firmware. It came with the SD card included and as a packaged turnkey device. So when I updated through the GUI I certainly assumed that the appropriate OS packages et al would be updated and I’d receive some sort of notification if I was on a dead-end software train.

Backups on my EmonPi failed since it runs out of space. I’m not sure which partition it makes the tarball on.

My paritions look like so:

Actually logging in I see the root is full:

pi@emonpi:~ $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        4114268 3913552         0 100% /
devtmpfs          494908       0    494908   0% /dev
tmpfs             499516       0    499516   0% /dev/shm
tmpfs             499516   31628    467888   7% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             499516       0    499516   0% /sys/fs/cgroup
tmpfs              30720      16     30704   1% /tmp
tmpfs               1024       4      1020   1% /var/lib/php/sessions
tmpfs               1024       0      1024   0% /var/tmp
/dev/mmcblk0p3  10464739  996317   8936710  11% /var/opt/emoncms
/dev/mmcblk0p1    258095   53033    205063  21% /boot
log2ram            51200   13412     37788  27% /var/log
tmpfs              99900       0     99900   0% /run/user/1000

I see that the backup is made in /opt/openergymonitor/data which is on the small root partition rather than on the /var/opt/emoncms mount

The failed backup left a 632M tar file in that directory:

root@emonpi:/opt/openenergymonitor/data# ls -lh
total 635M
-rw-r--r-- 1 root     root    0 Oct 17 10:42 dhcpd.leases
-rw-r--r-- 1 pi       pi   632M Jan 14 13:14 emoncms-backup-2020-01-14.tar
-rw-r--r-- 1 pi       pi   3.0M Jan 14 13:11 emoncms.sql
drwxr-xr-x 2 pi       pi   4.0K Dec  7 15:46 import
drwxr-xr-x 2 www-data pi   4.0K Dec  7 15:46 uploads

It seems like a bad idea to generate the backup on the root partition and in the event of a failure the cleanup should probably be improved.

How do I make a successful backup?

Steve

Elbow · 14 January 2020 13:45

Hmmm - seems like I’m in space trouble on the root partition generally.

I deleted that failed backup tarball and my root is still

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        4114268 3266496    645028  84% /

4G space in total. 3.26G used.

1G in /var:
127M in /var/lib/mysql,
265M in /var/lib/openhab2 (most in /var/lib/openhab2/tmp/kar/openhab-addons-2.5.0) ,
408M in /var/cache/apt,
1.67G in /usr
351M in /lib
98M /opt
(etc)

Is my system unusually large? - I do have 240+ feeds.

Can I do something with /var/cache/apt and /var/lib/openhab2/tmp/kar/openhab-addons-2.5.0 ?

borpin · 14 January 2020 13:48

Yes, there is a change being made in GitHub to correct that. Moving backup export and uploads location to /var/opt/emoncms/backup · Issue #51 · emoncms/backup · GitHub

I’m not at home but you need to change the location in the config.cfg under the backup module, but I cannot check where that is.

I can check later.

I am surprised just that file is filling the rootfs. I posted here a command that will find the largest files on your system.

I’d say yes to that! Does also depend on how often you are feeding the data. if 240 @ 10s then yes definitely!

Elbow · 14 January 2020 13:57

I did

mkdir /var/opt/emoncms/backups
chown pi:pi /var/opt/emoncms/backups

Then in /opt/emoncms/modules/backup/config.cfg I made it so:

# Destination location of exported backup .tar.gz - Must be writable
#backup_location="/opt/openenergymonitor/data"
backup_location="/var/opt/emoncms/backups"

and I see its writing there and hopefully will complete.

I did post a quick look via du to see where the space was being used.

borpin · 14 January 2020 14:11

The command I pointed to will tell you which files, not directories are the big hitters. Still think something amiss to be that full.

Elbow · 14 January 2020 15:08

Here’s files >25M that are on the root filesystem:

root@emonpi:/# find . -xdev -size +25M -ls
     2040  30284 -rw-rw-r--   1 root     users    31008729 Jun 24  2019 ./opt/vc/src/hello_pi/hello_video/test.h264
    60928 185752 -rw-r--r--   1 openhab  openhab  190203584 Dec 15 22:56 ./usr/share/openhab2/addons/openhab-addons-2.5.0.kar
     5761  38656 -rwxr-xr-x   1 root     root      39583268 Dec 17 06:39 ./usr/bin/node
    58811  63528 -rw-r--r--   1 root     root      65050102 Mar 29  2019 ./usr/lib/jvm/java-8-openjdk-armhf/jre/lib/rt.jar
    59576  55348 -rw-r--r--   1 root     root      56674300 Apr 18  2019 ./usr/lib/arm-linux-gnueabihf/libLLVM-8.so.1
    14620  26552 -rw-r--r--   1 root     root      27186324 Jan 23  2019 ./usr/lib/arm-linux-gnueabihf/libicudata.so.63.1
     5753  67448 -rw-r--r--   1 root     root      69065345 Dec 17 16:55 ./var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_buster_main_binary-armhf_Packages
    61225  30400 -rw-r--r--   1 openhab  openhab   31127195 Dec 18 12:18 ./var/lib/openhab2/tmp/kar/openhab-addons-2.5.0/org/openhab/addons/bundles/org.openhab.voice.marytts/2.5.0/org.openhab.voice.marytts-2.5.0.jar
   133157  49152 -rw-rw----   1 mysql    mysql     50331648 Jan  5 15:06 ./var/lib/mysql/ib_logfile0
   133158  49152 -rw-rw----   1 mysql    mysql     50331648 Oct 17 10:18 ./var/lib/mysql/ib_logfile1
      895 102400 -rw-------   1 root     root     104857600 Sep 26 01:24 ./var/swap
     5757  29992 -rw-r--r--   1 root     root      30711109 Dec 18 11:37 ./var/cache/apt/srcpkgcache.bin
    60910 185528 -rw-r--r--   1 root     root     189978934 Dec 15 22:58 ./var/cache/apt/archives/openhab2-addons_2.5.0-1_all.deb
     5774  80668 -rw-r--r--   1 root     root      82600272 Dec 15 22:59 ./var/cache/apt/archives/openhab2_2.5.0-1_all.deb
     1205  30144 -rw-r--r--   1 root     root      30863584 Apr 11  2019 ./var/cache/apt/archives/libpython2.7-dev_2.7.16-2_armhf.deb
     9435  30036 -rw-r--r--   1 root     root      30755458 Dec 18 12:18 ./var/cache/apt/pkgcache.bin

Two biggest files (>100M) are:

    60928 185752 -rw-r--r--   1 openhab  openhab  190203584 Dec 15 22:56 ./usr/share/openhab2/addons/openhab-addons-2.5.0.kar
    60910 185528 -rw-r--r--   1 root     root     189978934 Dec 15 22:58 ./var/cache/apt/archives/openhab2-addons_2.5.0-1_all.deb

I installed Openhab as per https://www.openhab.org/docs/installation/linux.html#installation which was linked to from https://guide.openenergymonitor.org/integrations/openhab/.

Steve

borpin · 14 January 2020 17:47

Ah, well that’ll explain it.

And then you modified it…

Would you expect it to update the OpenHab install as well?

Clearly you are not running a vanilla EmonPi, so that makes a big difference. It is a bit like applying an ECU mod to your car.

Elbow · 14 January 2020 19:06

Not sure why you are being unpleasant to a customer who bought 100s of pounds worth of hardware from the project.

On the page which says “openhab used to be part of emoncms but is no longer included - here’s how to install it” perhaps a warning could be added to help others that says that installing it may fill up your root file system and cause operational problems for your emonpi. Or to make it clear you are going off-piste and are on your own.

I must say that I had forgotten that I installed Openhab.

Since the install instructions pointed to were openhab instructions I would not have assumed that the emonpi would update it.

Could you explaining for me though - on /admin/view page in the “Updates” section the default offer is full update with the description “OS, Packages, Emonhib, Emoncms & Firmware (if new version)”.

Does this include the OS? Seems to say that it does.

If you pop it down there is no separate option for OS.

What is included in “Packages”. Apologies if this is documented somewhere.

borpin · 15 January 2020 09:29

I am simply pointing out that assumptions are the mother of all &**%$ ups . Equally, you cannot say ‘it should work as I bought a complete system’ when you have then done modifications to that ‘complete’ system. Yes, the advantage of it is that you can change things but that comes with risks. This is also a community forum, not a commercial helpdesk. Majority of those who offer help do so in their spare time.

The OS bit may be a hang over from previous update scripts. I think the OS update part was dropped as it caused issues at one point with MQTT server versions.

A number of parts of the EmonCMS system are separate GitHub Repositories. This should probably be referred to as ‘Modules’ rather than ‘Packages’. At one time ‘packages’ like OpenHAB and Node-Red were pre-installed, but this became problematic so the practice was dropped and installing such things on the EmonPi comes with a caveat emptor . Your best solution is to run such things on a separate Pi.

Elbow · 15 January 2020 09:39

OK thanks, both of us a little frustrated I guess. I’ve been both sides of the table.

Suggestions for the project for what its worth:

If update doesn’t update the OS then remove “OS,” from the description
More space for / in the partitioning scheme, or move more stuff to other partitions
Update places like openHAB - Guide | OpenEnergyMonitor to include a caution for naive users such as myself.
Failed backup should clean up work files (especially the tarball that is later compressed).

Thanks to you and all who work on the project.

borpin · 15 January 2020 09:45

@TrystanLea - All fair comments re docs.

On space for the root partition, that has been discussed, but there is a desire to maximise the space for data as that is the primary task of the EmonPi, so not an unreasonable design decision. Howver, the desire is to increase it for cards above 16GB - something I need to find time to do. Not putting the backup on the data partition was an error though.

pb66 · 15 January 2020 12:13

IIRC it’s the other way round, it previously wasn’t doing apt-get update/upgrade until Trystan changed the update script to modularize it and the description was written as “update all” does in fact now do an apt-get etc where is has not previously (hence all the previous issues caused when users ran their own independent apt-get updates).

borpin · 15 January 2020 12:32

I did check the scripts before I said that. It is in the install script in EmonScripts, but not the update scripts.

borpin · 15 January 2020 17:35

4 posts were split to a new topic: Should apt-upgrade be included in Update Script