Hi Glyn,
I have just installed the latest SD image. when i ran sudo emonSDexpand the emonpi reboot and on the screen I got
Paul
Looks good to me, try running $ df -h after a reboot to check it’s expanded
I would recommend using the backup module to export your data then re-importing once you have booted the new image, see:
https://guide.openenergymonitor.org/setup/import/
I would recommend downloading and flashing a fresh version of the latest image with Etcher then importing your data. Using the backup module. See above.
Just tried the new image, but having a few issues if anyone can help. Here are my notes and findings:
I used Etcher to load the image to a 32GB SD card and then loaded it into the rPi3.
Initial problem was wifi. My network is 10.0.0.* so wifi wouldn’t connect. Need to change details in wpa_supplicant.conf, but it isn’t in /data anymore, rather wpa_supplicant.conf is now in /etc. Added the SSID and passkey and wifi now works.
problem with expanding the file system - error message:
/usr/local/sbin/emonSDexpand: line 29: rpi-rw: command not found
I had to grow the filesystem using parted
some error messages on boot (lines from log shown a bit lower down):
chown invalid user: openhab:openhab and dataplicity
|ERROR|feedwriter.php|Can't connect to database:Connection refused
first bootupdate: line 24 permission denied
/home/pi/emonpi/./firstbootupdate: line 24: /home/pi/data/emonpiupdate.log: Permission denied
[ 10.540898] rc.local[603]: chown: invalid user: ‘openhab:openhab’
[ 10.553382] rc.local[603]: chown: invalid user: ‘dataplicity:dataplicity’
[ 14.132771] rc.local[603]: Job for mariadb.service failed because the control process exited with error code.
[ 14.134891] rc.local[603]: See "systemctl status mariadb.service" and "journalctl -xe" for details.
[ 17.047527] rc.local[603]: Job for emonhub.service failed because the control process exited with error code.
[ 17.051489] rc.local[603]: See "systemctl status emonhub.service" and "journalctl -xe" for details.
so had to go back to the old image - any thoughts on what I ca do next?
I’ve just tested using emonSDexpand and it worked fine on the new image, this error should just be ignored. Its interesting emonDexpand did not work for you. It will take about 30-40min then require a reboot.
Does mariadb eventually start correctly? Did you try and loadup Emoncms? I think that error is just due to mariadb starting up before the logfiles have been created in tmpfs /var/log, this is nothing to worry about since the service should then restart and run correctly after log files have been created.
The permission denied error for /home/pi/data/emonpiupdate.log is strange, I’ll take another look but I’ve not seen this happen on the new image before. Have you any idea how emonpiupdate.log has changed permissions?
Update: just double checked, and I’ve been unable to replicate the permissions issue emonpiupdate.log should be owned by user pi:pi
pi@emonpi:~ $ ls -la data/emonpiupdate.log
-rw-rw-rw- 1 pi pi 13206 Nov 13 15:07 data/emonpiupdate.log
You can fix this by:
sudo chown pi:pi /home/pi/data/emonpiupdate.log
It would be interesting to try and figure out why the permissions changed, could this have happened during the expansion process? Here are the default permissions for the ~/data
pi@emonpi:~/data $ ls -la
total 48
drwxrwxrwx 9 pi pi 1024 Nov 13 15:12 .
drwxr-xr-x 20 pi pi 4096 Oct 31 00:59 ..
drwxrwxrwx 3 root root 1024 Oct 31 01:01 @eaDir
-rw-rw-r-- 1 pi www-data 0 Oct 31 00:59 emoncms.conf
-rw-r--r-- 1 pi pi 1718 Nov 13 15:12 emoncms-wifiscan.log
-rw-rw-rw- 1 pi www-data 7463 Oct 31 00:59 emonhub.conf
-rw-rw-rw- 1 pi pi 13206 Nov 13 15:07 emonpiupdate.log
drwx------ 2 pi pi 12288 Aug 16 15:29 lost+found
drwxrwxrwx 5 mysql mysql 1024 Nov 13 15:08 mysql
drwxr-xr-x 2 www-data root 1024 Oct 31 00:59 phpfina
drwxr-xr-x 2 www-data root 1024 Oct 31 00:59 phptimeseries
drwxrwxrwt 2 root root 1024 Oct 18 17:33 @tmp
drwxr-xr-x 2 www-data root 1024 Oct 22 01:02 uploads
-rw-r--r-- 1 root root 137 Nov 13 15:20 wificheck.log
To anyone else following this thread, we sell ready expanded 16GB cards pre-loaded SD cards with the new image via our store:
https://openenergymonitor.com/emonsd-pre-loaded-raspberry-pi-sd-card/
I had a similar issue a while ago and raised this issue - Log file permissions and ownership · Issue #806 · emoncms/emoncms · GitHub I thought the issue might be logrotate but could not prove it.
@icenov could you do a ls -la on the log file directory and the parent directory please.
The issue your referred to is a separate issue. emoncms.log is handled via a different service and logrotate does not take effect to directories outside of /var/log
1 Like
Thanks for quick reply - it is now mostly up, after a couple of hangups and reboots. Permissions look correct:
drwxr-xr-x 13 root root 580 Nov 14 10:56 .
drwxr-xr-x 12 root root 4096 Aug 2 20:53 ..
drwxr-xr-x 2 root adm 100 Nov 14 10:06 apache2
drwxr-xr-x 2 root root 100 Nov 14 10:56 apt
-rw-r----- 1 root adm 10782 Nov 14 11:00 auth.log
-rw-r--r-- 1 root root 6039 Nov 14 10:06 boot.log
-rw------- 1 root utmp 0 Nov 14 10:06 btmp
-rw-r----- 1 root adm 33060 Nov 14 10:57 daemon.log
-rw-r----- 1 root adm 1177 Nov 14 10:06 debug
-rw-r--r-- 1 root root 3123 Nov 14 10:56 dpkg.log
-rw-rw-rw- 1 root root 372 Nov 14 10:06 emoncms.log
drwxr-xr-x 2 emonhub root 60 Nov 14 10:06 emonhub
drwxr-xr-x 2 pi root 60 Nov 14 10:06 emonpilcd
-rw-r----- 1 root adm 32534 Nov 14 11:00 kern.log
drwxr-xr-x 2 pi pi 40 Nov 14 10:06 logrotate
-rw-r----- 1 root adm 28204 Nov 14 11:00 messages
drwxr-xr-x 2 mosquitto mosquitto 60 Nov 14 10:06 mosquitto
-rw-rw-rw- 1 root root 0 Nov 14 10:06 mqtt_input.log
drwxr-xr-x 2 mysql adm 60 Nov 14 10:06 mysql
-rw-rw-rw- 1 root root 0 Nov 14 10:06 mysql.log
drwxr-xr-x 2 ntp ntp 40 Feb 15 2018 ntpstats
-rw-rw-rw- 1 root root 0 Nov 14 10:06 ntp_update.log
drwxr-xr-x 2 root root 40 Nov 14 10:06 openhab
drwxr-xr-x 2 redis redis 60 Nov 14 10:06 redis
-rw-rw-rw- 1 root root 0 Nov 14 10:06 service-runner.log
drwxr-xr-x 2 root root 60 Nov 14 10:06 supervisor
-rw-r----- 1 root adm 67782 Nov 14 11:00 syslog
-rw-r----- 1 root adm 190 Nov 14 10:06 user.log
-rw-rw-r-- 1 root utmp 1920 Nov 14 10:47 wtmp
and parent directory:
drwxr-xr-x 12 root root 4096 Aug 2 20:53 .
drwxrwxrwx 24 root root 4096 Oct 19 04:33 ..
drwxr-xr-x 2 root root 4096 Nov 13 17:25 backups
drwxr-xr-x 10 root root 4096 Aug 8 03:38 cache
drwxr-xr-x 38 root root 4096 Aug 18 01:30 lib
drwxrwsr-x 2 root staff 4096 Mar 13 2018 local
lrwxrwxrwx 1 root root 9 Jun 27 10:03 lock -> /run/lock
drwxr-xr-x 13 root root 580 Nov 14 10:56 log
drwxrwsr-x 2 root mail 4096 Jun 27 10:03 mail
drwxr-xr-x 2 root root 4096 Jun 27 10:03 opt
lrwxrwxrwx 1 root root 4 Jun 27 10:03 run -> /run
drwxr-xr-x 4 root root 4096 Jun 27 10:11 spool
-rw------- 1 root root 104857600 Jun 27 11:09 swap
drwxrwxrwt 5 root root 100 Nov 14 11:09 tmp
drwxr-xr-x 4 pi root 4096 Aug 15 22:05 www
Yes, DB is now up - so maybe this was the issue.
A couple of other observations for others if they have problems:
looks like ntp service is not installed so local (Australia) time wasn’t recognised. apt-get install ntp and tzone adjust has worked.
Worth adding ssh.txt file into boot directory as ssh is so much more convenient (IMHO) to check logs and updates etc - especially if the emon unit is in an awkward spot!
Import of backup data has mostly worked (sigh of relief!) but a couple of nodes have not come back to life. I’m working on those now.
A very nice update - thanks to all involved.
It look like i ran out of disk space, so i have expanded the storage on the SD card
pi@emonpi:~ $ df -h --total
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.9G 1.7G 2.1G 45% /
devtmpfs 484M 0 484M 0% /dev
tmpfs 489M 0 489M 0% /dev/shm
tmpfs 489M 13M 476M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 489M 0 489M 0% /sys/fs/cgroup
tmpfs 1.0M 0 1.0M 0% /var/tmp
tmpfs 30M 0 30M 0% /tmp
tmpfs 50M 1.5M 49M 3% /var/log
/dev/mmcblk0p1 43M 22M 21M 52% /boot
/dev/mmcblk0p3 11G 3.1G 6.8G 32% /home/pi/data
tmpfs 98M 0 98M 0% /run/user/1000
total 17G 4.7G 11G 31% -
Here is some more information. its looks like the SQL wont start.
journalctl -xe
Nov 19 08:31:36 emonpi sudo[5602]: pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/bin/systemctl status mysql.service
Nov 19 08:31:36 emonpi sudo[5602]: pam_unix(sudo:session): session opened for user root by pi(uid=0)
Nov 19 08:32:14 emonpi systemd[1]: mqtt_input.service: Service hold-off time over, scheduling restart.
Nov 19 08:32:14 emonpi systemd[1]: Stopped Emoncms MQTT Input Script.
-- Subject: Unit mqtt_input.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mqtt_input.service has finished shutting down.
Nov 19 08:32:14 emonpi systemd[1]: Started Emoncms MQTT Input Script.
-- Subject: Unit mqtt_input.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mqtt_input.service has finished starting up.
--
-- The start-up result is done.
Nov 19 08:32:14 emonpi mqtt_input[10844]: PHP Fatal error: Uncaught ErrorException: mysqli::__construct(): (HY000/2002): Connection refused in /var/www/emoncms/scripts/phpmqtt_input.php:61
Nov 19 08:32:14 emonpi mqtt_input[10844]: Stack trace:
Nov 19 08:32:14 emonpi mqtt_input[10844]: #0 [internal function]: exceptions_error_handler(2, 'mysqli::__const...', '/var/www/emoncm...', 61, Array)
Nov 19 08:32:14 emonpi mqtt_input[10844]: #1 /var/www/emoncms/scripts/phpmqtt_input.php(61): mysqli->__construct('127.0.0.1', 'emoncms', 'emonpiemoncmsmy...', 'emoncms', '3306')
Nov 19 08:32:14 emonpi mqtt_input[10844]: #2 {main}
Nov 19 08:32:14 emonpi mqtt_input[10844]: thrown in /var/www/emoncms/scripts/phpmqtt_input.php on line 61
Nov 19 08:32:14 emonpi mqtt_input[10844]: Fatal error: Uncaught ErrorException: mysqli::__construct(): (HY000/2002): Connection refused in /var/www/emoncms/scripts/phpmqtt_input.php:61
Nov 19 08:32:14 emonpi mqtt_input[10844]: Stack trace:
Nov 19 08:32:14 emonpi mqtt_input[10844]: #0 [internal function]: exceptions_error_handler(2, 'mysqli::__const...', '/var/www/emoncm...', 61, Array)
Nov 19 08:32:14 emonpi mqtt_input[10844]: #1 /var/www/emoncms/scripts/phpmqtt_input.php(61): mysqli->__construct('127.0.0.1', 'emoncms', 'emonpiemoncmsmy...', 'emoncms', '3306')
Nov 19 08:32:14 emonpi mqtt_input[10844]: #2 {main}
Nov 19 08:32:14 emonpi mqtt_input[10844]: thrown in /var/www/emoncms/scripts/phpmqtt_input.php on line 61
Nov 19 08:32:14 emonpi systemd[1]: mqtt_input.service: Main process exited, code=exited, status=255/n/a
Nov 19 08:32:14 emonpi systemd[1]: mqtt_input.service: Unit entered failed state.
Nov 19 08:32:14 emonpi systemd[1]: mqtt_input.service: Failed with result 'exit-code'.
Nov 19 08:32:20 emonpi sudo[5602]: pam_unix(sudo:session): session closed for user root
Nov 19 08:32:26 emonpi sudo[12497]: pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/bin/systemctl start mysql.service
Nov 19 08:32:26 emonpi sudo[12497]: pam_unix(sudo:session): session opened for user root by pi(uid=0)
Nov 19 08:32:26 emonpi systemd[1]: Starting MariaDB database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has begun starting up.
Nov 19 08:32:27 emonpi mysqld[12803]: 2018-11-19 8:32:27 1988702208 [Note] /usr/sbin/mysqld (mysqld 10.1.23-MariaDB-9+deb9u1) starting as process 12803 ...
Nov 19 08:32:30 emonpi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 08:32:30 emonpi systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Nov 19 08:32:30 emonpi systemd[1]: mariadb.service: Unit entered failed state.
Nov 19 08:32:30 emonpi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Nov 19 08:32:30 emonpi sudo[12497]: pam_unix(sudo:session): session closed for user root
and from the status of sql
pi@emonpi:~ $ sudo systemctl status mysql.service
● mariadb.service - MariaDB database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-11-19 08:32:30 UTC; 5min ago
Process: 12803 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 12628 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=e
Process: 12573 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 12511 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 12803 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
Nov 19 08:32:26 emonpi systemd[1]: Starting MariaDB database server...
Nov 19 08:32:27 emonpi mysqld[12803]: 2018-11-19 8:32:27 1988702208 [Note] /usr/sbin/mysqld (mysqld 10.1.23-MariaDB-9+deb9u1) starting as process 12803 ...
Nov 19 08:32:30 emonpi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 08:32:30 emonpi systemd[1]: Failed to start MariaDB database server.
Nov 19 08:32:30 emonpi systemd[1]: mariadb.service: Unit entered failed state.
Nov 19 08:32:30 emonpi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Any Ideas?
Is this a new or old SD Card? I wonder if the card is close to failure or if new, a fake card?
I suggest trying with a new card if this is a recycled one or test if it is fake (search Google to find out how to check)…
Hi @borpin
its not a fake card and is in good condition.
I did not expand the card when i first installed it and it looks like it ran out of space so i ran sudo emonSDexpand
Paul
My feeling is something got corrupted when it ran out of space. I’d be inclined to start from scratch, expand before anything else and see what happens.
I think you are right.
I think the SD card became full and thus damaged the SQL database.
I got another SD card and kept getting errors on start up,
So got a new card and it wrote ok, expanded ok and then restored the backup.
1 Like
Hi can I buy a new pre loaded SDcard with 30Oct18. If I can are there instructions then on how to copy data across and to insert it into the Pi.
Regards
John
I seem to be having an issue with expanding the sd card after flashing it with this new version. on running emonSDexpand I eventually get the following screens:
It works fine if I don’t run emonSDexpand, but once I do the card appears to be unreadable, ctrl-D doesn’t do anything
Did you want about 30min for the emonSDexpand process to expand the file-system? Did the Pi power off when it was finished?
Sure, we all pre-loaded SD card with the latest image via our online store:
https://openenergymonitor.com/emonsd-pre-loaded-raspberry-pi-sd-card/
Follow instructions here for how to use the backup Module to backup and move you data to the new image:
Thanks for your quick reply Glyn.
Nope, didn’t power off the Pi manually at all.
I ran the emonSDexpand command, then almost immediately the first 2 images showed up on an HDMI attached monitor, then it rebooted itself after around 30 seconds, and on boot the final image was shown (after about 30 seconds of other things saying ok) and nothing happened from that point.
After it reboots you need to leave it for 20-30min to apply the changes. You might not see any process running on the display when this happens. I’ve never actually run emonSDexpand with a hdmi monitor connected, I usually work with the emonPi headless. emonSDexpand worked when I used it last week.
What SD card are you using? Maybe it cannot handle the expansion process?
If your still having trouble we sell a pre-loaded and pre-expanded 16GB emonSD card via our online store:
https://openenergymonitor.com/emonsd-pre-loaded-raspberry-pi-sd-card/
I have the same issue with emonSDexpand. After rebooting, it cannot mount /home/pi/data
I’ve had a bit of a look at the script, and I think the problem is with the way the script handles /etc/fstab. The script contains:
# Ensure that the /data partition is unmounted after a reboot
sed -i 's/\/dev\/mmcblk0p3/#\/dev\/mmcblk0p3/' /etc/fstab
However, /etc/fstab uses PARTUUID to identify the partitions:
proc /proc proc defaults 0 0
PARTUUID=50c2568e-01 /boot vfat defaults 0 2
PARTUUID=50c2568e-02 / ext4 defaults,noatime,nodiratime 0 1
PARTUUID=50c2568e-03 /home/pi/data ext2 defaults,noatime,nodiratime 0 2I’ve modified sdpart_imagefile and it now works for me. I’ve created a pull request for the change:
1 Like