EmonPi not starting after reboot

Hardware emonPi
1

Hmm… so after 126 days of happy uptime I decided to update my EmonPi to get the new Emoncms solar history feature, but update through web interface failed - claimed some files modified in git pull which I don’t think they were - and also complained out of space.

Turns out /var/log was full - I actually don’t think it it was doing logrotate as setup was wrong (I see log rotation was changed to logger, I moved some old logs manually to make space and did a git pull in /var/www/emoncms to get latest update - then decided to reboot to let logger rotate the big emoncms.log etc… but its not come back.

I eventually 10 mins later did a power off/on, emonpi booted and did usual boot sequence, its now been sat at “Starting Raspberry Pi” (EDIT: on LCD) for the last 10 mins, responds to ping but not to SSH or http which is worrying - any suggestions how to recover?

I can’t work out why the base raspbian would not start even if emoncms was corrupted in some way?

2

Strangely it appears to be up… but filtering incoming connections which is strange?

~]# nmap -A

Starting Nmap 5.21 ( http://nmap.org ) at 2016-05-11 16:04 BST
Nmap scan report for
Host is up (0.022s latency).
Not shown: 997 filtered ports
PORT STATE SERVICE VERSION
22/tcp closed ssh
80/tcp closed http
443/tcp closed https

Service detection performed. Please report any incorrect results at Nmap OS/Service Fingerprint and Correction Submission Page .
Nmap done: 1 IP address (1 host up) scanned in 6.11 seconds

3

Very strange, have you been able to initiate an SSH or HTTP session? If not your only real option for debugging is to connect a HDMI monitor.

What emonSD pre-build SD card version are you using?

/var/log is stored in tmpfs and should be cleared on reboot.

Would recomend updating to latest 03-May16-emonSD.

4

Thanks Glyn,

Yes - I gave up and was getting frustrated so took it off the wall (which is pain as one CT is inside CU - luckily my newly arrived EmonTX is handy to swap over to as fast as possible for the CT).

Its coming up in emergency mode, just about to google for root password for pi… as I am not sure if has one as everything is done as “pi” with sudo as far as I know!

Some sort of card corruption maybe…

5

Yes that’s almost certainly a corruption issue. Was it one of our pre-build emonSD SD card images? There should be a file in /boot FAT partition with emonSD-XXX with XXX being the date of the image. If so, could you find out the date of the build for me?

6

Sorry I forgot that, its emonSD_18Dec2015

I am not sure if its corruption… or fstab issue… just looking now at the card on another pi…

7

Hmm, the vFat partition is dirty:
`~ $ sudo fsck -C /dev/sda1
fsck from util-linux 2.20.1
dosfsck 3.0.13, 30 Jun 2012, FAT32, LFN
0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.

  1. Remove dirty bit
  2. No action
    ? 1
    Leaving file system unchanged.
    /dev/sda1: 72 files, 9955/31189 clusters
    `

But I also see there is no error handling in the emon fstab:
$ cat fstab tmpfs /tmp tmpfs nodev,nosuid,size=30M,mode=1777 0 0 tmpfs /var/log tmpfs nodev,nosuid,size=50M,mode=1777 0 0 tmpfs /var/lib/dhcp tmpfs nodev,nosuid,size=1M,mode=1777 0 0 tmpfs /var/lib/dhcpcd5 tmpfs nodev,nosuid,size=1M,mode=1777 0 0 tmpfs /var/lib/openhab tmpfs nodev,nosuid,size=40M,mode=1777 0 0 proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults,noatime,nodiratime 0 2 /dev/mmcblk0p2 / ext4 defaults,ro,noatime,nodiratime,errors=remount-ro 0 1 /dev/mmcblk0p3 /home/pi/data ext2 defaults,rw,noatime,nodiratime,errors=remount-ro 0 2

And it was that stopping the boot… going to put it back in emonPi now… maybe there needs to be a way to stop emonPi halting on boot (as you can’t even access the SD card with emonPi) with something like:
/dev/mmcblk0p1 /boot vfat defaults,noatime,nodiratime,nofail 0 2

  • Will test in a minute, seems to be Systemd change in Jessie from googling…
8

The following changes were made to fstab in an update 13 days ago:

‘commit’ was removed to work with latest RasPi3 firmware

Forum thread discussion: Tips n Tricks from my own Pi appliance... | Archived Forum

9

Thanks, I am actually not sure its an fs issue now.

On boot I get two failed:
[FAILED] Failed to start load/save random seed [FAILED] Failed to mount /var/lib/dhcpd5 ... [DEPEND] Dependency failed for local file system. [DEPEND] Dependency failed for netfilter persistent configuration

This seem to be the cause of the halt, the local FS one seems to be the dhcpd5 one… I am loathed to load a new image as I have a Raspberry Pi B installed (not 2/3) and I spent some time recompiling Node Red to work with the other processor type… :frowning:

Going to have dinner and think about it… suspect need more googling on that service.
EDIT: To add, I don’t have the commit=180 in my fstab from December

10

Ok, my boot issue was caused by dhcpcd5 packaging being missing, so not starting (but being in the systemctld startup) so I can now boot emonpi… and it sems half upgraded so looking at that now.

Is there a way to run the upgrade script from the command line not emonhub?

11

And finally getting there… something wrong with the update to do with the database changing from root as user to emoncms as user with v9… below is where it goes wrong.

`Update Emoncms database
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 24
PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98

Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/emoncms/Lib/dbschemasetup.php on line 98
[“CREATE TABLE app_config (userid int(11), data text) ENGINE=MYISAM”,“CREATE TABLE feeds (id int(11) not null auto_increment primary key, name text, userid int(11), tag text, time int(10), value double, datatype int(11) not null, public tinyint(1), size int(11), engine int(11) not null, processList text) ENGINE=MYISAM”,“CREATE TABLE input (id int(11) not null auto_increment primary key, userid int(11), name text, description text, nodeid text, processList text, time int(10), value float) ENGINE=MYISAM”,“CREATE TABLE schedule (id int(11) not null auto_increment primary key, userid int(11), name varchar(30), expression text, timezone varchar(64) Default ‘UTC’, public tinyint(1)) ENGINE=MYISAM”,“CREATE TABLE users (id int(11) not null auto_increment primary key, username varchar(30), email varchar(30), password varchar(64), salt varchar(32), apikey_write varchar(64), apikey_read varchar(64), lastlogin datetime, admin int(11) not null, gravatar varchar(30), name varchar(30), location varchar(30), timezone varchar(64) Default ‘UTC’, language varchar(5) Default ‘en_EN’, bio text) ENGINE=MYISAM”,“CREATE TABLE rememberme (userid int(11), token varchar(40), expire datetime) ENGINE=MYISAM”,“CREATE TABLE multigraph (id int(11) not null auto_increment primary key, name text, userid int(11), feedlist text) ENGINE=MYISAM”]`

emoncmsdbupdate.php seems to take its login from somewhere up he script chain… is there a config file that already got overwritten with new defaults before the update is made?

Anyway around this, maybe grant user emoncms database access directly?

I kinda bust my local emoncms, its still uploading just fine to emoncms.org, when I login, which I can’t do after upgrade runs as I get the error:
Can't connect to database, please verify credentials/configuration in settings.php Error message: Access denied for user 'emoncms'@'localhost' (using password: YES)

And have to set settings.php back to root/raspberry which is why I am thinking its a DB issue?

When I do login, I don’t see my feeds, the emonpi inputs are all under “unknown”… yeah I broke it…

Any suggestions welcome :wink:

Going to sleep on it for now…

12

And while I am talking to myself :wink:

I seem to have fixed it all with adding manually emoncms user with access to the emoncms database directly in mysql. So just for anyone else hitting this:

emonpi$ mysql --user=root --password=raspberry mysql

mysql> CREATE USER ‘emoncms’@‘localhost’ IDENTIFIED BY ‘emonpiemoncmsmysql2016’;
mysql> GRANT ALL PRIVILEGES
ON emoncms.*
TO ‘emoncms’@‘localhost’;
mysql> exit;

I am now off to take a backup… :minidisc:

13

Nice, good detective work figuring out these issues! :sunglasses: . Glad you have got there in the end.

I’m surprised there was not a MYSQL Emoncms user already preset. The emonSD_18Dec2015 was an early beta release based on Minibian lightweight Jessie. Now you are up and running I would recomend running and update (which should pull in the backup module) then making a backup to import your data to the latest emonSD image. We intended the 03May16 to be a LTS release. See emonSD download & changelog.

The update can be ran from command line. You will just need to create a flag manually:

$ rpi-rw
$ touch /var/tmp flag
$ ~/emonpi/./update

After updating you should be able to (after a refresh) see the backup module present in local Emoncms

14

Thanks Glyn, good chance to polish up my very rusty mysql… and now have a much better understanding of EmonPI! I already did the command line update last night (figured out upgrade flag in tmp from script) - that’s where the SQL failure came from above - definitely no emoncms mysql user in the December 15th image by the way. Upgrade now runs clean no errors.

The problem I have still is that the nodes all come under NoGroup not EmonPi, and there are no nodes for the EmonTX its recieving.

emonpi-nogroup.png1130×563 33.4 KB

I pulled down the 3rd May SD image and put that on another card as part of the diagnostic, this worked fine in the emonpi. I currently have the card in a spare pi so then I was able to backup my emonpi (original SD) and load the backup on the new image on the spare pi (no emonpi shield).

Initially the new image/card had emonpi and emontx under nodes as you would expect (this card was in the emonpi for diagnoses so picked them up), once I restored the backups it was gone and I just get the same view as the live machine. I should have taken backup on new card for reference (theme here!) - I did factory reset on the new image but now there are no nodes (as there is no emonpi attached).

I also noticed I have no dashboard link - see above screenshot.

Is there a way to just pull the database history data folder over onto the new cards and let the new image pick up the feeds then re-configure the emoncms.org upload again?

Thanks for the help :slight_smile:

(This is proving a mildly expensive excercise as the “spare” pi is a new pi3 I bought for a new project, but having seen how much faster node-red runs on pi3 over B I might just have to upgrade emonPi to 3!)

15

I’m not sure how best to do this, however if you do a standard export / import your old inputs and feeds will be imported onto the new image. However the new (correctly named) inputs will quickly appear. You just then need to delete the old input processes and log the new inputs to feeds

16

Ok thanks… the problem is that on the still live system the inputs are logging to the feeds still (and to emoncms.org) but not showing locally.

I will have to go for it and swap the new card with May 3rd image into the emonpi I think and carry on from there.

17

Oh and before I forget, you have typo on the backup/restore page Glyn:

CAUTION ALL EMONCMS ACCOUNT DATA WILL BE OVERWRITTEN BY THE IMPORTETD DATA

18

Thanks, been fixed:

19

Hi, yesterday i had a main failure, and my emonpi stopped. After the mains back, the emonpi not restarted, the LCD shows reboot only, it not possible to connect via HTTP or SSH. Verzión 2.90.
What can I do?

Regards, Péter

20

Considering the time between your posts, I’d suggest it is likely the SD card is corrupt I am afraid. Do you have a backup? If not starting from scratch is the only option.

You need to try a different SD card to prove it is that which is preventing boot. If you have a spare known good card, the new SD image is available for download.

You would also be best starting a new thread for any further queries.