emonPi SD Card Build

Discussion related to the building of a new SD card from the ground up as described by Glyn in his Jan 2016 posting:

I want to add emoncms to an existing Pi2 running the latest 2016-03-18-raspbian-jessie.img downloaded from the official site onto an 8GB card. I have the option of downloading the latest pre-built card from openenergymonitor based on Jessie Lite and restoring the full package set OR starting from scratch with my existing card and following the build instructions. I have chosen to do the latter running the Pi2 headless from an OSX terminal window.

I have succeeded in following sections 1 through to 4. Thus I now have installed the core packages & set the hostname/password, created the additional data partition and modified the fstab file (left the root filesystem as rw at present) and set up the serial port. Section 4 required the I2C module to be activated using raspbi-config but was then also successful.

I am, however, struggling to implement section 5, in particular the mosquito MQTT service. The initial set of commands finishes with the line pecl install Mosquitto-alpha. The pecl command is not installed on my Jessie image. I have searched for more information on the installation of this package and have implemented the following commands which I believe should work prior to above command:

sudo apt-get install php-pear
sudo apt-get install php5-dev
sudo apt-get install libcurl3-openssl-dev

I was encouraged to see that the auto detection of the libmosquitto location was asked & the command completed successfully. However, the final two lines:

“configuration option “php_ini” is not set to php.ini location”
“You should add “extension=mosquitto.so” to php.ini”

are causing me some issues. The only php.ini file I can locate on my card is at /etc/php5/cli/php.ini and I have added the line extension=/usr/lib/php5/20131226/mosquitto.so to the Dynamic Extensions section.
I am not sure what the first of these 2 lines means.

The PHP extension config files given next in section 5 build do not exist and neither does the /etc/php5/apache2/ directory or its subdirectory conf.d. I have created these and issued the commands. The following commands editing /etc/mosquitto/passwd & creating a password in section 5 were done successfully.

However, the publish and subscribe test given does not work with the subscribe command hanging in a new terminal window. I am loathe to continue with section 6 until this test works as it will almost certainly cause later issues. I should add that I am familiar enough with implementing a set of Linux instructions in a terminal window but not necessarily the understanding of the commands themselves. This is particularly so of the MQTT service so I am not clear how to identify the problem.

Any thoughts or guidance on the validity of what I have tried and how to proceed would be greatly appreciated

Not sure what’s going on here. As you can see from the build from scratch guide there, it’s quite involved! I would recommend for downloading our pre built image then updating it to the full Jessie. This can be done quite easily with a couple of apt get commands (search raspberrypi forum). It will be much easier for us to provide support if we know you are starting from a known base.

However it should be possible to build from scratch. I’ll try and investigate your issue over the weekend…

Thanks Glyn!

If it is any help, I have checked the list of servers using netstat -at:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:1883 : LISTEN
tcp 0 0 :afpovertcp : LISTEN
tcp 0 0 :webmin : LISTEN
tcp 0 0 :ssh : LISTEN
tcp 0 184 192.168.1.116:ssh propagationmatter:62057 ESTABLISHED
tcp 0 0 192.168.1.116:ssh propagationmatter:62044 ESTABLISHED
tcp6 0 0 [::]:1883 [::]:
LISTEN
tcp6 0 0 localhost:4700 [::]:
LISTEN
tcp6 0 0 [::]:ssh [::]:
LISTEN

Port 1883 does appear to be listening. My suspicion is that one of the .php, .conf or .ini files may be mis-located, missing or have incorrect content, particularly as some of the sub directories expected in the guide I was following during the pecl install were not quite the same. This could, however, just be one of those directory structure changes that do occur between releases!

My reason for starting with the full Jessie image was to connect a 7" HDMI touch screen & Rii wireless keyboard to the Pi2 to display emoncms in a window on the Pi2 itself before turning off the low write mode and moving content to an HDD.

However, I will download the pre-built card as you suggest and try to ‘fatten it’ back to the full Jessie. I will also experiment with mounting it in an OS X VMWare Linux Mint VM and compare the MQTT directory structures and file contents to my own partial build.

Just a thought for future forum debate of the emonBase option! “Given the increasing power & memory of the Pi2 & Pi3 since inception and increasing size of cheap SD cards, it would be useful to have a pre-built image based upon full Jessie

Many Thanks!

I’ve built emoncms on a minimal Jessie install and it really was not a big deal. One way of doing this would be to build the prebuilt image and go and look for the files and possibly the settings.

Can you subscribe and publish to Mosquitto directly? Is the service running? I did this in 2 terminal windows (one publish one subscribe) to test Mosquitto. Subscribe is not hanging, it is just waiting for something to be published!

If you do find the solution, please post :smile:

Thanks to Brian’s suggestion, I did some further testing and have discovered that the problem I had was trying to follow the build script too literally without that deeper understanding of how the MQTT service and the commands mosquitto_pub and mosquitto_sub themselves operate. These commands will be of second nature to the emonPi developers but, in a past life, I used to test internationally agreed mathematical radio models that had to be implementable from only a written text alone. As such, once I realised that:

mosquitto_sub -v -u ‘emonpi’ -P ‘emonpimqtt2016’ -t ‘test/topic’

has to be issued in the 2nd window prior to:

mosquitto_pub -u ‘emonpi’ -P ‘emonpimqtt2016’ -t ‘test/topic’ -m ‘helloWorld’

in the first window, all was OK! Hence, I would like to see the order of these 2 commands reversed in the build scripts.

P.S. In a similar vein, in section 1 of the build script, if the intention is to change the user pi password, then I believe the command should be either passwd or sudo passwd pi.

I am very appreciative of your build script which I am using both as as a means to a customised working emonBase and a Linux learning exercise! I always find an implementation made by a user removed from the coalface is a very valuable exercise. Keep up the good work!

I am now happy to proceed with the remainder of the build script and will keep you informed of any other difficulties that I may encounter. :smile:

Nice! Good work figuring this out. I’ve updated the build guide as you suggested to make the testing procedure clearer.

I have continued to the installation of emonHub (emon-pi) variant at the end of build section 6.
I believe that the emonhub install script given in the first line:

git clone GitHub - openenergymonitor/emonhub: Python service linking and decoding input to MQTT & Emoncms && emonhub/install

is inconsistent with the two following lines:

sudo rm ~/data/emonhub.conf and

sudo ln -s /home/pi/emonhub/conf/emonpi.default.emonhub.conf /home/pi/data/emonhub.conf

Line 34 of the install script moves ~/emonhub/conf/emonpi.default.emonhub.conf to ~/data/emonhub.conf since it does not already exist. Those 2 lines then proceed to delete the ~/data/emonhub.conf file before attempting to symlink that filename back to the original ~/emonhub/conf/emonpi.default.emonhub.conf file which is no longer present.

Either the emonhub/install script needs to change or, as I have done, change the first of those two lines:

sudo mv ~/data/emonhub.conf ~/emonhub/conf/emonpi.default.emonhub.conf to restore a default file.

Presumably this issue would not arise when upgrading as a ~/data/emonhub.conf would exist and line 34 would not be done.

Your thoughts please, Glyn?

I have now completed the build as far as section 9 for Emoncms. Sections 7 and 8 proceeded relatively smoothly once I had mentally merged the instructions given in the section itself with the links to the more detailed outsourced instructions. No other major problems were encountered and no other errors occurred.

I had hoped that Emoncms would now start and register a new user. Although the URL : 192.168.1.116/emoncms brings up the login page with the register link, I have been unable to register a new user on the register page. The site returns 101 in the top left corner and displays a red box containing the word undefined.

I have looked in the /var/log/apache2/error.log and the following error is present:

[Wed May 11 15:27:30.064581 2016] [:error] [pid 2674] [client 192.168.1.91:51903] PHP Warning: session_start(): open(tmp/sess_fmseqvmtee6la574jf0ndjsss6, O_RDWR) failed: No such file or directory (2) in /var/www/emoncms/Modules/user/user_model.php on line 101, referer: http://192.168.1.116/emoncms/

The line number displayed on the page agrees with the failing line in the user_model.php file. Line 101 in that file is a function call to session_start(); . I can find no definition of that function in any of the files in the user directory. The contents of the tmp file that failed to open are:

userid|s:1:“1”;username|s:8:“mfhamer”;read|i:1;write|i:1;admin|s:1:“1”;lang|s:5:“en_EN”;pi@emonpi:/var/www/emoncms

I have set $enable_multi_user = true; in the emoncms settings.php file to keep the register option displayed.

Incidentally, a direct json call: http://192.168.1.116/emoncms/user/register.json produces a similar message:


Warning: session_start(): open(tmp/sess_fmseqvmtee6la574jf0ndjsss6, O_RDWR) failed: No such file or directory (2) in /var/www/emoncms/Modules/user/user_model.php on line 101

{“success”:false,“message”:“Missing username, password or email parameter”}

I am entering a valid name, password and e-mail.

I am a loss as to how to debug this as I am at the limits my understanding of the processes involved. I am slowly coming to the conclusion that Glyn’s advice to add packages to the pre-built card is probably my best option (tedious and laborious as that might be). This build process is very complex and is probably only suited to users with advanced Linux skills. Nevertheless, I have come quite a long way and would like to understand why emoncms does not register a new user. Any ideas would be welcome

Thanks!

I think it can be done with a single apt-get. Try asking on raspberrypi forums. I will take a look tonight. What packages are you missing that you need?

I want the full GUI back together with webmin, tightvnc and apple file protocol in netatalk that I previously installed. My searches to date have not located a single apt-get but I think I could run sudo dpkg -l > file.txt on each build and then run a diff command on the two text files. Hopefully, the right top level package choice would bring in many of the dependencies with it.

It might well be more involved than I previously thought. I thought I had a script to do this somwhere.

I’ve just posted a topic on the RasPi forums to ask the question for you.

Yes and it goes back to the question of how much knowledge someone is expected to have :smile:

Glad it helped.

Following up on a comment to the reply to the question posed by Glyn, I have discovered the following comprehensive guide to install either a LXDE or XFCE based GUI onto Jessie Lite. It also includes details on how to add a range of extras and should be sufficient for me to get the functionality I require. For anybody who wants to add a GUI to Jessie Lite I do recommend reading the guide as a number of interlinked packages are needed.

Although I have not quite been able to complete the emonPi build successfully, it has not been wasted time as I do now have a better, if still incomplete, understanding of the many processes & techniques that the development team have used to overcome the limitations of running a monitoring system from an SD card.

Many thanks to Glyn and the team for their sterling work!

Glyn,

Not sure if you noticed a footnote at the end of the raspberry pi blog on the latest release of raspbian.

Basically it mentions that you can now put your wpa_supplicant file with said and password into boot when you have flashed the SD card and then on boot this is copied to the correct place and hey presto you have WiFi up and running. Essentially means if you have a copy of a working wpa supplicant file you can get a new image running without having to connect a screen and keyboard.

This should be in your SD card build and maybe also in the pi setup guide?

Simon

We have developed our own Wifi config for Emoncms to connect to Wifi:

However, to use our Wifi connect module, you need to be connected via Ethernet. I can see the advantage of being able to just pop your wifi details direct onto the SD card. Do you have a copy of our latest image to hand? Could you test to see if this works?

I have seen elsewhere systems that go into a hotspot mode on first boot so you can connect by say a phone to that hotspot, set the WiFi network to use and then reboot. Would make the need for connecting by cable redundant.

Hi Glyn, I haven’t got a copy. But here is the full text of the footnote which also mentions resizing the SD card automatically as well.

Finally, there are a couple of setup-related features:

• When flashing a new Raspbian image, the file system will automatically be expanded to use all the space on the card when it is first booted.

• If a wpa_supplicant.conf file is placed into the /boot/ directory, this will be moved to the /etc/wpa_supplicant/ directory the next time the system is booted, overwriting the network settings; this allows a Wifi configuration to be preloaded onto a card from a Windows or other machine that can only see the boot partition.

I’d second this. Having to connect to an ethernet cable can be a pain if your router is inaccessible or just a pain to get to (my case).

Simon

Apologies for 2 replies but I should have added to my earlier reply that the new wifi setup is really for developers and not end users. Maybe I should have done a separate topic on this.

It would definitely be worth mentioning in the raspi installation guide which is the one I follow when setting up my system.

Simon

Paul??