Local Ubuntu Emoncms and Redis

Lykke · 2 May 2018 21:38

Yes, I did the php7 install and I did include that bit also. There were no error messages.

I’ll do the emoncms install again and doublecheck the output.

Lykke · 2 May 2018 22:05

This is the output when trying to install redis, seems there is a file missing:

lykke@lykke-Standard-PC-i440FX-PIIX-1996:~$ sudo pecl install swift/swift redis
downloading Swift-5.1.0.tgz ...
Starting to download Swift-5.1.0.tgz (113,740 bytes)
.........................done: 113,740 bytes
downloading redis-4.0.2.tgz ...
Starting to download redis-4.0.2.tgz (212,617 bytes)
...done: 212,617 bytes
install ok: channel://pear.swiftmailer.org/Swift-5.1.0
24 source files, building
running: phpize
Configuring for:
PHP Api Version:         20151012
Zend Module Api No:      20151012
Zend Extension Api No:   320151012
enable igbinary serializer support? [no] : y
enable lzf compression support? [no] : y
building in /tmp/pear/temp/pear-build-rootlb7MEv/redis-4.0.2
running: /tmp/pear/temp/redis/configure --with-php-config=/usr/bin/php-config --enable-redis-igbinary=y --enable-redis-lzf=y
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for a sed that does not truncate output... /bin/sed
checking for cc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether cc accepts -g... yes
checking for cc option to accept ISO C89... none needed
checking how to run the C preprocessor... cc -E
checking for icc... no
checking for suncc... no
checking whether cc understands -c and -o together... yes
checking for system library directory... lib
checking if compiler supports -R... no
checking if compiler supports -Wl,-rpath,... yes
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking target system type... i686-pc-linux-gnu
checking for PHP prefix... /usr
checking for PHP includes... -I/usr/include/php/20151012 -I/usr/include/php/20151012/main -I/usr/include/php/20151012/TSRM -I/usr/include/php/20151012/Zend -I/usr/include/php/20151012/ext -I/usr/include/php/20151012/ext/date/lib -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
checking for PHP extension directory... /usr/lib/php/20151012
checking for PHP installed headers prefix... /usr/include/php/20151012
checking if debug is enabled... no
checking if zts is enabled... no
checking for re2c... no
configure: WARNING: You will need re2c 0.13.4 or later if you want to regenerate PHP parsers.
checking for gawk... no
checking for nawk... nawk
checking if nawk is broken... no
checking whether to enable redis support... yes, shared
checking whether to enable sessions... yes
checking whether to enable igbinary serializer support... y
checking whether to enable lzf compression... y
checking use system liblzf... no
checking for igbinary includes... configure: error: Cannot find igbinary.h
ERROR: `/tmp/pear/temp/redis/configure --with-php-config=/usr/bin/php-config --enable-redis-igbinary=y --enable-redis-lzf=y' failed

pb66 · 2 May 2018 22:13

Why on both questions have you chosen to say yes when the default suggestion was “no”?

enable igbinary serializer support? [no] : y
enable lzf compression support? [no] : y

the file that is reported as missing is related to the first of those questions.

Try again, but this time try accepting the suggestions, unless you have good reason to do other wise and know what that entails or might need in the way of additional libs etc.

[edit] Oh and by the way, the command you need is just

sudo pecl install redis

the original command is to install 2 packages and “swift/swift” doesn’t need reinstalling each time whilst trying to sort phpredis out.

You should find that (php-)redis installs ok without uninstalling as it aborted, but if needed you can remove it first and then reinstall without having to start a fresh.

sudo pecl uninstall redis
sudo pecl install redis

Lykke · 3 May 2018 06:47

Actually I did try the “No” option, or rather I thought I tried it. Looking through the output again, I realize that I put in an “n” in place of just hitting enter.

enable igbinary serializer support? [no] : n
enable lzf compression support? [no] :

Tried again to just hit enter, big fat fingers, and now it seems to install correctly. Will try it out when I get home from work.

bruce_miranda · 3 May 2018 07:18

How do I check for sure if my redis is installed correctly and working.

When I look at the phpinfo on the Apache server, under the redis section I see.

redis

Redis Support
enabled
Redis Version
4.0.1
Available serializers
php, igbinary
Available compression
lzf

pb66 · 3 May 2018 09:51

Perhaps @Lykke can confirm what he see’s once he has got it installed.

I do not have v4.0.1 installed yet to test and I have just found a note in the changelogs stating there are “breaking changes” in post v4 versions, I do not know if they affect us or how to test for them so I have opened an issue on emoncms that hopefully @TrystanLea or @nchaveiro might be able to answer.

github.com/emoncms/emoncms

Possible breaking api changes in php-redis

opened 09:42AM - 03 May 18 UTC

closed 08:48AM - 17 Jan 19 UTC

pb66

question

Whilst helping a user on the OEM forum I noticed that PECL was installing v4.0.2… and read the [changelog](https://pecl.php.net/package-changelog.php?package=redis&release=4.0.2). The changelog states "*** WARNING! THIS RELEASE CONTAINS BRAKING API CHANGES! ***" for all versions above 4 and all I could find on those changes is this issue https://github.com/phpredis/phpredis/issues/1333 which has a couple of interesting linked issues. @TrystanLea or @chaveiro are you able to confirm if these changes are going to impact emoncms? Can we continue to use the latest phpredis versions, now at 4.x.x without changes? Or can the required changes be made to emoncms? Or do we need to stick with 3.x.x (currently 3.1.6) and only get critical updates moving forward? There is always a concern about how long devs will maintain a backwards compatible version and it also presents a "hiccup" in the installation guide as we will need to define a version (eg pecl install redis-3.1.6) instead of leaving it open (eg pecl install redis) and then that needs maintaining in both the guide and any future install or update scripts. I hope for option 1, or 2 if possible, I think 3 will start to reintroduce the issues we have not long seen resolved over v2.2.5 vs >v2.2.7 and again with php5 vs php7 prior to v3.1.3. The apt-get package is still at [3.1.1 for Stretch](https://packages.debian.org/stretch/php-redis) which is prior to the rounding issue fix we saw in 3.1.3 and 2.2.5 for Jessie which is prior to the memory leak fix in v2.2.7 (current emonSD image version) so I think we do still need to stick with PECL. I am unable to test v4.0.0 right now as both my test and live instances are currently on the same server so I cannot use on "test" without the risk of breaking "live". Any thoughts?

I cannot say yet whether v4 is a problem of if emoncms requires change, but worse case scenario I guess is that you may need to use an earlier version if there is a problem or until we are sure it’s ok. Eg

sudo pecl uninstall redis
sudo pecl install redis-3.1.6

Lykke · 3 May 2018 16:35

Finished installing redis (4.02). Like the installer suggested, I added “extension=redis.so” to php.ini in the apache2 folder (correct folder?). No errors.

Finished the emoncms and the sync module install. Only install issues I encountered was rw permissions and incorrect paths. After emoncms bootup and update, sync connects OK to emoncms.org and the sync-feed list is populated. On pressing “download”, the remote feed shows up in the local feed list, but no data is downloaded.

Emoncms reports Redis 3.06

Anything I should test or try before following Pauls advice to roll back redis to 3.1.6 ?

TrystanLea · 3 May 2018 17:27

Did you install the service runner required by the sync module?
Installation notes can be found at the bottom here: GitHub - emoncms/sync: In development: Download or upload emoncms feed data between local and remote emoncms server's

pb66 · 3 May 2018 17:27

You’ve lost me a bit there, the guide in the emoncms repo is complete, we do not edit php.ini as a rule. The correct way is to use phpenmod top create symlinks in the necessary places to allow php to dynamically create it’s settings.

Aside from installing the Debian packages for php-pear and php-dev etc included in the main package list, installing the redis extension consists of the following 4 steps

Updating the PECL index channel with sudo pecl channel-update pecl.php.net
Install the redis package with sudo pecl install redis (or maybe a specific version)
Make that extension “available” using printf "extension=redis.so" | sudo tee /etc/php/7.0/mods-available/redis.ini 1>&2 to add a file in “mods-available”
Enable the extension with sudo phpenmod redis

at that point you should be done!

If there are any further steps required due to unusual paths etc because of the environment you are installing to I’m afraid I do not know about those.

running pecl info redis to tell you what version phpredis is installed ie available to enable.

running php -m will tell you what modules are enabled, “redis” should be hopefully be listed.

from the local machine you should also be able to navigate to http://myserver/emoncms/php-info.php to see a fair bit of info and at the end there should be a “redis” section that tells you about the phpredis extension.

And also if you look at the last section on the admin page there should be a “redis” entry with version number in there too if it’s successfully enabled, installed but not enabled modules are not listed there.

This is not to be confused with the Redis server section on the admin page which is the database application itself, not the php extension, that is what I believe you are looking at when you say “Emoncms reports Redis 3.06” as you have already said you’ve installed v4.0.2 and mentioned “rolling back to 3.1.6” which you cannot do if you are on 3.0.6.

Here is a “System information” dump that I have just pinched from a recent post by @Paul

Server Information


Emoncms	Version	9.8.29 : 2018.05.01
	Modules	Administration : App v1.1.0 : Dashboard v1.1.1 : EventProcesses : Feed : Input : CoreProcess : Schedule : Time : User : Visualisation
Server	OS	Linux 4.9.59-v7+
	Host	raspberrypi raspberrypi (127.0.1.1)
	Date	2018-05-03 15:43:33 BST
	Uptime	15:43:33 up 6 min, 3 users, load average: 0.45, 0.98, 0.62
HTTP	Server	Apache/2.4.25 (Raspbian) HTTP/1.1 CGI/1.1 443
MySQL	Version	5.5.5-10.1.23-MariaDB-9+deb9u1
	Host	localhost (127.0.0.1)
	Date	2018-05-03 15:43:33 (UTC 01:00‌)
	Stats	Uptime: 394 Threads: 3 Questions: 21 Slow queries: 0 Opens: 19 Flush tables: 1 Open tables: 13 Queries per second avg: 0.053
Redis	Version	3.2.6
	Host	localhost:6379 (127.0.0.1)
	Size	130 keys (701.08K)
	Uptime	0 days
MQTT	Version	1.4.10
	Host	localhost:1883 (127.0.0.1)
Pi	CPU Temp	54.23°C
Memory	RAM	Used: 39.82% Total: 875.78 MB Used: 348.72 MB Free: 527.06 MB
	Swap	Used: 0.00% Total: 100 MB Used: 0 B Free: 100 MB
Disk	Mount	Stats
	/	Used: 67.00% Total: 6.86 GB Used: 4.6 GB Free: 1.93 GB
	/boot	Used: 50.62% Total: 41.15 MB Used: 20.83 MB Free: 20.32 MB
	/mnt/data	Used: 17.82% Total: 458.49 GB Used: 81.71 GB Free: 376.78 GB
PHP	Version	7.0.27-0+deb9u1 (Zend Version 3.0.0)
	Modules	apache2handler : bz2 v7.0.27-0+deb9u1 : calendar v7.0.27-0+deb9u1 : Core v7.0.27-0+deb9u1 : ctype v7.0.27-0+deb9u1 : curl v7.0.27-0+deb9u1 : date v7.0.27-0+deb9u1 : dio v0.1.0 : dom v20031129 : exif v7.0.27-0+deb9u1 : fileinfo v1.0.5 : filter v7.0.27-0+deb9u1 : ftp v7.0.27-0+deb9u1 : gd v7.0.27-0+deb9u1 : gettext v7.0.27-0+deb9u1 : hash v1.0 : iconv v7.0.27-0+deb9u1 : json v1.4.0 : libxml v7.0.27-0+deb9u1 : mbstring v7.0.27-0+deb9u1 : mcrypt v7.0.27-0+deb9u1 : mosquitto v0.4.0 : mysqli v7.0.27-0+deb9u1 : mysqlnd : openssl v7.0.27-0+deb9u1 : pcre v7.0.27-0+deb9u1 : PDO v7.0.27-0+deb9u1 : pdo_mysql v7.0.27-0+deb9u1 : Phar v2.0.2 : posix v7.0.27-0+deb9u1 : readline v7.0.27-0+deb9u1 : redis v3.1.6 : Reflection v7.0.27-0+deb9u1 : session v7.0.27-0+deb9u1 : shmop v7.0.27-0+deb9u1 : SimpleXML v7.0.27-0+deb9u1 : sockets v7.0.27-0+deb9u1 : SPL v7.0.27-0+deb9u1 : standard v7.0.27-0+deb9u1 : sysvmsg v7.0.27-0+deb9u1 : sysvsem v7.0.27-0+deb9u1 : sysvshm v7.0.27-0+deb9u1 : tokenizer v7.0.27-0+deb9u1 : wddx v7.0.27-0+deb9u1 : xml v7.0.27-0+deb9u1 : xmlreader v7.0.27-0+deb9u1 : xmlwriter v7.0.27-0+deb9u1 : xsl v7.0.27-0+deb9u1 : Zend OPcache v7.0.27-0+deb9u1 : zip v1.13.5 : zlib v7.0.27-0+deb9u1 :

In there you will see the Redis server is in it’s own section and at version 3.2.6 and in the last “PHP - Modules” section, the Redis extension for PHP is listed as “redis v3.1.6” (listed means it’s enabled).

As for the Sync module, I think we should confirm redis is ok before attempting the sync module, there are many other bits to that process that could stand between a button press and a succesful download, not just redis. It needs to create a temp flag file, there needs to be a cronjob running to pick up on that flag and run a script which has to reach the server and be able to write it locally so there are many permissions and paths that may need checking if this is not a straightforward emonPi/emonSD.

So are you able to confirm redis is both installed and enabled using one or more of the above methods?

Since this is a fresh install and on a more exotic OS/machine and you are trying to use some very new software that hasn’t been widely tested, perhaps you should roll-back to 3.1.6, there are too many unknowns, if the module is enabled and the sync module still doesn’t work we don’t know if it’s due to a version 4.0.2 issue or not, at least with 3.1.6 we know that should work ok and reduces the unknowns to just an uncommon environment and an untested emoncms module.

bruce_miranda · 3 May 2018 17:56

Didn’t know the emoncms site had its own phpinfo page. I wrote my own!

Anyway, I have everything as per Paul’s checklist. So I’ll assume my redis is set up.

Just for info, on my Adminstration page Redis (server) is at 3.0.6 and PHP redis is at 4.0.1

Everything appears to work fine. But I haven’t used sync.

pb66 · 3 May 2018 18:13

So your redis server is behind our normal version and your redis extension is ahead of our norm (ie server 3.2.6 and extension 3.1.6) I have no idea how or if that combo will work so please let us know how you get on in due course.

Lykke · 3 May 2018 18:23

Thanks you for the comprehensive answer!

Emoncms and PECL reports redis 4.02 in the PHP section. PHP confirm redis. I’ll try the roll back

3.1.6 install OK. What do you make of this?:

Build process completed successfully
Installing ‘/usr/lib/php/20151012/redis.so’
install ok: channel://pecl.php.net/redis-4.0.2
configuration option “php_ini” is not set to php.ini location
You should add “extension=redis.so” to php.ini

bruce_miranda · 3 May 2018 18:33

I noticed that too. But I haven’t specified a version. This is what apt and/or pecl have installed.
I just ran an apt update and there no new redis-server for Ubuntu 16.04 on the repositories.

Lykke · 3 May 2018 18:39

Yes - created a crontab in my user home directory.

Lykke · 3 May 2018 18:52

Now I have installed redis 3.1.6 All Pauls check reports redis 3.1.6 is up and running and on top I did a reboot of the VM.
Unfortunately no progress on the sync tool:

I pushed the two missing download buttoms, the feeds appear in the feed list, but no data

pb66 · 3 May 2018 19:05

Was that before or after the rollback? It still shows redis-4.0.2.

If you uninstalled and reinstalled it might have removed the file and enabled symlinks (steps 3 and 4 from my post above) and needs them re doing perhaps?

Perhaps the php.ini file needs to be edited for your particular environment? I cannot say as I do not use synology or VMs.

It’s not just the OS but a combination of the OS and the Architecture eg there might be a different version for a ARM device running Stretch to a amd64 device running Stretch etc

pb66 · 3 May 2018 19:08

Sorry i have little or no experience with the sync module, I will see what @Trystan comes up with before I wade into that unknown

TrystanLea · 3 May 2018 20:39

@Lykke Could you try setting $homedir in your emoncms settings.php file to reflect your system and try again? If that does not work could you try running sync/sync_run.php directly

sudo php sync_run.php

It might be a path issue.

Lykke · 3 May 2018 21:39

No luck with activating $homedir in settings.php

Running sync_run.php produces this output, guess you are onto something:

lykke@lykke-Standard-PC-i440FX-PIIX-1996:~/sync$ php sync_run.php
SYNC: Starting
PHP Warning: chdir(): No such file or directory (errno 2) in /home/lykke/sync/sync_run.php on line 8
PHP Warning: require(process_settings.php): failed to open stream: No such file or directory in /home/lykke/sync/sync_run.php on line 9
PHP Fatal error: require(): Failed opening required ‘process_settings.php’ (include_path=‘.:/usr/share/php’) in /home/lykke/sync/sync_run.php on line 9

I’ve changed /var/www/emoncms to /var/www/html/emoncms and set the $homedir to my home folder. That helped, but:

lykke@lykke-Standard-PC-i440FX-PIIX-1996:~/sync$ php sync_run.php
SYNC: Starting
SYNC: Home dir: /home/lykke
SYNC: Connected to Redis
{“action”:“download”,“local_id”:4,“remote_server”:“https://emoncms.org”,“remote_id”:xxxxx,“remote_apikey”:“yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy”}
Cannot open local metadata file
{“action”:“download”,“local_id”:3,“remote_server”:“https://emoncms.org”,“remote_id”:xxxxxx,“remote_apikey”:“zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz”}
Cannot open local metadata file

TrystanLea · 4 May 2018 07:51

Looks like your almost there In your last step can you try: sudo php sync_run.php