Failed upgrade 8.5 to 9.8.30

I just upgraded my pi’s emoncms from low-write-v8.5 to 9.8.30. I did this by “gitting” the latest version from github. I followed the upgrade instructions from there.

However, I notice the web page is now substantially different. And the problem is - I have no “inputs.” My feeds are there, and show data (i.e. the sizes are not zero). But, I cannot display them, and they show no new data coming in, even though emonhub logs and phpfina data files show updates are coming in. I don’t see anything odd in /var/log/…, although I rebooted a couple of times.

What do I do now?

Have you updated the database files via the button on the emoncms admin page?

Be aware that there were significant changes to the dashboard module it now needs to be installed seperately and many if not all your dashboards will need revisting to reselect the feeds and re-save etc.

If you are running redis, you may need to flush that too, either via the button on the admin page or via the commandline

sudo redis-cli FLUSHALL

More ifo might be useful if the above doen’t help.

Where is this hosted? Where do the inputs that no longer update originate?

Thanks for your help.

In answer to your questions, and also a bit of additional information.

This is hosted on the RasPi that has been serving as an EmonBase for several years. EmonCms was working (version 8.5) on that. It was and still is receiving inputs from emonTx nodes as I write this, and they show up in the phpfina data files. I did update the database as you suggest. I also just did the sudo redis-cli FLUSHALL, followed by a reboot.

At this point, no inputs show on Setup->Inputs. All of my historic feeds exist under “NoGroup”. One other feed exists by itself All feeds show 0 bytes of data and N/A for Updated.

I cloned the SD card for the upgrade, so I can still go back to the prior state.

If I were to do that, can you say what steps I should take?

Again, thanks.

John

BTW… the reason I want to upgrade is that the emoncms version I have has long standing bugs in its graphing.

Sorry John, but no I am not familiar enough to guide you, I am just aware there are significant differences and maybe able to chuck in a couple of scraps of vaguely useful info here and there.

Are you able to log in to emoncms ok? if so can you add your “server information” from the admin page here?

Can you also try an input api to see if a new input can be created or updated from the browser, if not, are there any error messages given in the browser console (ctrl-f12)?

I assume you are running emonhub, if so are there any error messages logged there or does emoncms reply “ok”?

Is there any error messages displayed in the log panal of the admin page?

Have you reviewed settings.php? I have no doubt there will be differences between your original copy and the latest copy.

Server Information
Emoncms Version 9.8.30 : 2018.05.08
Modules Administration : EventProcesses : Feed : Input : CoreProcess : Schedule : Time : User : Visualisation
Server OS Linux 3.18.11+
Host emonpi emonpi (127.0.1.1)
Date 2018-05-27 20:24:39 UTC
Uptime 20:24:39 up 6 min, 1 user, load average: 0.59, 0.42, 0.21
HTTP Server Apache/2.2.22 (Debian) HTTP/1.1 CGI/1.1 80
MySQL Version 5.5.43-0+deb7u1
Host localhost (127.0.0.1)
Date 2018-05-27 20:24:38 (UTC 00:00‌​)
Stats Uptime: 63779 Threads: 1 Questions: 418 Slow queries: 0 Opens: 56 Flush tables: 1 Open tables: 49 Queries per second avg: 0.006
Memory RAM Used: 18.22% Total: 435.36 MB Used: 79.32 MB Free: 356.03 MB
Disk Mount Stats
/ Used: 85.84% Total: 2.61 GB Used: 2.24 GB Free: 223.65 MB
/boot Used: 33.93% Total: 55.95 MB Used: 18.98 MB Free: 36.96 MB
/home/pi/data Used: 2.88% Total: 24.83 GB Used: 732.04 MB Free: 22.85 GB
PHP Version 5.4.41-0+deb7u1 (Zend Version 2.4.0)
Modules apache2handler : bcmath : bz2 : calendar : Core v5.4.41-0+deb7u1 : ctype : curl : date v5.4.41-0+deb7u1 : dba : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.2.1 : libxml : mbstring : mcrypt : mhash : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.1 : posix : redis v2.2.7 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.4.41-0+deb7u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : zip v1.11.0 : zlib v2.0 :

Client Information
HTTP Browser Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:60.0) Gecko/20100101 Firefox/60.0
Screen Resolution 3840 x 2160
Window Size 1198 x 1067

I just tried the input API. I used JSON-like. It worked - it created three inputs. I have now deleted them since they were test only.

I did spot a log in the emonhub.log, though:

2018-05-27 20:33:21,101 DEBUG RFM2Pi 132 adding frame to buffer => [1527453201, 9, 27, 2, 2697, 12, 5.23, 32.6, 0, 0, 0, 0, 0, 1, -43]
2018-05-27 20:33:21,104 DEBUG RFM2Pi 132 Sent to channel’ : ToEmonCMS
2018-05-27 20:33:21,149 WARNING emoncmsorg emoncmsorg couldn’t send to server, URLError: timed out

That suggests that it isn’t able to send to my local EmonCMS via the mqtt messaging system.

However, that doesn’t explain emoncms’ failure to see any data in the existing feeds.

I think the architecture is like this. Correct?

Remote Device --[http]->emonhub->mqtt->emonCms <—> mySql. So it would appear that my mySql tables are messed up?

With just a quick look at that, It appears neither low-write mode or redis are enabled so I would check your settings.php as suggested above. and it looks like there is no dashboard module installed, also mentioned above.

I got here by following the upgrade steps from 8.5. Would that have failed to do as you suggest?

redis-server is running. The previous installation was low-write mode.

I am looking at settings.php but I don’t see how to enable low-write or redis in it - do you have a link? Ditto on dashboard module installation.

My prior copy/paste apparently didn’t get the full PHP Modules list.

Here: apache2handler | bcmath | bz2 | calendar | Core v5.4.41-0+deb7u1 | ctype | curl | date v5.4.41-0+deb7u1 | dba | dom v20031129 | ereg | exif v1.4 | fileinfo v1.0.5 | filter v0.11.0 | ftp | gettext | hash v1.0 | iconv | json v1.2.1 | libxml | mbstring | mcrypt | mhash | mysql v1.0 | mysqli v0.1 | openssl | pcre | PDO v1.0.4dev | pdo_mysql v1.0.2 | Phar v2.0.1 | posix | redis v2.2.7 | Reflection | session | shmop | SimpleXML v0.1 | soap | sockets | SPL v0.2 | standard v5.4.41-0+deb7u1 | sysvmsg | sysvsem | sysvshm | tokenizer v0.1 | wddx | xml | xmlreader v0.1 | xmlwriter v0.1 | zip v1.11.0 | zlib v2.0 |

No, there’s no mention of mqtt there. But it does look like you are running an emonSD image if you are yusing MQTT and emonhub “emonpi variant”, that’s a very different kettle of fish. You are updating just a part of an ecosystem in which all parts have evolved together, I do not think just changing bits of it will work.

I think the mqtt_input and buffered write and the way MQTT is used in gerenral has changed, In fact you may be using the old nodes module that is now retired.

The settings.php file will be your old file you need to look at the emonpi.default.settings.php file as a template and put your specific settings into it from your old settings.php file, creating a new settings.php file based on the new template. But that is just a small part of the things you need to change.

I would recommend going back to your old image temporarily and do some reading up on updating from a 2015 image to the latest emonSD. It will most likely involve backing up your data and creating a brand new SD image to import your data to.

Tks. I didn’t realize it had changed that much. I started with this back in July 2015, IIRC, so it is indeed old. And I am using emonBase, not emonPi although I guess that doesn’t matter much. Also, I don’t feed the .org site - I just use the data on my pi.

I think that backing it up and moving that data into a brand new installation probably is the best bet, as you suggest. I’ll take a look at it. If the SQL schema has changed, then hopefully there are scripts that will let me use the old data without writing my own SQL.

A bit more.

I am back on the working system, where I can graph data and receive new data.

I am peeking at the mysql database. It does not have my data! It appears that my data is all in phpfina directory in nn.dat files.

If I upgrade, what do I do with these files? Does the data ever end up in the database, or stay in these files?

The data storage is explained in Learn | Electricity Monitoring | Timeseries.

As far as I’m aware, the MySQL database is used only for the metadata. The ‘real’ data is in the phpfina files.

Thanks. That seems to be it. Any idea why my updated version didn’t just pick up the phpfina data and use it?

No idea, I’m afraid. I don’t know the inner workings of emonCMS.