Multiple problems after upgrade to 9.9.6

I run a stand-alone emoncms on my home server and am having all sorts of problems after upgrading to 9.9.6 this morning. I followed the normal upgrade procedure:

  • pull all of the modules and base code from git
  • switch to the 9.9.6 branch in the base directory
  • run the database upgrader
  • restart php-fpm to ensure I don’t have any cached code

However, while my dashboards load and data is still collected, there are multiple problems.

  • Dashboards have lost some of their widgets, I think this is related to the next issue
  • My feeds are all fubared, the feeds page won’t even load and the inputs show wait… on all of the entries in the process lists. When I try and edit an input any feed log or Power to KWh entries look like Feedid "+feedid+" does not exists or was deleted
  • The MySolar app wants to reinitialize everything probably because of the feed issue

Any idea what might have happened during the update? If I revert back to the 9.8.6 branch MySolar loads and it is getting data, but all of the inputs show “Unsupported” and none of the other dashboards work (likely because the dashboard module updates aren’t compatible with 9.8.6). I haven’t tried restoring the database yet, I figure that is a last resort as I don’t want to risk losing any data that has been collected in the past 30 mins.

One additional piece of data – I see that the feeds have NULL in several of the columns in the database. Is this normal after the upgrade?

mysql> select * from feeds;
+-----+----------------------------------+--------+--------+------------+------------------+----------+--------+------+--------+-------------+--------+------+
| id  | name                             | userid | tag    | time       | value            | datatype | public | size | engine | processList | server | unit |
+-----+----------------------------------+--------+--------+------------+------------------+----------+--------+------+--------+-------------+--------+------+
|   1 | Main A Power                     |      1 | Node 1 | 1486345200 |           628.36 |        1 |      0 | NULL |      6 | NULL        |      0 |      |
|   2 | Main B Power                     |      1 | Node 1 | 1486345200 |           806.45 |        1 |      0 | NULL |      6 | NULL        |      0 |      |

and I see that server and unit are new columns. Should they have been populated with something other than 0 and an empty value?

That’s quite a step back, there are a lot of changes in the last 9.9.5 to 9.9.6 merges, does 9.9.5 work?

What version were you running beforehand? I assume <9.8.6, if it was <9.8 (or there abouts) the dash module was changed and moved out of the main repo, since then it needs installing separately as per the app module and graph module etc,

I assume you tried the usual culprits and updated the database and cleared the browser cache?

I was running 9.8.6 before the upgrade. The database update went fine with no errors and I’ve even tried running in an incognito window to make sure I have nothing cached on the browser side. I’ve checked a couple of different branches, the 9.9.0 tag is the last time the feeds work – 9.9.3 and on show the same issue that I’m having with 9.9.6.

The real issue seems to be the feeds. I have a hunch that if I can figure out what is causing that I will solve the dashboard and MySolar app issues.

Hello @sjthespian, could you try going to the api emoncms/feed/list.json directly? Are there any errors in the result?

Do you see any errors in your browsers console log?

Ok, that is useful – it throws a couple of errors.

<b>Warning</b>:  fopen(/var/lib/phptimeseries/feed_90.MYD): failed to open stream: No such file or directory
<br/>
<b>Warning</b>:  filesize(): stat failed for /var/lib/phptimeseries/feed_90.MYD in <b>/srv/www/htdocs-www/emoncms/Modules/feed/engine/PHPTimeSeries.php</b> on line <b>71</b><br />

which is interesting, as I don’t even have a /var/lib/phptimeseries directory! I just have /var/lib/phpfina/ and /var/lib/phpfiwa/. Any idea when that was added? I just created it, but emoncms isn’t adding any data files to it. Is there a way to populate it from phpfina and phpfiwa?

Glad that found something. Could you repeat your mysql command above but search for phptimeseries feeds:

select * from feeds where engine=2;

Perhaps there’s a feed that did not delete correctly… if its not there then it may be that the redis cache is out of sync with your mysql data, you could clear redis (assuming your not using redis for other applications) with:

redis-cli flushall

emoncms will then reload the content.

That turns up one feed:

mysql> select * from feeds where engine=2;
+----+-----------------+--------+--------+------------+-----------------+----------+--------+------+--------+-------------+--------+------+
| id | name            | userid | tag    | time       | value           | datatype | public | size | engine | processList | server | unit |
+----+-----------------+--------+--------+------------+-----------------+----------+--------+------+--------+-------------+--------+------+
| 90 | Main Power kWhd |      1 | Node 1 | 1469832794 | 6.7553338555542 |        2 |      0 | NULL |      2 | NULL        |      0 |      |
+----+-----------------+--------+--------+------------+-----------------+----------+--------+------+--------+-------------+--------+------+
1 row in set (0.01 sec)

And I just fixed the problem – although I still don’t know what caused it…

touch /var/lib/phptimeseries/feed_90.MYD

That lets the json feed return clean json with no HTML embedded in it, which fixes both my feeds and my dashboards! More importantly, the MySolar app now returns a dashboard (which is more than a bit depressing since it’s raining today and I’m not producing any power. :slight_smile: )

Thanks for the pointers in the right direction!

1 Like

I’d either leave it on stable or master branches.

Can you post the server information by clicking on ‘copy to clipboard’ and then just pasting here (no further formatting required).

Great to hear that fixed it, not sure either what caused it, will keep an eye out for similar issues.

Server Information
Services
redis-server Active Running
Emoncms Version 9.9.6
Modules Administration : App v1.0.0 : Dashboard v1.3.3 : energy : event : EventProcesses : Feed : Input : mqtt : notify : openbem : packetgen : CoreProcess : report : Schedule : Time : User : Visualisation
Git URL: GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data : Branch: master : Describe: 9.9.4-206-gacbfca0
Server OS Linux 4.4.0-141-generic
Host morpheus : morpheus.lapseofthought.com : (172.16.42.16)
Date 2019-02-13 08:38:37 PST
Uptime 08:38:37 up 39 days, 12:30, 1 user, load average: 0.43, 0.66, 0.94
HTTP Server nginx/1.14.1 HTTP/1.1 CGI/1.1 443
MySQL Version 5.7.25-0ubuntu0.16.04.2-log
Host localhost (127.0.0.1)
Date 2019-02-13 08:38:36 (UTC -08:00‌​)
Stats Uptime: 1736004 Threads: 12 Questions: 15936957 Slow queries: 0 Opens: 69486010 Flush tables: 7 Open tables: 416 Queries per second avg: 9.180
Redis Version 3.0.6
Host localhost:6379 (127.0.0.1)
Size
Uptime 39 days
Memory RAM Used: 77.09% Total: 7.69 GB Used: 5.93 GB Free: 1.76 GB
Swap Used: 8.01% Total: 39.89 GB Used: 3.2 GB Free: 36.7 GB
Disk Mount Stats
/ Used: 68.34% Total: 450.08 GB Used: 307.59 GB Free: 119.61 GB
/home Used: 91.71% Total: 750 GB Used: 687.83 GB Free: 62.17 GB
/boot Used: 23.93% Total: 471.6 MB Used: 112.84 MB Free: 334.41 MB
/srv/archive Used: 65.26% Total: 400 GB Used: 261.03 GB Free: 138.97 GB
/srv/backup Used: 80.60% Total: 2 TB Used: 1.61 TB Free: 397.33 GB
/srv/db Used: 1.09% Total: 688.98 GB Used: 7.5 GB Free: 681.48 GB
/srv/images Used: 43.62% Total: 1.18 TB Used: 527.16 GB Free: 681.48 GB
/srv/timemachine Used: 44.76% Total: 1.2 TB Used: 549.95 GB Free: 678.85 GB
/srv/www Used: 11.87% Total: 773.28 GB Used: 91.79 GB Free: 681.48 GB
/usbdisk5g Used: 0.00% Total: 681.48 GB Used: 128 KB Free: 681.48 GB
/usbdisk5g/containers Used: 0.00% Total: 681.48 GB Used: 128 KB Free: 681.48 GB
/usbdisk5g/deleted Used: 0.00% Total: 681.48 GB Used: 128 KB Free: 681.48 GB
/usbdisk5g/deleted/images Used: 0.00% Total: 681.48 GB Used: 128 KB Free: 681.48 GB
/var/lib/lxd/containers/chef-ubuntu.zfs Used: 0.55% Total: 685.27 GB Used: 3.79 GB Free: 681.48 GB
/var/lib/lxd/images/f70ceacb0c15661d8603d041c5c5e94a5de63fbac54803963be700cbed6f721e.zfs Used: 0.09% Total: 682.11 GB Used: 640.63 MB Free: 681.48 GB
/srv/media/books Used: 61.44% Total: 20.94 TB Used: 12.87 TB Free: 8.07 TB
/srv/media/video Used: 61.44% Total: 20.94 TB Used: 12.87 TB Free: 8.07 TB
/srv/media/mp3 Used: 61.44% Total: 20.94 TB Used: 12.87 TB Free: 8.07 TB
PHP Version 7.0.32-0ubuntu0.16.04.1 (Zend Version 3.0.0)
Modules calendar v7.0.32-0ubuntu0.16.04.1 : cgi-fcgi : Core v7.0.32-0ubuntu0.16.04.1 : ctype v7.0.32-0ubuntu0.16.04.1 : curl v7.0.32-0ubuntu0.16.04.1 : date v7.0.32-0ubuntu0.16.04.1 : dom v20031129 : exif v7.0.32-0ubuntu0.16.04.1 : fileinfo v1.0.5 : filter v7.0.32-0ubuntu0.16.04.1 : ftp v7.0.32-0ubuntu0.16.04.1 : gd v7.0.32-0ubuntu0.16.04.1 : gettext v7.0.32-0ubuntu0.16.04.1 : gnupg v1.3.6 : hash v1.0 : iconv v7.0.32-0ubuntu0.16.04.1 : imagick v3.4.0RC6 : imap v7.0.32-0ubuntu0.16.04.1 : intl v1.1.0 : json v1.4.0 : ldap v7.0.32-0ubuntu0.16.04.1 : libxml v7.0.32-0ubuntu0.16.04.1 : mbstring v7.0.32-0ubuntu0.16.04.1 : mcrypt v7.0.32-0ubuntu0.16.04.1 : mysqli v7.0.32-0ubuntu0.16.04.1 : mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $ : openssl v7.0.32-0ubuntu0.16.04.1 : pcre v7.0.32-0ubuntu0.16.04.1 : PDO v7.0.32-0ubuntu0.16.04.1 : pdo_mysql v7.0.32-0ubuntu0.16.04.1 : pdo_sqlite v7.0.32-0ubuntu0.16.04.1 : Phar v2.0.2 : posix v7.0.32-0ubuntu0.16.04.1 : pspell v7.0.32-0ubuntu0.16.04.1 : radius v1.4.0b1 : readline v7.0.32-0ubuntu0.16.04.1 : redis v3.1.1 : Reflection v7.0.32-0ubuntu0.16.04.1 : session v7.0.32-0ubuntu0.16.04.1 : shmop v7.0.32-0ubuntu0.16.04.1 : SimpleXML v7.0.32-0ubuntu0.16.04.1 : snmp v0.1 : soap v7.0.32-0ubuntu0.16.04.1 : sockets v7.0.32-0ubuntu0.16.04.1 : SPL v7.0.32-0ubuntu0.16.04.1 : sqlite3 v7.0.32-0ubuntu0.16.04.1 : standard v7.0.32-0ubuntu0.16.04.1 : sysvmsg v7.0.32-0ubuntu0.16.04.1 : sysvsem v7.0.32-0ubuntu0.16.04.1 : sysvshm v7.0.32-0ubuntu0.16.04.1 : tokenizer v7.0.32-0ubuntu0.16.04.1 : wddx v7.0.32-0ubuntu0.16.04.1 : xml v7.0.32-0ubuntu0.16.04.1 : xmlreader v7.0.32-0ubuntu0.16.04.1 : xmlwriter v7.0.32-0ubuntu0.16.04.1 : xsl v7.0.32-0ubuntu0.16.04.1 : Zend OPcache v7.0.32-0ubuntu0.16.04.1 : zip v1.13.5 : zlib v7.0.32-0ubuntu0.16.04.1
Client Information
HTTP Browser Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Screen Resolution 1920 x 1200
Window Size 1440 x 797

Oh *&^%$$% the fix for the formatting of the information does not seem to have worked :angry:

It sort-of works – it’s there, just a whole lot wider than the display… :slight_smile: My guess it’s the entry with /var/lib/lxd/images/f70ceacb0c15661d8603d041c5c5e94a5de63fbac54803963be700cbed6f721e.zfs throwing off the column width.

@sjthespian, I just tried it for me in the sandbox and it worked which implies you are not actually correctly up to date.

It should look like this Test formatting of copy to clipboard

Can you please checkout the stable branch and the do git status please.

That cut-n-paste was from 9.9.6. I just switched to stable:

> git co stable
Previous HEAD position was acbfca0... Update version.txt
Branch stable set up to track remote branch stable from origin.
Switched to a new branch 'stable'
> git status
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean

And here is a new copy of the server info:

Server Information
Services
redis-server Active Running
Emoncms Version 9.9.6
Modules Administration : App v1.0.0 : Dashboard v1.3.3 : energy : event : EventProcesses : Feed : Input : mqtt : notify : openbem : packetgen : CoreProcess : report : Schedule : Time : User : Visualisation
Git URL: GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data : Branch: * stable : Describe: 9.9.4-205-g22d2e6c
Server OS Linux 4.4.0-141-generic
Host morpheus : morpheus.lapseofthought.com : (172.16.42.16)
Date 2019-02-13 08:55:34 PST
Uptime 08:55:34 up 39 days, 12:47, 1 user, load average: 1.82, 1.15, 1.04
HTTP Server nginx/1.14.1 HTTP/1.1 CGI/1.1 443
MySQL Version 5.7.25-0ubuntu0.16.04.2-log
Host localhost (127.0.0.1)
Date 2019-02-13 08:55:34 (UTC -08:00‌​)
Stats Uptime: 1737021 Threads: 12 Questions: 15944529 Slow queries: 0 Opens: 69526899 Flush tables: 7 Open tables: 416 Queries per second avg: 9.179
Redis Version 3.0.6
Host localhost:6379 (127.0.0.1)
Size
Uptime 39 days
Memory RAM Used: 80.83% Total: 7.69 GB Used: 6.22 GB Free: 1.47 GB
Swap Used: 8.00% Total: 39.89 GB Used: 3.19 GB Free: 36.7 GB
Disk Mount Stats
/ Used: 68.33% Total: 450.08 GB Used: 307.53 GB Free: 119.67 GB
/home Used: 91.77% Total: 750 GB Used: 688.25 GB Free: 61.75 GB
/boot Used: 23.93% Total: 471.6 MB Used: 112.84 MB Free: 334.41 MB
/srv/archive Used: 65.26% Total: 400 GB Used: 261.03 GB Free: 138.97 GB
/srv/backup Used: 80.60% Total: 2 TB Used: 1.61 TB Free: 397.33 GB
/srv/db Used: 1.09% Total: 688.98 GB Used: 7.5 GB Free: 681.48 GB
/srv/images Used: 43.62% Total: 1.18 TB Used: 527.16 GB Free: 681.48 GB
/srv/timemachine Used: 44.76% Total: 1.2 TB Used: 549.95 GB Free: 678.85 GB
/srv/www Used: 11.82% Total: 772.86 GB Used: 91.37 GB Free: 681.48 GB
/usbdisk5g Used: 0.00% Total: 681.48 GB Used: 0 B Free: 681.48 GB
/usbdisk5g/containers Used: 0.00% Total: 681.48 GB Used: 0 B Free: 681.48 GB
/usbdisk5g/deleted Used: 0.00% Total: 681.48 GB Used: 0 B Free: 681.48 GB
/usbdisk5g/deleted/images Used: 0.00% Total: 681.48 GB Used: 0 B Free: 681.48 GB
/var/lib/lxd/containers/chef-ubuntu.zfs Used: 0.55% Total: 685.27 GB Used: 3.79 GB Free: 681.48 GB
/var/lib/lxd/images/f70ceacb0c15661d8603d041c5c5e94a5de63fbac54803963be700cbed6f721e.zfs Used: 0.09% Total: 682.11 GB Used: 640.63 MB Free: 681.48 GB
/srv/media/books Used: 61.44% Total: 20.94 TB Used: 12.87 TB Free: 8.07 TB
/srv/media/video Used: 61.44% Total: 20.94 TB Used: 12.87 TB Free: 8.07 TB
/srv/media/mp3 Used: 61.44% Total: 20.94 TB Used: 12.87 TB Free: 8.07 TB
PHP Version 7.0.32-0ubuntu0.16.04.1 (Zend Version 3.0.0)
Modules calendar v7.0.32-0ubuntu0.16.04.1 : cgi-fcgi : Core v7.0.32-0ubuntu0.16.04.1 : ctype v7.0.32-0ubuntu0.16.04.1 : curl v7.0.32-0ubuntu0.16.04.1 : date v7.0.32-0ubuntu0.16.04.1 : dom v20031129 : exif v7.0.32-0ubuntu0.16.04.1 : fileinfo v1.0.5 : filter v7.0.32-0ubuntu0.16.04.1 : ftp v7.0.32-0ubuntu0.16.04.1 : gd v7.0.32-0ubuntu0.16.04.1 : gettext v7.0.32-0ubuntu0.16.04.1 : gnupg v1.3.6 : hash v1.0 : iconv v7.0.32-0ubuntu0.16.04.1 : imagick v3.4.0RC6 : imap v7.0.32-0ubuntu0.16.04.1 : intl v1.1.0 : json v1.4.0 : ldap v7.0.32-0ubuntu0.16.04.1 : libxml v7.0.32-0ubuntu0.16.04.1 : mbstring v7.0.32-0ubuntu0.16.04.1 : mcrypt v7.0.32-0ubuntu0.16.04.1 : mysqli v7.0.32-0ubuntu0.16.04.1 : mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $ : openssl v7.0.32-0ubuntu0.16.04.1 : pcre v7.0.32-0ubuntu0.16.04.1 : PDO v7.0.32-0ubuntu0.16.04.1 : pdo_mysql v7.0.32-0ubuntu0.16.04.1 : pdo_sqlite v7.0.32-0ubuntu0.16.04.1 : Phar v2.0.2 : posix v7.0.32-0ubuntu0.16.04.1 : pspell v7.0.32-0ubuntu0.16.04.1 : radius v1.4.0b1 : readline v7.0.32-0ubuntu0.16.04.1 : redis v3.1.1 : Reflection v7.0.32-0ubuntu0.16.04.1 : session v7.0.32-0ubuntu0.16.04.1 : shmop v7.0.32-0ubuntu0.16.04.1 : SimpleXML v7.0.32-0ubuntu0.16.04.1 : snmp v0.1 : soap v7.0.32-0ubuntu0.16.04.1 : sockets v7.0.32-0ubuntu0.16.04.1 : SPL v7.0.32-0ubuntu0.16.04.1 : sqlite3 v7.0.32-0ubuntu0.16.04.1 : standard v7.0.32-0ubuntu0.16.04.1 : sysvmsg v7.0.32-0ubuntu0.16.04.1 : sysvsem v7.0.32-0ubuntu0.16.04.1 : sysvshm v7.0.32-0ubuntu0.16.04.1 : tokenizer v7.0.32-0ubuntu0.16.04.1 : wddx v7.0.32-0ubuntu0.16.04.1 : xml v7.0.32-0ubuntu0.16.04.1 : xmlreader v7.0.32-0ubuntu0.16.04.1 : xmlwriter v7.0.32-0ubuntu0.16.04.1 : xsl v7.0.32-0ubuntu0.16.04.1 : Zend OPcache v7.0.32-0ubuntu0.16.04.1 : zip v1.13.5 : zlib v7.0.32-0ubuntu0.16.04.1
Client Information
HTTP Browser Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Screen Resolution 1920 x 1200
Window Size 1440 x 797

Have you forced a page refresh of the admin page - clutching at straws. Only other explanation is Safari doing something odd which I cannot test.

Safari doing something odd… isn’t that a given? :slight_smile:

I did a full shift-reload of the page before posting the new server info. And on my end I’m using Chrome since my safari beachballs more than it works these days (too many JIRA, Service Now and Splunk tabs open).

The point of me asking for the info was that the services mqtt_input & service-runner (as well as feedwriter but I doubt you use that) have changed and if you use them you might be interested as you did a manual upgrade.

I’m not using mqtt with emoncms, although I do use it with a couple other automation services (hassio for example). I don’t believe I’m using feedwriter either.

What I really need to do is move this over to a raspberry pi so i can just use the provided image instead of doing this all manually…

BTW, the example you gave is how it looks to me in chrome. The only issue is that the table is incredibly wide due to some unwrappable items in the first column.

I run mine standalone so it is not usually any more bother. In someways it is less.