This update, brings together a lot of work on the emoncms feed engines and related tools, as well as improvements to menu’s & the admin interface.
I’ve put together a short video of the main changes here:
Visible changes to the end user:
Improved CSV export support to include standard fixed interval and time-zone aligned daily, weekly, monthly and annual intervals (uses new back-end implementation). Option to return result with or without averaging.
New CSV import tool including interval detection available on the feeds page. Efficient import via new backend post_multiple engine method. See: https://github.com/emoncms/emoncms/pull/1752
New interactive feed API documentation page available from the feeds page
Improved support for updating existing data in PHPFina and PHPTimeseries
Support for averaging in rawdata, bargraph and multigraph visualisations
Fixed support for showing last day, week, month, year in visualisations built using delta mode option on cumulative kWh feeds (see below), while some apps implemented this already, this update provides consistent support across all visualisations.
Faster implementation of virtual feeds and data results that are consistent with the underlying source feed data, fixes some slightly strange behaviour seen with virtual feeds that may have arisen due to evolution of emoncms after virtual feeds were first introduced. See: https://github.com/emoncms/emoncms/pull/1748 & https://github.com/emoncms/emoncms/pull/1746
Improvements to the menu system including: animations, shadows and other styling changes, top menu auto hides when scrolling down, thanks to @chaveiro
Admin page improvements including, additional disk stats, improved write load implementation, cpu, machine, bios and current user info, thanks to @chaveiro
Translation work thanks to @thib66
Back-end changes for the interest of developers
Large change to PHPFina, PHPTimeSeries and MysqlTimeSeries feed engine implementation providing a single method for requesting feed data with options to specify returning the result as fixed interval, timezone aligned daily, weekly, monthly, annual, with or without averaging and output formats json or csv. This reduces the amount of code duplication required to provide these outputs with separate methods. Focus is on data request behaviour consistency across all options, enabling easier cross feed comparison and data manipulation in emoncms graphs, apps and outside of emoncms via CSV export.
Switch to using server side calculation of kWh/d, month, year from cumulative kWh feeds (delta mode) - fixes missing current day, week, month when using delta mode across all visualisations and removes code required to otherwise achieve the same result in each app. See: https://github.com/emoncms/emoncms/pull/1761
php7.0 compatibility mode for remember me, thanks to @mabi
support for sendmail in email.php, thanks to @federicobriata
New post_multiple methods in phpfina and phptimeseries engines for more efficient csv import, these methods replace the bulk buffer prepare and save and post methods reducing duplication. Improves support for data updates for these engines. E.g can import blocks of data via new CSV import tool to replace existing data in the feed.
Thanks for the update, I installed this last weekend & noticed the animations on the menu.
I had to re-create (i.e. de-select, then reselect the data points) all of my graphs, they appeared to be configured correctly however the data was not correctly displayed, it was as though there was some issue with the y axis values. One annoying side effect of this was that all of my custom line colours were deleted and I ended up with the pre-assigned colours because once a custom line colour is set, the values cannot be viewed (i.e. the RGB values are all zero).
Hopefully you can resolve this issues from my vague notes so others are not effetced.
I also just updated to 11.0.5 and am no longer able to load data into a feed with timestamps into the future. I use a Solcast forecast and was successfully loading a five day forecast using the feed API:
I did have to edit the file at
/var/www/emoncms/Modules/feed/engine/PHPFina.php
to allow this.
Since I have upgraded to 11.0.5, the feeds do not appear to accept data with a timestamp that is in the future. The API call does not return any error and the Emoncms log is also silent, but no future data is visible on any graphs. Can anyone identify how to get the feeds in V11 to accept timestamps in the future?
Hello @Jaddache I cant seem to replicate the issue that you are seeing with posting data in the future here, it seems to work fine for me and I’ve been doing similar with solcast with no apparent issue.
Are you seeing this on a raspberryPi based system? are you using redis and the feedwriter service etc?
Might be worth trying a restart of the feedwriter service…
I’ve updated the emoncms app module to fix an issue with dataplicity which was blocking the config.js script, see: Dataplicity Wormhole & EmonCMS Errors - #10 by TrystanLea. @MyForest this might affect your heat pump app? Just needs a filename change from config.js to appconf.js.
Tristan, it would have been the previous stable release. I know there’s not a lot of info to help you, I just accepted it as a feature and fixed my graphs. Then yesterday saw this in my weekly email topic update.
One other issue was that on this and the previous stable release I’ve had difficulty downloading a backup with the download link not working using the Edge browser. Eventually after many attempts of creating & downloading, after a backup creation I right clicked and opened in a new tab which then successfully downloaded a file, though I didn’t check to see if it was valid data because I don’t know to check.
Yes it is a raspberry pi based system using the emonSD-08May21 image.
Further investigation reveals it is only one particular feed, a daily time series that I was using to display daily solcast kWhr totals. The feed with the 30min solcast forecast is working and displaying as expected.
I probably need to go back to my actual desired outcome which is to display historical actual solar and consumption kWh totals and forecast daily solar kWh totals on a single bar chart with the bars covering the period of 00:00 - 00:00 local time. I use nodered to source and load the solcast 30min forecast into a feed out to 5 days ahead - this is still working with 11.0.5. I calculate the associated daily kWh forecast totals along the way and was loading these into a separate feed (this is the one that does not appear to be accepting new data). The historical daily totals are calculated using a Power to kWh/d input process.
I created a new feed (86400 sec interval) and used the new data import functionality to copy the values from the problematic feed (this works really well by the way). This new feed did initially accept the 4 days of future values which I graphed successfully 30 min ago but now the future values are no longer visible on a graph.
Update: Rebooted the pi, swapped to another browser and cleared out all its caches this morning and the new feed now appears to be accepting data and displaying future data in graphs.
