Emoncms v11 master branch testing before merge to stable (merged)

TrystanLea · 4 January 2022 14:28

Update: v11 has now been released as stable, see: Emoncms v11 stable release

Hello all!

This update, currently in the master branch for testing, brings together a lot of work over recent months on the emoncms feed engines and related tools, as well as improvements to menu’s & the admin interface.

I’ve put together a short video here of the main changes:

Visible changes to the end user:

Improved CSV export support to include standard fixed interval and timezone aligned daily, weekly, monthly and annual intervals (uses new backend implementation). Option to return result with or without averaging.
PHPTimeSeries averaging support
Feeds can now be created directly on the feeds page (via an extended version of the virtual feed creation dialog). See: https://github.com/emoncms/emoncms/pull/1751
New CSV import tool including interval detection available on the feeds page. Efficient import via new backend post_multiple engine method. See: https://github.com/emoncms/emoncms/pull/1752
New interactive feed API documentation page available from the feeds page
Improved support for updating existing data in PHPFina and PHPTimeseries
Support for averaging in rawdata, bargraph and multigraph visualisations
Fixed support for showing last day, week, month, year in visualisations built using delta mode option on cumulative kWh feeds (see below), while some apps implemented this already, this update provides consistent support across all visualisations.
Faster implementation of virtual feeds and data results that are consistent with the underlying source feed data, fixes some slightly strange behaviour seen with virtual feeds that may have arisen due to evolution of emoncms after virtual feeds were first introduced. See: https://github.com/emoncms/emoncms/pull/1748 & https://github.com/emoncms/emoncms/pull/1746
Wider time format support when requesting data both in terms of request parameters and format of the timestamps returned, see: https://github.com/emoncms/emoncms/pull/1767
Improvements to the menu system including: animations, shadows and other styling changes, top menu auto hides when scrolling down, thanks to @chaveiro
Admin page improvements including, additional disk stats, improved write load implementation, cpu, machine, bios and current user info, thanks to @chaveiro
Translation work thanks to @thib66

Backend changes for the interest of developers

Large change to PHPFina, PHPTimeSeries and MysqlTimeSeries feed engine implementation providing a single method for requesting feed data with options to specify returning the result as fixed interval, timezone aligned daily, weekly, monthly, annual, with or without averaging and output formats json or csv. This reduces the amount of code duplication required to provide these outputs with separate methods. Focus is on data request behaviour consistency across all options, enabling easier cross feed comparison and data manipulation in emoncms graphs, apps and outside of emoncms via CSV export.
Related updates to API documentation, feed.js library, apps, dashboard and graph module.
Refactoring of visualisations and apps to use new feed API, feed.js and vis.helper.js, improving code reuse and reducing use of legacy code.
Removed deprecated graph visualisation (replaced with graph module), backwards compatibility fix included.
Switch to using server side calculation of kWh/d, month, year from cumulative kWh feeds (delta mode) - fixes missing current day, week, month when using delta mode across all visualisations and removes code required to otherwise achieve the same result in each app. See: https://github.com/emoncms/emoncms/pull/1761
Removal of feed datatype field (deprecated). See: https://github.com/emoncms/emoncms/pull/1736
php7.0 compatibility mode for remember me, thanks to @mabi
support for sendmail in email.php, thanks to @federicobriata
New post_multiple methods in phpfina and phptimeseries engines for more efficient csv import, these methods replace the bulk buffer prepare and save and post methods reducing duplication. Improves support for data updates for these engines. E.g can import blocks of data via new CSV import tool to replace existing data in the feed.
Removal of separate insert_data and update_data feed_model methods, both insert and update can now be done via post method. See: https://github.com/emoncms/emoncms/pull/1769

Help with testing of the latest master branch including latest master for all modules would be much appreciated. I will post a second post below with the steps to do this.

Thanks all!

Github v11 pull request https://github.com/emoncms/emoncms/pull/1762

TrystanLea · 4 January 2022 14:34

If you would like to test v11 while it’s in the master branch for testing, the steps are as follows:

Make sure that you are up to date with the main updates on the Admin > Update page (click on Full update)

Navigate to Admin > Components and click on the yellow Master button at the top:

Once complete restart the emoncms_mqtt and feedwriter services on the System info page:

That should be it, the latest version should be displayed as 11.0.4 (as of the time of this post) in the emoncms footer.

mjr · 4 January 2022 16:25

Seems to work for me, but I had to adjust one of my graphs because one of the lines on it kept returning to zero between measurement points. After removing and re-adding that feed, it seems to work fine again. Anything I should try to break?

mabi · 4 January 2022 16:38

Is redis needed to update from Admin?

mabi · 4 January 2022 16:41

Thanks for the update! Just a documentation fix: my (mabi) small contribution was on “php7.0 compatibility mode for remember me” not on sendmail support.

JJC · 4 January 2022 17:30

@TrystanLea

should that not be 11.0.4

TrystanLea · 4 January 2022 18:23

Thanks @mjr ok good to know that fixed it, cant think of things specifically but Im sure issues will surface through wider testing

TrystanLea · 4 January 2022 18:24

@mabi yes redis is needed and sorry I will amend above, thanks again!

Robert.Wall · 5 January 2022 17:24

I’ve got a problem with some of the colours - it’s not new, but is carried over from V10.7.4

The sub-menu item, as in Graph, and when the light sidebar is chosen, is only barely visible when not highlighted (on my LCD laptop screen anyway). It is invisible when highlighted.

A question: In this screenshot

Screenshot 2022-01-05 16:05:49

what is the name of the highlighted Node under the cursor (without cheating by putting the image into Gimp and enhancing it)?

To my (nearly 73 years old) eyes, even the 3 nodes above don’t have sufficient contrast to be easy to read, and “My Graph” in white is almost invisible.

Have you used one of the readability tools to evaluate things like this against WCAG guidelines, for example? These suggest a contrast ratio of 3.5:1.
Using this contrast grid I think the ‘no mouse-over’ ratio is about 1.5, and highlighted it is below 1.1. “My Graphs” comes out at about 1.2.

[Hint: Type each colour in use - background, text, highlighted text - into the text box “Rows & Columns”] and then look up the contrast ratio at the intersection of the appropriate row & column.]

TrystanLea · 6 January 2022 10:57

Thanks @Robert.Wall I will have a look at that.

NA7KR · 11 January 2022 00:34

My inputs are not updating now

on emonpi

I have reboot

> service-runner trigger sent for '/var/www/emoncms/scripts/serialmonitor/start.sh 38400 /dev/ttyAMA0'
> |emonPiCM V1.0.0
> |OpenEnergyMonitor.org
> LCD found i2c 0x27
> OK 5 0 0 7 0 7 0 182 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 15 106 0 0 0 40 43 24 5 253 255 0 0 8 0 36 3 0 0 255 255 255 255 0 0 0 0 2 0 0 0 48 117 48 117 48 117 0 0 0 0 (-64)
> OK 5 0 0 7 0 7 0 184 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 1 0 4 0 5 0 109 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 1 0 4 0 5 0 90 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 82 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 80 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 15 109 0 0 0 34 43 85 8 253 255 0 0 76 0 212 2 0 0 255 255 255 255 0 0 0 0 39 0 0 0 48 117 48 117 48 117 0 0 0 0 (-75)
> OK 15 109 0 0 0 7 43 26 5 253 255 255 255 7 0 47 3 0 0 255 255 255 255 0 0 0 0 2 0 0 0 48 117 48 117 48 117 0 0 0 0 (-66)
> OK 5 2 0 4 0 6 0 78 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 83 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 78 89 163 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 60 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 86 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 15 111 0 0 0 4 43 23 5 253 255 254 255 7 0 54 3 0 0 255 255 255 255 0 0 0 0 2 0 0 0 48 117 48 117 48 117 0 0 0 0 (-65)
> OK 5 2 0 4 0 6 0 86 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 89 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 15 112 0 0 0 11 43 15 5 253 255 255 255 7 0 57 3 0 0 255 255 255 255 0 0 0 0 2 0 0 0 48 117 48 117 48 117 0 0 0 0 (-66)

Edit - Formatted text. BT, Moderator

MyForest · 11 January 2022 00:43

Hi @TrystanLea

Looks pretty good, I can see many of the new features working as described.

As you know I have a pretty hacky Docker container that looks like this. I added the BRANCH section so I could do this testing:

FROM openenergymonitor/emoncms

RUN rm -rf /var/www/emoncms
RUN mkdir /var/www/emoncms

ENV BRANCH master

RUN git clone --branch ${BRANCH} https://github.com/emoncms/emoncms.git /var/www/emoncms
WORKDIR /var/www/emoncms/Modules

RUN git clone --branch ${BRANCH} https://github.com/emoncms/dashboard.git
RUN git clone --branch ${BRANCH} https://github.com/emoncms/graph.git
RUN git clone --branch ${BRANCH} https://github.com/emoncms/app.git
RUN git clone --branch ${BRANCH} https://github.com/emoncms/device.git
RUN git clone --branch ${BRANCH} https://github.com/emoncms/postprocess.git

The only change I needed to make was to use vis.helper.js from the Lib:

Thanks

David

NA7KR · 11 January 2022 06:48

Got it working but set 110v and reporting 220v why?

Problem was datacodes = h, h, h, h, h change to datacodes = h

Volts are working on EMONTX ok


[interfacers]
##
[[SerialTx0]]
     Type = EmonHubTx3eInterfacer
      [[[init_settings]]]
           com_port= /dev/ttyUSB0
           com_baud = 115200
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,
          nodeoffset = 8
[[SerialTx1]]
     Type = EmonHubTx3eInterfacer
      [[[init_settings]]]
           com_port= /dev/ttyUSB1
           com_baud = 115200
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,
          nodeoffset = 9
[[SerialTx2]]
     Type = EmonHubTx3eInterfacer
      [[[init_settings]]]
           com_port= /dev/ttyUSB2
           com_baud = 115200
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,
          nodeoffset = 10

[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        com_baud = 38400                        # 9600 for old RFM12Pi
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,
        group = 210
        frequency = 433
        baseid = 5                              # emonPi / emonBase nodeID
        calibration = 110V                      # (UK/EU: 230V, US: 110V)
        quiet = true                            # Disable quite mode (default enabled) to enable RF packet debugging, show packets which fail crc
        # interval =  300                         # Interval to transmit time to emonGLCD (seconds)
     
[[MQTT]]

    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 127.0.0.1
        mqtt_port = 1883
        mqtt_user = emonpi
        mqtt_passwd = emonpimqtt2016

    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,

     
       node_format_enable = 1
       node_format_basetopic = emonhub/

       
        nodevar_format_enable = 1
        nodevar_format_basetopic = emon/




[nodes]


[[5]]
    nodename = emonpi
    [[[rx]]]
       names = HotTub1,HotTub2,power1pluspower2,vrms,t1
       datacode = h 
       scales = 1,1,1,0.01,0.01
       units = W,W,W,V,C
[[8]]
    nodename = emontx3cm15
    [[[rx]]]
       names = MSG,Vrms, Grid1, Solar1, Air1, Garage
       scales = 0,1,1,1,1,1
       units = n,V,W,W,W,W

[[9]]
  nodename = emontx3cm16
    [[[rx]]]
       names = MSG,Vrms, Grid2, Solar2, Air2, Furnace
       scales = 0,1,1,1,1,1
       units = n,V,W,W,W,W

[[10]]
  nodename = emontx3cm17
    [[[rx]]]
       names = MSG,Vrms, Dryer1, Dryer2, Washer, Microwave
       scales = 0,1,1,1,1,1
       units = n,V,W,W,W,W

[For future reference, when posting code or output, please put 3 ‘backticks’ (normally found at the top left of the keyboard) on a line of their own before the code, and 3 more backticks also on a line of their own after the code:

```
code
```

If it is something like php you can add a language identifier after the first 3 backticks: ```php or even ```text if you don’t want any language markup applied. - Moderator (RW) ]

NA7KR · 11 January 2022 18:37

Looks calibration = 110V is not work now
1st no power about 0
2nd power connecred 204v

How do Change it 110v?

OK 5 4 3 28 3 32 6 28 0 226 4 48 117 48 117 48 117 48 117 48 117 3 0 0 0 0 0 0 0 43 40 0 0 13 39 0 0 (-0)
OK 5 204 1 199 1 147 3 164 89 226 4 48 117 48 117 48 117 48 117 48 117 3 0 0 0 0 0 0 0 43 40 0 0 14 39 0 0 (-0)

I see in log:
MainThread Setting RFM2Pi calibration: 110V (2p)

NA7KR · 11 January 2022 18:58

Found ther problem:

Change back to JEELIB and show 110v

Robert.Wall · 11 January 2022 22:35

With the new emonPiCM software “Native RFM69 (New)” that you chose, there is no need for the complicated and inflexible calibration = 110V, you can calibrate the voltage exactly in emonCMS using Setup →Admin → Serial Monitor, using that to send calibration commands to the emonPiCM in exactly the same way as you would with a programmer and laptop to an emonTx.

The documentation is part of the emonPiCM release: The emonPiCM

Thelmike · 12 January 2022 10:14

Until now only 1 thing: in the dashboard, in element chart of type zoom the 5 and 1 minute zooms did not show a graph…

TrystanLea · 12 January 2022 20:30

Thanks @Thelmike I can replicate that bug with the realtime visualisation, will have a look at what’s causing the issue, thanks!

MyForest · 13 January 2022 01:25

@TrystanLea I have an ugly bug that I’ve found but it’s probably not gating because there’s a workaround.

The masthead sometimes dissappears when I go back to my phone after a while.

That’s clearly unhelpful. I can’t reproduce it, so I can’t tell you if it’s device specific, time related or some other thing. If I were to randomly guess I’d think it was most likely related to the browser snoozing the page and the content being mistakenly dropped.

It still holds the same space on the page, so it’s not being removed from the DOM.

I’m on:

Fairphone 4
Android 11
Firefox Beta
AdGuard AdBlocker
uBlock Origin

I didn’t see this behaviour a week ago or ever before that. Unfortunately since then I’ve changed my phone and my EmonCMS version to the master branch.

I’ve tried to reproduce on my old phone but seeing as I can’t purposefully reproduce it anywhere I don’t know if it’s just not appearing because of that.

I have seen it about five times now. I can always fix it by refreshing the page.

Specifically it’s the div with class menu-top bg-menu-top that is going completely white.

Aha, it looks like it’s related to scrolling and it not re-appearing when you scroll back up. I can now reproduce it reliably and on multiple mobile devices, but not on desktop. It might be related to this change, I’m investigating:

Yep, if I back out the changes to menu.css it stays broken but if I also back out menu.js the menu stays in place when you scroll (as you’d expect) and never goes white.

Of course I’m seeing this because I’m working on mobile with the side menus hidden. On desktop with the side menus hidden I see the menu transition back from the top a fraction of a second (yes, probably 0.3s) after I scroll back to the top.

@nchaveiro I’ve tried to see what I could tweak in menu.js but it’s a bit more than I can absorb as I’m not a front-end developer. I think I’d pretty much just be turning it all on / off which isn’t useful information for you.

Here’s my client info to help (I hope):

Ooh, here’s another client where it’s tripping up: