Emoncms v11 master branch testing before merge to stable (merged)

[TrystanLea]

Update: v11 has now been released as stable, see: Emoncms v11 stable release

Hello all!

This update, currently in the master branch for testing, brings together a lot of work over recent months on the emoncms feed engines and related tools, as well as improvements to menu’s & the admin interface.

I’ve put together a short video here of the main changes:

Visible changes to the end user:

• Improved CSV export support to include standard fixed interval and timezone aligned daily, weekly, monthly and annual intervals (uses new backend implementation). Option to return result with or without averaging.

• PHPTimeSeries averaging support

• Feeds can now be created directly on the feeds page (via an extended version of the virtual feed creation dialog). See: https://github.com/emoncms/emoncms/pull/1751

• New CSV import tool including interval detection available on the feeds page. Efficient import via new backend post_multiple engine method. See: https://github.com/emoncms/emoncms/pull/1752

• New interactive feed API documentation page available from the feeds page

• Improved support for updating existing data in PHPFina and PHPTimeseries

• Support for averaging in rawdata, bargraph and multigraph visualisations

• Fixed support for showing last day, week, month, year in visualisations built using delta mode option on cumulative kWh feeds (see below), while some apps implemented this already, this update provides consistent support across all visualisations.

• Faster implementation of virtual feeds and data results that are consistent with the underlying source feed data, fixes some slightly strange behaviour seen with virtual feeds that may have arisen due to evolution of emoncms after virtual feeds were first introduced. See: https://github.com/emoncms/emoncms/pull/1748 & https://github.com/emoncms/emoncms/pull/1746

• Wider time format support when requesting data both in terms of request parameters and format of the timestamps returned, see: https://github.com/emoncms/emoncms/pull/1767

• Improvements to the menu system including: animations, shadows and other styling changes, top menu auto hides when scrolling down, thanks to @chaveiro

• Admin page improvements including, additional disk stats, improved write load implementation, cpu, machine, bios and current user info, thanks to @chaveiro

• Translation work thanks to @thib66

Backend changes for the interest of developers

• Large change to PHPFina, PHPTimeSeries and MysqlTimeSeries feed engine implementation providing a single method for requesting feed data with options to specify returning the result as fixed interval, timezone aligned daily, weekly, monthly, annual, with or without averaging and output formats json or csv. This reduces the amount of code duplication required to provide these outputs with separate methods. Focus is on data request behaviour consistency across all options, enabling easier cross feed comparison and data manipulation in emoncms graphs, apps and outside of emoncms via CSV export.

  • https://github.com/emoncms/emoncms/pull/1743
  • https://github.com/emoncms/emoncms/pull/1745
  • https://github.com/emoncms/emoncms/pull/1754
  • https://github.com/emoncms/emoncms/pull/1771

• Related updates to API documentation, feed.js library, apps, dashboard and graph module.

• Refactoring of visualisations and apps to use new feed API, feed.js and vis.helper.js, improving code reuse and reducing use of legacy code.

• Removed deprecated graph visualisation (replaced with graph module), backwards compatibility fix included.

• Switch to using server side calculation of kWh/d, month, year from cumulative kWh feeds (delta mode) - fixes missing current day, week, month when using delta mode across all visualisations and removes code required to otherwise achieve the same result in each app. See: https://github.com/emoncms/emoncms/pull/1761

• Removal of feed datatype field (deprecated). See: https://github.com/emoncms/emoncms/pull/1736

• php7.0 compatibility mode for remember me, thanks to @mabi

• support for sendmail in email.php, thanks to @federicobriata

• New post_multiple methods in phpfina and phptimeseries engines for more efficient csv import, these methods replace the bulk buffer prepare and save and post methods reducing duplication. Improves support for data updates for these engines. E.g can import blocks of data via new CSV import tool to replace existing data in the feed.

• Removal of separate insert_data and update_data feed_model methods, both insert and update can now be done via post method. See: https://github.com/emoncms/emoncms/pull/1769

Help with testing of the latest master branch including latest master for all modules would be much appreciated. I will post a second post below with the steps to do this.

Thanks all!

Github v11 pull request https://github.com/emoncms/emoncms/pull/1762

[TrystanLea]

If you would like to test v11 while it’s in the master branch for testing, the steps are as follows:

1. Make sure that you are up to date with the main updates on the Admin > Update page (click on Full update)

2. Navigate to Admin > Components and click on the yellow Master button at the top:

3. Once complete restart the emoncms_mqtt and feedwriter services on the System info page:

That should be it, the latest version should be displayed as 11.0.4 (as of the time of this post) in the emoncms footer.

[mjr]

Seems to work for me, but I had to adjust one of my graphs because one of the lines on it kept returning to zero between measurement points. After removing and re-adding that feed, it seems to work fine again. Anything I should try to break?

[mabi (Marco Bisioli)]

Is redis needed to update from Admin?

[mabi (Marco Bisioli)]

Thanks for the update! Just a documentation fix: my (mabi) small contribution was on “php7.0 compatibility mode for remember me” not on sendmail support.

[JJC (John Cabra)]

@TrystanLea

That should be it, the latest version should be displayed as 10.0.4 (as of the time of this post) in the emoncms footer.

@TrystanLea

should that not be 11.0.4

[TrystanLea]

Thanks @mjr ok good to know that fixed it, cant think of things specifically but Im sure issues will surface through wider testing

[TrystanLea]

@mabi yes redis is needed and sorry I will amend above, thanks again!

[Robert.Wall]

I’ve got a problem with some of the colours - it’s not new, but is carried over from V10.7.4

The sub-menu item, as in Graph, and when the light sidebar is chosen, is only barely visible when not highlighted (on my LCD laptop screen anyway). It is invisible when highlighted.

A question: In this screenshot

what is the name of the highlighted Node under the cursor (without cheating by putting the image into Gimp and enhancing it)?

To my (nearly 73 years old) eyes, even the 3 nodes above don’t have sufficient contrast to be easy to read, and “My Graph” in white is almost invisible.

Have you used one of the readability tools to evaluate things like this against WCAG guidelines, for example? These suggest a contrast ratio of 3.5:1.
Using this contrast grid I think the ‘no mouse-over’ ratio is about 1.5, and highlighted it is below 1.1. “My Graphs” comes out at about 1.2.

[Hint: Type each colour in use - background, text, highlighted text - into the text box “Rows & Columns”] and then look up the contrast ratio at the intersection of the appropriate row & column.]

[TrystanLea]

Thanks @Robert.Wall I will have a look at that.

[NA7KR (Kevin Roberts)]

My inputs are not updating now

on emonpi

I have reboot

> service-runner trigger sent for '/var/www/emoncms/scripts/serialmonitor/start.sh 38400 /dev/ttyAMA0'
> |emonPiCM V1.0.0
> |OpenEnergyMonitor.org
> LCD found i2c 0x27
> OK 5 0 0 7 0 7 0 182 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 15 106 0 0 0 40 43 24 5 253 255 0 0 8 0 36 3 0 0 255 255 255 255 0 0 0 0 2 0 0 0 48 117 48 117 48 117 0 0 0 0 (-64)
> OK 5 0 0 7 0 7 0 184 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 1 0 4 0 5 0 109 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 1 0 4 0 5 0 90 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 82 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 80 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 15 109 0 0 0 34 43 85 8 253 255 0 0 76 0 212 2 0 0 255 255 255 255 0 0 0 0 39 0 0 0 48 117 48 117 48 117 0 0 0 0 (-75)
> OK 15 109 0 0 0 7 43 26 5 253 255 255 255 7 0 47 3 0 0 255 255 255 255 0 0 0 0 2 0 0 0 48 117 48 117 48 117 0 0 0 0 (-66)
> OK 5 2 0 4 0 6 0 78 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 83 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 78 89 163 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 60 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 86 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 15 111 0 0 0 4 43 23 5 253 255 254 255 7 0 54 3 0 0 255 255 255 255 0 0 0 0 2 0 0 0 48 117 48 117 48 117 0 0 0 0 (-65)
> OK 5 2 0 4 0 6 0 86 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 5 2 0 4 0 6 0 89 89 176 4 48 117 48 117 48 117 48 117 48 117 2 0 0 0 0 0 0 0 237 3 0 0 201 3 0 0 (-0)
> OK 15 112 0 0 0 11 43 15 5 253 255 255 255 7 0 57 3 0 0 255 255 255 255 0 0 0 0 2 0 0 0 48 117 48 117 48 117 0 0 0 0 (-66)

Edit - Formatted text. BT, Moderator

[MyForest (David Bowen)]

Hi @TrystanLea

Looks pretty good, I can see many of the new features working as described.

As you know I have a pretty hacky Docker container that looks like this. I added the BRANCH section so I could do this testing:

FROM openenergymonitor/emoncms

RUN rm -rf /var/www/emoncms
RUN mkdir /var/www/emoncms

ENV BRANCH master

RUN git clone --branch ${BRANCH} https://github.com/emoncms/emoncms.git /var/www/emoncms
WORKDIR /var/www/emoncms/Modules

RUN git clone --branch ${BRANCH} https://github.com/emoncms/dashboard.git
RUN git clone --branch ${BRANCH} https://github.com/emoncms/graph.git
RUN git clone --branch ${BRANCH} https://github.com/emoncms/app.git
RUN git clone --branch ${BRANCH} https://github.com/emoncms/device.git
RUN git clone --branch ${BRANCH} https://github.com/emoncms/postprocess.git

The only change I needed to make was to use vis.helper.js from the Lib:

Enable compatability with EmonCMS 11

Thanks

David

[NA7KR (Kevin Roberts)]

Got it working but set 110v and reporting 220v why?

Problem was datacodes = h, h, h, h, h change to datacodes = h

Volts are working on EMONTX ok

[interfacers]
##
[[SerialTx0]]
     Type = EmonHubTx3eInterfacer
      [[[init_settings]]]
           com_port= /dev/ttyUSB0
           com_baud = 115200
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,
          nodeoffset = 8
[[SerialTx1]]
     Type = EmonHubTx3eInterfacer
      [[[init_settings]]]
           com_port= /dev/ttyUSB1
           com_baud = 115200
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,
          nodeoffset = 9
[[SerialTx2]]
     Type = EmonHubTx3eInterfacer
      [[[init_settings]]]
           com_port= /dev/ttyUSB2
           com_baud = 115200
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,
          nodeoffset = 10

[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        com_baud = 38400                        # 9600 for old RFM12Pi
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,
        group = 210
        frequency = 433
        baseid = 5                              # emonPi / emonBase nodeID
        calibration = 110V                      # (UK/EU: 230V, US: 110V)
        quiet = true                            # Disable quite mode (default enabled) to enable RF packet debugging, show packets which fail crc
        # interval =  300                         # Interval to transmit time to emonGLCD (seconds)
     
[[MQTT]]

    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 127.0.0.1
        mqtt_port = 1883
        mqtt_user = emonpi
        mqtt_passwd = emonpimqtt2016

    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,

     
       node_format_enable = 1
       node_format_basetopic = emonhub/

       
        nodevar_format_enable = 1
        nodevar_format_basetopic = emon/




[nodes]


[[5]]
    nodename = emonpi
    [[[rx]]]
       names = HotTub1,HotTub2,power1pluspower2,vrms,t1
       datacode = h 
       scales = 1,1,1,0.01,0.01
       units = W,W,W,V,C
[[8]]
    nodename = emontx3cm15
    [[[rx]]]
       names = MSG,Vrms, Grid1, Solar1, Air1, Garage
       scales = 0,1,1,1,1,1
       units = n,V,W,W,W,W

[[9]]
  nodename = emontx3cm16
    [[[rx]]]
       names = MSG,Vrms, Grid2, Solar2, Air2, Furnace
       scales = 0,1,1,1,1,1
       units = n,V,W,W,W,W

[[10]]
  nodename = emontx3cm17
    [[[rx]]]
       names = MSG,Vrms, Dryer1, Dryer2, Washer, Microwave
       scales = 0,1,1,1,1,1
       units = n,V,W,W,W,W

[For future reference, when posting code or output, please put 3 ‘backticks’ (normally found at the top left of the keyboard) on a line of their own before the code, and 3 more backticks also on a line of their own after the code:

```
code
```

If it is something like php you can add a language identifier after the first 3 backticks: ```php or even ```text if you don’t want any language markup applied. - Moderator (RW) ]

[NA7KR (Kevin Roberts)]

Looks calibration = 110V is not work now
1st no power about 0
2nd power connecred 204v

How do Change it 110v?

OK 5 4 3 28 3 32 6 28 0 226 4 48 117 48 117 48 117 48 117 48 117 3 0 0 0 0 0 0 0 43 40 0 0 13 39 0 0 (-0)
OK 5 204 1 199 1 147 3 164 89 226 4 48 117 48 117 48 117 48 117 48 117 3 0 0 0 0 0 0 0 43 40 0 0 14 39 0 0 (-0)

I see in log:
MainThread Setting RFM2Pi calibration: 110V (2p)

[NA7KR (Kevin Roberts)]

Found ther problem:

Change back to JEELIB and show 110v

[Robert.Wall]

@NA7KR

Change back to JEELIB and show 110v

With the new emonPiCM software “Native RFM69 (New)” that you chose, there is no need for the complicated and inflexible calibration = 110V, you can calibrate the voltage exactly in emonCMS using Setup →Admin → Serial Monitor, using that to send calibration commands to the emonPiCM in exactly the same way as you would with a programmer and laptop to an emonTx.

The documentation is part of the emonPiCM release: The emonPiCM

[Thelmike (Michiel Nieuwstraten)]

Until now only 1 thing: in the dashboard, in element chart of type zoom the 5 and 1 minute zooms did not show a graph…

[TrystanLea]

Thanks @Thelmike I can replicate that bug with the realtime visualisation, will have a look at what’s causing the issue, thanks!

[MyForest (David Bowen)]

@TrystanLea I have an ugly bug that I’ve found but it’s probably not gating because there’s a workaround.

The masthead sometimes dissappears when I go back to my phone after a while.

That’s clearly unhelpful. I can’t reproduce it, so I can’t tell you if it’s device specific, time related or some other thing. If I were to randomly guess I’d think it was most likely related to the browser snoozing the page and the content being mistakenly dropped.

It still holds the same space on the page, so it’s not being removed from the DOM.

I’m on:

Fairphone 4
Android 11
Firefox Beta
AdGuard AdBlocker
uBlock Origin

I didn’t see this behaviour a week ago or ever before that. Unfortunately since then I’ve changed my phone and my EmonCMS version to the master branch.

I’ve tried to reproduce on my old phone but seeing as I can’t purposefully reproduce it anywhere I don’t know if it’s just not appearing because of that.

I have seen it about five times now. I can always fix it by refreshing the page.

Specifically it’s the div with class menu-top bg-menu-top that is going completely white.

Aha, it looks like it’s related to scrolling and it not re-appearing when you scroll back up. I can now reproduce it reliably and on multiple mobile devices, but not on desktop. It might be related to this change, I’m investigating:

Auto hide top menu

Yep, if I back out the changes to menu.css it stays broken but if I also back out menu.js the menu stays in place when you scroll (as you’d expect) and never goes white.

Of course I’m seeing this because I’m working on mobile with the side menus hidden. On desktop with the side menus hidden I see the menu transition back from the top a fraction of a second (yes, probably 0.3s) after I scroll back to the top.

@nchaveiro I’ve tried to see what I could tweak in menu.js but it’s a bit more than I can absorb as I’m not a front-end developer. I think I’d pretty much just be turning it all on / off which isn’t useful information for you.

Here’s my client info to help (I hope):

Ooh, here’s another client where it’s tripping up: