Emoncms Graph module developments

Emoncms
1

I’ve done a little more work on the emoncms graph module, also known as the data viewer accessible from the feeds list page. The main additions are:

  • Ability to save graphs and embed these in dashboards
  • Settable plot colour and fill
  • Tooltips
  • Watt hours in window for power feeds
  • Settable yaxis min and max
  • Improved responsive design

I’ve put together a short demo video here of these new additions:

With a bit more work it should be possible to replace the rawdata, bargraph, realtime and multigraph visualisations with graphs created using the graph module and I would like to add in the datapoint editor features into the same interface as well, providing one versatile interface for exploring, editing and exporting emoncms data.

Using the averaging API to get around the feed interval vs post interval issue affecting some feeds

Part of the instigation for the above update however was to provide a way of creating embeddable graphs that used the averaging API selectable using the data viewer. Im aware from several emoncms.org users of the issue where many of the visualisations are showing incomplete data for feeds that used to be recorded as PHPFiwa feeds on emoncms.org, now stored as PHPFina feeds. With the averaging mode enabled it averages across null values in the feed showing a better representation of the underlying data, much in the same way as PHPFiwa did before.

The problem where data appears missing in this way is due to the feed interval being at a more regular interval than the actual data posting interval. The data then saves to disk as a valid value followed by several null values followed by the next valid value. This presents a problem for the standard phpfina data request method that just pulls out a datapoint at regular intervals as the request may miss the valid datapoint and pull out a neighbouring null value instead. The averaging api averages across several datapoints and ignores null values and so if there is at least one valid value in each segment a non null value will be returned.

3 Likes
2

huge work trystan, congratulations!

3

Hi @TrystanLea, excellent work I look forward to trying the new additions, I do use it a fair bit and I have a minor feature request I wonder if you could add to you improvements, I would reaaly like the legend to toggle the plots as is done on the current multigraphs.

I know we can select and deselect feeds from the left hand panel but it would be really useful if we could toggle within the subset of selected feeds without the screen rescaling and/or the legend entries changing order/colour as we select and deselect whilst inspecting certain plots against others, it gets very difficult to focus and the multigraph therefore has some superiority in that area, it would be fantastic to have the one page for all graphing functions, which is where you appear to be heading with this.

I would also like to see “graphing” in the dropdown menu, I often set out to view a group of feeds and to go via the feeds page and select a single feed seems like a backdoor, I would much prefer to go direct to “Graphs” and pick my feeds.

EDIT - Just tried adding “graphs” to the dropdown and it works (woohoo! :raised_hands:)

2016-08-28 (6).png1917×361 43.5 KB

I just created a “graph/graph_menu.php” (that’s for other readers, I’m sure you are more than familiar with how it’s done :slight_smile: )

?php
    $domain = "messages";
    bindtextdomain($domain, "Modules/graph/locale");
    bind_textdomain_codeset($domain, 'UTF-8');
    $menu_dropdown_config[] = array('name'=> dgettext($domain, "Graphs"), 'icon'=>'icon-retweet', 'path'=>"graph" , 'session'=>"write", 'order' => 25 );

I chose priority 25 to put it between feeds and dashboards as that seems a logical order and used the “icon-retweet” as it sort of looks like graph plots (tried the “icon-random” for the same reason but it causes the “Graphs” word to shift left and not line up with the other menu items)

Would you be interested in a PR for this?

1 Like
4

@TrystanLea I have now tried the improved version and think it’s a great improvement, Would you also consider the following minor points.

The list on the left can get a bit long and with feed names reused, very confusing, can the list be broken down into groups of feeds by tag? like many of the current feed selection lists are. I have one site that has literally hundreds of feeds, but they are all named similarly and distinguished as various area’s of that site by tag, so I have at least 12 and upto 36 of each feed name.

Can the tag be included/excluded from the legend, again like the current multigraphs?

The current “graph” legend also currently shows the feed id which maybe useful during creation and investigations etc but the saved and then embedded graphs IMO shouldn’t ideally display the feed id it’s IMO a tad ugly and not useful for general viewing. Perhaps the table at the bottom of the graphs page could have “Feed Id” and “Feed name” , rather than the 2 parsed into one “Feed” column and the Id omitted from the legend?

2016-08-28 (3).png1209×431 21.8 KB

These are not criticisms, I like where you are heading with this, hopefully these very minor points will assist you

1 Like
5

Thanks Bernardino, Paul! Good idea about the clickable legend, I will try and do that. I was thinking that it would be quite nice to have the statistics: min, max, mean, watt hours, stdev shown in a box when you hover on each feed in the legend but wasn’t quite sure how to detect the feed being hovered over but that should be part of the show/hide code in the multigraph thanks to Chaveiro.

Happy to merge a pull request with the menu great!

Happy to remove the id from the legend, I will look into making the tag selectable.
I see your point on sorting the number of feeds in a hierarchy via tags in the sidebar, I reckon that will be the more complicated one to implement… i might have to come back to that one.

6

Great tis’ done!

7

Mixing bars and plots is cool, here is a quick example of a graph (I can now save and embed etc) of the cpu temperature of a Pi installed at a site where the PV inverter is overheating and cooking everything in the plant room including my monitor, so it’s something I check frequently until they fix the overheating.

It would be nice to have some bar width control once we start playing with daily totals etc, The “fill” box isn’t utilized with bars, perhaps that could be also toggle “filling” the report period (not quite 100%, perhaps 90% to provide gaps) to toggle “line bars” or fill out to “interval bars” eg 1 day or 1 hour for the job at hand?

2016-08-28 (7).png1499×463 47.9 KB

8

Great, yes I was wondering whether to make it possible to select different interval periods for each feed. So that its possible to add a daily bargraph but then have a higher resolution temperature feed overlayed, It could then size the bars to the interval of the feed… quite a bit of work to implement though.

I have enabled the toggling of series, I see now its done by jquery.flot.togglelegend.min.js. Which is not quite as easy to then adapt for showing statistics as I suggested above, so might leave that for now.

9

How important is the ability to hide/show the tag in the legend? If it was always visible would that be an issue?

10

4 posts were merged into an existing topic: Bugs in new graph module

11

Personally I find it very useful and feel I would be lost without it, Having looked through some of my multigraphs I use it a fair bit (I have lots of graphs some with some without tags). When graphing a group of items with the same tag or items that only appear once in any account it is preferable to omit the tag as it is ugly, confusing and takes up screen/graph real estate. But equally there are times when it is absolutely vital they are there eg 12 feeds called temperature isn’t much use without the tags to say what area/rooms they are.

Does it prevent other things from working? or is it just the work involved? I think many (myself, very much included) would appreciate the choice and the effort made to provide that.

I would just like to avoid having a graph called “Area1” (could be a room or a building etc) with a legend saying

Area1 Lights power
Area1 Heating power
Area1 Ventilation power
Area1 temperature
etc etc

I have some tags like “RE_block_2_room_4” and “Main_block_DB3_L1” so things could get ugly quite quickly.

If they cannot be toggled I guess it would have to be always included, but that does seem like a big step backwards IMO, sorry!

EDIT - Love the toggling of series though :+1:
At first using the cpu_temp example above it would resize each time I toggled cpu_temp_max but not the other 2, discovered that setting the max scale sorted that issue, I had previously edited min scale to get rid of dead graph area but left max at auto. Works perfectly when both min scale and max scales are set.

14

Not sure if related to this addition, but during the last 2 days I have problems creating dashboards. Almost every type of visualization I add is red with the “Not configured or Authentication not valid” message. At first it only applied to Multigraphs, but this morning I cannot seem to add any graph at all.

Granted, I am not a super user, but I have managed to cobble together some dashboards in the past without any issues. I did a search on the community forum before posting here and did not find any others having the same or similar issues.

15

I have just checked this and can confirm your suspicions, it is this module that is preventing any other visualization from being configured correctly.

@TrystanLea, I have just tried adding a random selection of visualizations to a new dashboard and while the red box appears after selecting a vis’ configuring it does not allow it to display, even if saved and then “view” the vis’ appears unconfigured.

I tried removing the graphs Module and all was restored and working correctly. I then re-cloned the graphs module and lost the issue returned. Although removing or adding the module alone doesn’t change anything unless the page is refreshed so it is something that is loaded when the page gets loaded. The only noticeable difference to that page with or without the module installed is the addition of the “graph” option at the top of the visualization selection dropdown.

I previously had the graph module installed for a couple of months at least and had not noticed any problem before updating yesterday,

16

Thanks @Kristjan_Ulfsson & @pb66, I think I’ve managed to fix the issue, a full page refresh may be needed to download updated javascript.

I will look into adding the option to show/hide the tag :slight_smile:

17

Yep, that got it! thanks @TrystanLea

Much appreciated, I hope it isn’t too difficult to implement,

18

Sorted the toggleable legend, tag and saving of the show/hide missing data field :slight_smile: (top-right hand corner above the graph)

19

Fantastic! works a charm :joy:

Is it possible to add the tag to the “Feed” column of the graphs page too? eg feedid:tag:feedname rather than just feedid:feedname.

If space is an issue perhaps the histogram buttons could be smaller square buttons with “Histogram” written just once at the head of the column.

20

I have just spotted a misleading anomaly that you may need to take a peek at!

Using the same cpu_temp example from above

2016-08-29.png1461×1154 100 KB

.

when we “Show CSV output” the 9x cpu_temp_max and 9x cpu_temp_min values are listed alongside the first 9 cpu_temp datapoints, appearing to give them timestamps spanning 2hrs not 8days.
.

2016-08-29 (1).png1452×1115 71.7 KB

EDIT - I have just noticed something that maybe of value, the number of datapoints and the timestamps seem to be driven by the first selected item on the table ie the top one, using the same example again, by deselecting the yellow cpu_temp trace in the lefthand panel and reselecting it so it effectively moves from the top to the bottom of the table, I now ONLY have the first 9 datapoints of each feed (data subset) included in the CSV output and they are 86400 secs apart.

2016-08-29 (4).png993×1167 69.9 KB

21

Shall I remove the feed id in the table? Perhaps with tag:name its not needed?

22

IMO you should keep it on the work area, I would certainly like to be able to find the feed id, if you remove it could you provide a tooltip/hoverover? There again should it remain visible for assisting users that provide screendumps? I think keep it!

ID access is something I have been meaning to bring up in a general sense too. The old processlist edit page used to include the input ID at the head of the page, now it seems there is no way to obtain an input id without using a input/list api call or adding a “+input” to a processlist and getting the id from hovering over the shorthand processlist buttons on the input page.

And in contrast the feeds page prominently shows all feedids in the 1st column and also sorts by feedid too, so lists become very confused once you start retrospectively adding feeds at a later date. the default sort order should be based on feed name (within the tag groups) and perhaps the column display order should be Name first.

I would like to see less focus on feedids on the feeds page without losing them altogether like the inputs, and bring back the inputid in a discreet but logical location.

EDIT - Sorry slid a bit off topic there, maybe I should start a new thread about the general use of ids and their visibility (or lack of)