Missing data handling in “delta” option of graphs

Emoncms
1

Hi all,

I have an index feed that stopped working for 3 month, so that the data looks something like:

35, null, … , null, 473, 474, …

When I put this on a graph with the “delta” option checked, I get a 473 high spike where the index restarts. This ruins my dashboards and it isn’t easy to fix.

Wouldn’t it be more logical to consider that

473 - null = null

… or could we at least get a UI option to enable this?

Thanks!
Franck

2

Hello Franck, while the delta option is designed for use with feeds created with the power_to_kwh or wh_accumulator input process which both join across missing data filling in these null values, your suggestion sounds like it could work. I have created an issue on the emoncms github to look at it: Check for null values when using delta mode for graphs · Issue #1331 · emoncms/emoncms · GitHub

3

Thanks Trystan!

4

The issue is in fact affecting many more feeds than I had realized.
I am sending index information from my electronic utility meter at a 10s interval, and unavoidably about 10% of the values are missing.
When I graph that feed with the “delta” option checked I get those CSV values:

timestamp, index, irrelevant
1562490170, 2.0, null
1562490180, 1.0, null
1562490190, null, null
1562490200, 11648190.0, null
1562490210, 2.0, null
1562490220, 1.0, null
1562490230, 1.0, null
1562490240, 1.0, null
1562490250, 2.0, null
1562490260, null, null
1562490270, null, null
1562490280, 11648199.0, null
1562490290, 2.0, null
1562490300, 1.0, null
1562490310, 1.0, null
1562490320, 1.0, null
1562490330, 1.0, null
1562490340, null, null
1562490350, 11648207.0, null
1562490360, 1.0, null
1562490370, 1.0, null
1562490380, null, null
1562490390, 11648212.0, null
1562490400, 1.0, null

I will explore the input processors to see if there is a way to log the last input value to a feed when the input goes missing… but if input processing is triggered by the input being updated I assume this won’t be possible?

5

Hi Trystan,

A simple patch to graph.js does the trick for me locally; it is however not backward compatible (though I can’t really see who would benefit from the old behavior…).

The old logic was arbitrarily updating a value[i] to 0 when value[i-1] was missing, which created the jumps.

Let me know what you think!
Franck

Changed_logic_to_compute_feed_deltas_so_that_delta_is_null_if_either_value_is_missing_.zip (686 Bytes)

6

Hi Franck, The best route here is to raise a PR (Pull Request) in GitHub speak. Are you familiar with GitHub - it can be intimidating but is reasonably simple for small issues like this, and can be done via the web interface.

7

Hmm, yes I am but I don’t have the privileges to create a branch (tried yesterday):

D:\emoncms\graph>git push --set-upstream origin bugfix/deltas-on-missing-values
Password for 'https://[email protected]@github.com':
remote: Permission to emoncms/graph.git denied to franck102.
fatal: unable to access 'https://github.com/emoncms/graph.git/': The requested URL returned error: 403

Do you have any naming patterns for branches (bugfix/xxx, feature/xxx)?

Franck

8

If it is just a change in one file, the simplest way is to just edit that file in GitHub and it will create a branch for you automatically as you submit it (it will warn you you cannot edit the repository directly).

Once the changes have been accepted and merged, go back and delete the branch.

If this needs a number of files to be changed, I use GitHub for windows, which works remarkably well. If you do go down this route,

  • First take a fork of the Emoncms repository to your GitHub account.
  • Clone that fork to your local machine (with GitHub for windows).
  • Create a new branch.
  • Make your changes.
  • Push the changes to GitHub
  • Create the Pull Request (PR) from GitHub for your changes.

It is called a PR as you are requesting the maintainers of the repository to ‘pull’ in your changes. This is slightly counter intuitive initially!

The other thing I found helpful when trying to get my head around Git, was what upstream, origin, and local means.

  • upstream - this is the original repository (so emoncms/emooncms)
  • origin - this is your fork of the upstream repository (YourName/emoncms)
  • local - this is the clone (usually of the origin) you are actually making changes to.

Took me a while to understand this!

[edit]
What I don’t do, is run my live system off my forked repo - it easily gets out of sync. I keep a repo specifically for making changes and overwrite those files to live to test. Once doen I reset the ‘live’ system back to the main emoncms repo.

9

The only thing I had missed was creating my own fork… I just submitted the PR.

Franck

2 Likes
10

Thankyou @franck102, I’ve merged the pull request.

1 Like