Emoncms.org: editrealtime visualization nearly unusable (feature disabled)

[iov (IanV)]

I am seeing strange thing in emoncms.org using the editrealtime visualization.

1. The timestamps of editpoints within a 10s time frame change depending on the zoom!
2. If you set a point it can be ignored and the current value also gets updated.
3. If you set a point it can be accepted and the current value also gets updated
4. You sometimes cannot set a point. But setting the next point sets the one you could not!

If I correct a historic value I then have to correct for any setting of current values. A bit of a pain for accumulating feeds to say the least. As you try to correct further ‘set of current value’ can occur.

Hopefully useful feedback that can help identify what is going wrong. Anyone got a tip that gets this visualization behaving properly?

[TrystanLea]

@IanV editrealtime and editdaily both need a lot of work to bring them back to being useful tools on emoncms.org due to changes to the way data is handled as part of handling emoncms.org scaling. I actually thought I had disabled these to avoid confusion, so have done that for now.
Sorry to not be able to reply more positively with a solution to this one

Trystan

[iov (IanV)]

Thanks for the speedy reply.

I would say that sorting basic single point edit, so no delete or multiply feature, would be useful. It was the only way to set a feed value without needing to understand the API. I will have to adjust Kwh accumulators using API when they do not quite match my meters. Not sure all users will be comfortable with API calls, get it wrong and the feed is broken!

May I request a rework to a safe single point editor please, perhaps with a guaranteed set current value feature. Thanks for all the work.

[miq]

@iov

I would say that sorting basic single point edit, so no delete or multiply feature, would be useful

+1

I was playing with the firmware of my DIY emonTx today and made mistakes causing it sending 0 (zero) values to emoncms.org. After I fixed the mistakes, I quickly looked for the “Edit Realtime” at the visualization page at emoncms.org to change feed values but there is no more choice for ‘Edit Realtime’, and I found out from this post that that feature has been deleted .

But I was able to fix the wrong feed values using the feed API - yup - one by one. Lucky me I only had to change a few dozens of data.
But I prefer using the ‘Edit Realtime’ if it is available

Cheers,

[TrystanLea]

@Iov @miq I will add datapoint editing to the emoncms.org priority list alongside sorting the input api to return error messages on invalid input.

[overeasy (Inactive)]

@TrystanLea

I will add datapoint editing to the emoncms.org priority

This is an important issue. I did commercial scale data systems for decades. I saw things go from mainframes, to minis, to desktops, and now back to mainframes (cloud servers). I’ve seen many a well designed versatile software system fall under it’s own weight by trying to be what it is not and be all things to all people. This may sound like a rant, but it’s time to stop and think about what this eMonCMS thing is.

The current design is elegant and flexible. It accepts data in real time, or a stream of historical data in chronological order, and stores that information for later retrieval. If I read the history correctly, the data structures have evolved to leverage rapid retrieval over a broad spectrum of the timeline.

I appreciate the problems related here with correcting inaccuracies in the historical data, but my sense is that comes with a cost to the integrity of the design, the performance of the system. It may seem like an oversight, a bug, but it isn’t. It’s the fundamental design that precludes using it as a database that can be updated historically. The only oversight was not documenting the perhaps unintended consequence as the storage engines evolved to their present emphasis on retrieval.

A few weeks ago I had an interaction with Trystan on basically this same topic. He told me historical update doesn’t work, and after thinking about that for awhile, I’m fine with that. I’ve got a storage engine on the IoTaWatt that tries to accomplish pretty much the same things as eMonCMS. Likewise, changing history is a huge can of worms.

If there is a source of more accurate data than what was originally posted, then the feed can be deleted and the data sent again, in chronological order, to rebuild the feed with the corrected information. In the end, a simple do-over may prove easier for all involved.

@miq

made mistakes causing it sending 0 (zero) values

@miq

I was able to fix the wrong feed values using the feed API - yup - one by one.

This isn’t normal usage and I question whether it needs to be accommodated.

[miq]

Hi Trystan and Bob,

Thanks, really appreciate the hard work behind the scene in making emoncms.org available for everyone. I’ve been using emoncms.org service for more than a year now, and been a happy user. I wish I could help in a better way, but I am just a standard user and just starting out with programming (since Arduino become very popular).

@overeasy

This isn’t normal usage and I question whether it needs to be accommodated.

Do you mean to accommodate the datapoint editing? It was there once (via the “Edit Realtime” on visualisation page), so it was accommodated, but now it’s gone.

[overeasy (Inactive)]

@miq

It was there once (via the “Edit Realtime” on visualisation page), so it was accommodated, but now it’s gone.

You didn’t say when that was, and I’m too new at this to know when it might have gone away. Looking at the history in the archives, there were a lot of changes three to four years ago. There are also several different formats that complicate the issues. Changing the entries in a straight fixed interval series isn’t my concern. What troubles me is are the implications for the series with averaging, and a series with cumulative data.

In the simplest case of a straight cumulative data series, changing a year old entry in a one minute interval file would necessitate updating the subsequent half-million records. Do it twice - a million. If the roll forward is batched to do it once, the protocol needs to be changed to group transactions. How does a query know if a roll forward is in progress on such a file? If it knows, how does it tell you?

If you change an entry in a simple power feed, but te input process list that supplies that feed goes on to post the same power to a Kwh (cumulative) feed, does the Kwh feed need to be updated by implication?

I’m just asking if holding Trystan’s feet to the fire to preserve arcane functionality may cost more than it’s worth. I don’t see a big need to this on a regular basis. This is, after all, history. It’s not supposed to change. We are dealing with fault correction.

@overeasy

If there is a source of more accurate data than what was originally posted, then the feed can be deleted and the data sent again, in chronological order, to rebuild the feed with the corrected information. In the end, a simple do-over may prove easier for all involved.

[iov (IanV)]

Some great points. Edit is not normal usage.

My experience is that it is useful to be able to set the current value of a feed. I have Kwh feeds that are ideally the same as the meter readings, they drift so occasionally need the current value setting. I have to accept the disturbance caused… My solar will no longer display that days totals correctly for example.

I can see a case for deleting feed data before a date. If would be a huge server burden to have to do a mass load of the history into a new feed to truncate data. Mass load is perhaps not an activity emoncms.org could support.

I use commercial services where if the data goes bad - tough. You live with it or throw away and start again. I can see they use a clear model with none of the ‘allow edits’ issues kindly detailed earlier…

I appreciate emoncms.org for having a flexible approach. Recognising the high complexity involved it may only be sensible to support limited use cases for edit(set of current feed value) and delete(truncate) of feed data in emoncms.org GUI. Perhaps a more manageable ask.

The GUI editrealtime on emoncms.prg was removed a few days ago as it was flawed. I will use an API call to adjust KWh feeds (recognising emoncms makes no historic adjustments/re-calculations) - to replace its use for adjustments. It will be much harder to do.

Most of the emoncms.org edit features may need to be deprecated as being too burdensome. What it does and does not do may need to me made clearer for local emoncms install users.

Thanks for the discussion. Hopefully useful Trystan!

[TrystanLea]

Thanks @overeasy @iov @miq, good points. Especially about the potential issues with averaging, cumulative data and effect on feeds further down a process list. I appreciate the call for a pause to ask whether this is something that we need to be supporting @overeasy.

I was thinking of the case of a spike that might be useful to remove. But there is then the danger as you well say @overeasy that an editing function that would be fine for removing a spike would actually need to then be much more than just a single datapoint editor in order to deal with the other editing use-cases.

I will think throught this some more and reply properly once I’ve had some more time to think through the options and investigate what is involved.

Spikes could be removed with the postprocess module which is a module I would like to come back to and make available on emoncms.org.

The other side of the coin is to add fault checking to the monitoring hardware so that data is only posted to emoncms if within a valid range.

[sroof (Steve Roof)]

Hi miq and others -
Can you describe how you used the API to correct wrong values? I’ve tried sending this sort of post:
http://emoncms.org/input/post.json?time=1491307200&node=9&apikey=XXX&json=ch1:2.0

but it does not replace the wrong value with the desired value. Is there a specific API command to delete or replace values?
Thanks!

[sroof (Steve Roof)]

I found it! This works! (from Emoncms - site api)
To update a single data point in feed ID = 1695, I just entered this line in my browser address bar and hit enter:
https://emoncms.org/feed/update.json?id=1695&time=1491307500&node=9&apikey=xxxx&value=2.0

[nomad-fr]

Hi, in my case, my power graph swith to negative because I plug my clip-on current sensor CT on the wrong side.

So I wanted to apply a ‘x -1’ on a range of data.

So I write a scrip based on : usefulscripts/remove_spike.php at master · emoncms/usefulscripts · GitHub

<?php

$dir = "/var/lib/phpfina/";

function usage($scriptname)
{
  print $scriptname ." usage: \n";
  // print $scriptname ." -i feedid -n minvalue -x maxvalue\n";
  print $scriptname ." -i feedid -n minvalue\n";	
}

// arguments parameters
$opts = getopt("i:n:x:");

// Handle command line arguments
foreach (array_keys($opts) as $opt)
  switch ($opt)
  {
    // feed id
    case 'i': $id = $opts['i'];  break;
    // min value
    case 'n': $min = $opts['n']; break;
  }

// check all parameters are good
if (isset($id) && isset($min))
{
  print "Feed ID   : ".$id  ."\n";
  print "Min value : ".$min ."\n";
}
// Error display usage
else
{
  usage($argv[0]);
  exit(1);
}

for ($n=0; $n<4; $n++)
{
    // if (file_exists($dir.$id."_$n.dat"))
    if (file_exists($dir.$id.".dat"))	
    {
	$npoints = floor(filesize($dir.$id.".dat") / 4.0);	
	$fh = fopen($dir.$id.".dat","c+");
        for ($i=0; $i<$npoints; $i++)
        {
            $tmp = unpack("f",fread($fh,4));
            $val = $tmp[1];
            if (!is_nan($val))
            {
                if ($val<$min)
                {
		    $newval = $val * -1;
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",$newval));
                }
            }
        }
        fclose($fh);
    }
}

Now I’m looking for a way to correct my Power To Kwh day, but it seems to be a little more complicated.