Query recently measured values

Derb · 2020-03-10T23:08:34.706Z

Hello,

i wanted to create a bash script, that runs every x seconds and calculates the average of all the measured power values in the last minute.

When playing around with the feed API , i noticed that ‘/feed/data.json’ and ‘/feed/average.json’ don’t give me the most recent values.
I assume, the reason for this is, that these commands only access data, that has already been written to the SD card?

So my idea was to run a script every 10 seconds, that gathers the last value and adds it to a rotating logfile on a ramdisk (/tmp). Or maybe it would be better to get the last value every 5 seconds and only store it if it’s changed, so I’m sure not to miss any updates.
Then I would make another script hat runs every 30 seconds to gather the last 6 values from my file and calculate the average.

Maybe somebody has a better solution for this, that im not aware of?

Thanks and best regards

Bill.Thomson · 2020-03-11T00:18:34.915Z

Sounds like one of these is what you’re looking for.

image1060×194 8.66 KB

The complete list is on the Feed API page of emonCMS. (v10.2.0)
The screenshot was taken from a local copy, not emoncms.org

Derb:

So my idea was to run a script every 10 seconds

What are you going to use to run your Bash script at 10 second intervals?

Derb · 2020-03-11T07:59:22.077Z

Thanks for your reply.

Yes, I already tested the /feed/value.json feature and it works.
I was just thinking If there was maybe another (cleaner) way to get all the values at once, whitout creating my own buffer.

What are you going to use to run your Bash script at 10 second intervals?

Ok, i see, this won’t work with cronjobs, so I guess it has to be a script that runs continuously.

Bill.Thomson · 2020-03-11T16:54:28.360Z

Derb:

this won’t work with cronjobs,

Right. The minimum interval when using cron is one minute.

Derb:

I guess it has to be a script that runs continuously.

I use a Python script to read my meter every 5 seconds.
Because I have 5 jobs that run at inervals ranging from 5 seconds to once per month,
I use a Python module called Advanced Python Scheduler.

But, it sounds like you don’t have that level of complexity.
Here’s a thread that gives several examples of using Python to run jobs at an inverval of your choice:

stackoverflow.com

How to repeatedly execute a function every x seconds?

python, timer

asked by davidmytton on 09:07PM - 23 Jan 09 UTC

Two stand out. In particular, this one for simplicity:

import time
starttime=time.time()
while True:
  print "tick"
  time.sleep(60.0 - ((time.time() - starttime) % 60.0))

and this one, for the following features:

Exception handling: As far as possible on this level, exceptions are handled properly, i. e. get logged for debugging purposes without aborting our program.
No chaining: The common chain-like implementation (for scheduling the next event) you find in many answers is brittle in the aspect that if anything goes wrong within the scheduling mechanism ( threading.Timer or whatever), this will terminate the chain. No further executions will happen then, even if the reason of the problem is already fixed. A simple loop and waiting with a simple sleep() is much more robust in comparison.
No drift: Keeps an exact track of the times it is supposed to run at. There is no drift depending on the execution time (as in many other solutions).
Skipping: Will skip tasks if one execution took too much time (e. g. do X every five seconds, but X took 6 seconds). This is the standard cron behavior (and for a good reason). Many other solutions then simply execute the task several times in a row without any delay. For most cases (e. g. cleanup tasks) this is not wished. If it is wished, simply use next_time += delay instead.

import time, traceback

def every(delay, task):
  next_time = time.time() + delay
  while True:
    time.sleep(max(0, next_time - time.time()))
    try:
      task()
    except Exception:
      traceback.print_exc()
      # in production code you might want to have this instead of course:
      # logger.exception("Problem while executing repetitive task.")
    # skip tasks if we are behind schedule:
    next_time += (time.time() - next_time) // delay * delay + delay

def foo():
  print("foo", time.time())

every(5, foo)

djh · 2020-03-11T17:00:16.602Z

In a bash script it is normal to simply use the sleep command to wait for a few seconds.

Bill.Thomson · 2020-03-11T17:03:20.795Z

The problem with that is you end up with a job that runs with 5 seconds between jobs,
vice every five seconds.

From the thread referenced above:

Actually this is not the answer : time sleep() can only be used for waiting X seconds after every execution. For example , if your function takes 0.5 seconds to execute and you use time.sleep(1) , it means your function executes every 1.5 seconds , not 1.

djh · 2020-03-11T17:07:48.859Z

It’s not clear that running at exactly five seconds intervals is a requirement.

Most calculations won’t take anywhere near the half-second mentioned in the thread. If they did then one can approximate by sleeping 4.5 seconds.

Bill.Thomson · 2020-03-11T17:13:13.074Z

If he’s going to feed the data to emonCMS, and that appears to be the case, viz:

Derb:

When playing around with the feed API , i noticed that ‘/feed/data.json’ and ‘/feed/average.json’ don’t give me the most recent values.

it does indeed need to be at accurately spaced intervals.
Here’s why:

EmonLibCM - Version 2 (Support) emonTx

I’m not the right person to answer his, however my perception is the mechanism is as follows: Incoming data is dropped into a “slot” depending on its arrival time. While the drift between the sending clock and the receiving clock remains negligible (and of course the ideal starting situation is when the incoming data arrives exactly in the middle of its slot), all is well. If the drift is consistent in one direction - say the sending clock runs slow, the data will arrive late and eventually fal…

djh · 2020-03-11T17:19:21.849Z

Bill.Thomson:

If he’s going to fed the data to emonCMS, and that appears to be the case

I’m not sure why making use of APIs to get data implies he will also use APIs to post data? And the description of the purpose gives no hint that is the case either.

Bill.Thomson:

it does indeed need to be at accurately spaced intervals.

That’ll be why emonTx DON’T post at exactly ten second intervals then??? And for that matter don’t have any mechanism to ensure their clocks are synchronised with NTP such as the raspi uses.

Perhaps we should wait for @Derb to clarify rather than argue about how many angels are on the head of this particular pin.

Bill.Thomson · 2020-03-11T17:23:56.022Z

djh:

I’m not sure why making use of APIs to get data implies he will also use APIs to post data? And the description of the purpose gives no hint that is the case either.

OK, fair enough.

djh:

That’ll be why emonTx DON’T post at exactly ten second intervals then???

Correct. But, the recommendation is at steady i.e. non-drifiting, intervals just short of 10 seconds.

Derb · 2020-03-11T18:16:35.737Z

Well, basically it’s very simple.

I want to create an early alert system, that warns me, when the average power consumption within the last minute exceeds a certain limit.

Setup:
Raspberry Pi, running the most recent emonSD image.
Power usage will be measured by an emonTx, which sends the data to the RPi over serial connection (10 sec interval)

The problem:
The API commands /feed/average.json and /feed/data.json don’t give me the most recent values. (I assume they only return values, that have already been stored to the SD card)

My idea for a workaround:
So my workaround would be, to have a script running, that continuously retrieves the last value+timestamp by calling the API command /feed/timevalue.json.
This command always provides the most recent value, even if it’s not yet saved to disk.

Whenever there is a changed timestamp, i add the value to a buffer.
Then i calculate the average for the last minute using the last 6 values from my buffer.

I’m basically just asking, if there is an easier way to do it, that i am not aware of, as i don’t know emonCMS very well. Maybe there is a better approach to retrieve the measurements from a specific interval.

borpin · 2020-03-11T18:34:22.390Z

Derb:

I’m basically just asking, if there is an easier way to do it, that i am not aware of, as i don’t know emonCMS very well. Maybe there is a better approach to retrieve the measurements from a specific interval.

I’d be inclined to feed the data into Node-Red or Home Assistant (via MQTT) for this sort of automation.

You can decrease the buffering in emoncms Feedwriter so the data comes through more quickly. The downside is more writes to the SD Card.

Derb · 2020-03-11T21:03:48.790Z

Thanks, the Node-Red aproach looks interesting, i will play around a bit with it.

Just to be sure, your second sentence is not related to node-red, but to the standard API right?
So, the Node-Red/MQTT solution would not require more writes to the SD card, which is definitely something im trying to avoid.

borpin · 2020-03-11T21:58:08.577Z

Yes - 2 approaches.

The length of the buffer Feedwriter uses is set in the settings.ini.

If you create a publish to MQTT process on the emonCMS Input, it is pushed out immediately so no delay in NR getting it.

Derb · 2020-03-12T09:19:15.580Z

Thanks for the hint with Node-Red. I set it up and it works like a charm.

I am using the “smooth” function calculating the mean value for the last 6 and 360 values.
In fact i need two averages, one over the last minute and one for the last hour.

I will also use NR to send the information to my Media-PC Pi, which will then trigger an acoustic alarm via GPIO.

Thank you very much.