If I store the incoming data hourly - which makes sense as the data only changes on the hour - and then apply the rate_of_change input process I get zeros. Why is that?
“Unix timestamp”, “32:Gas_RATEFIXED”, “32:Gas_PHPFIXED”
1605505740, null, null
1605509340, 0, 24081.2
1605512940, 0, 24081.2
1605516540, 0, 24081.8
1605520140, 0, 24082.3
Yes indeed that is the issue, which you didn’t previously share with us.
If you wrote the interfacer then you should know how to set the read/send interval.
Did you use any other interfacer as a base or start from scratch? I have no idea whether the current emonpi emonhub version still has the code or not, but it did used to have a generic “interval” setting, if it is still intact, then setting interval = 3600 in the runtimesettings of your custom interfacer should slow that down to 1 read and send per hour.
This would make sense if it were that simple, but unfortunately that isn’t the case. This isn’t really fixable in emoncms whilst there is 10s data arriving where only 1 in 360 packets is valid each hour. The likelihood of that one good packet being the one packet that is saved each time an hour passes is slim, possible with some careful timing etc, but slim and probably fragile too. emoncms is not looking for a good value in each hour, it is saving whatever the value is when the hour passes, so with 359 values in 360 being 0, there is a good chance the recorded value will be 0.
You must slow the rate down, essentially your data is (incorrectly) suggesting that no gas is used for 59mins and 50secs and all the gas used since the last hourly data, has in fact been used in the last 10s. emoncms cannot undo that error, how could it possibly know which are fake zeros and actual zeros? If you were consuming 24/7 then a simple “skip if zero” process could allow only the >0 values through, but if (for example) you use no gas overnight then use loads in the first hour in the morning, emoncms will believe that gas has in fact been used over the period since the last non-zero value the night before and the consumption per hour will be way out.
Hi Paul,
thank you for your detailed explanations. That makes sense. What I previously failed to understand is that even if I store the data hourly the input processing still takes place every 10s, leading to that hit and miss situation that you described. It might be useful if there would be some kind of separate, analytical postprocessor in emoncms which runs at a different rate than the input processor.
This is unfortunately quite challenging because apart from the gas meter data I am also getting lots of other data that changes every 10 seconds from the same data telegram. So the data needs to be read by the interfacer every 10 seconds.
What I also considered doing is to build a separate interfacer that picks up the gas value from the telegram every hour. But then I think this will not work either as actually it might be the EmoncmsHTTPInterfacer that needs to be split into two, one for the 10 second data and one for the hourly data upload. Is my thinking correct here? Or can I indeed add another serial interfacer reading the gas meter data on an hourly basis (how do I take care of the conflicts of the serial port?). Or would it be better to actually run a completely different python process on a different thread?
Did you notice my post no. 15 above?
It would be comparatively simple for the front end processor to send a radio packet on a new NodeID on a change, then your emonCMS would react to that new message. The hard part would be putting that code in, and maintaining it in the face of updates.
Hi Robert, you are right, I seem to have overlooked it. Or rather, when I read it I did not fully understand it yet. Now I am beginning to. What’s a radio packet in the context of emoncms?
Another, even more elegant solution could IMHO be to add the kWh_to_power function to the Emoncms post processor. The post processor operates on feeds rather than inputs, and I do have a feed containing hourly data points of the gas meter as a result of the log to feed using hourly intervals.
How would I go about adding this process to Emoncms?
I’m not an expert in emonCMS - so someone else needs to answer that.
Simply, the message that’s transmitted to convey the data. It’s sent by radio from an emonTx to the emonPi/emonBase, from there it’s a wired serial connection into the Raspberry Pi. The emonPi’s own energy data joins it on the wired connection into the Pi.
Whatever the source, you need to make up a new message that pretends to come from a different node - from there on, you can treat it entirely separately inside emonCMS, and I can’t see that any changes (other than configuration) will be needed in emonCMS.
Is the gas meter wired into the pulse input of the emonPi or an emonTx? - the top post doesn’t mention this.
Here in the Netherlands, the gas meter is connected via a short cable to the data infrastructure of the electricity meter, and I have no idea how that part works. The electricity meter makes a data message (a text-based telegram) including the gas meter counter available via an S0 serial interface every 10 seconds. A serial_to_USB cable connects the S0 RJ11 jack of the meter to the USB port of the RPi and my EmonHub interfacer reads it from there.
I suppose I could try to create a second interfacer running hourly, just to pick up the gas message once per hour to deliver it to a different node, as you suggest.
Do you think I should open a new thread on how to add a new post process to EmonCMS?
Ah, that’s something else that we were not aware of. In that case, my suggestion for processing the pulses in the emonPi’s front end can never work, because you never see the actual pulse from the meter.
I have never needed to find out in detail how the interfacers work. Paul created emonHub but even though others have changed things since, he is still probably the best person to tell you what might work and what might not. As I understand it now, in essence your problem is the interfacers as they exist process and send the data onwards when a message arrives. What you seem to need is an interfacer that picks out the gas data, notices a change in the count, and only then sends the data onwards, probably under a new NodeID.
My personal view is don’t split the discussion into two threads. If you do, the second will only be the solution to the first, and that will confuse everybody. If it becomes sensible and feasible to split the “new interfacer” discussion out, that can be done later.
Ok, the fact that this device works that way, would seem, to me, a good reason to make the interfacer for that device accommodates that. It should be relatively easy to make the interfacer pass everything but the gas reading on every payload except for a single hourly payload that has all including the gas reading.
This could be done by counting the payloads and every 360th gets the gas reading or every 3600 secs since start of emonhub or on the hour or first payload thereafter or you could add a offset in seconds to the settings so that (for example) a setting of “30” would send the gas reading on every payload received at 30secs past the hour, this seems like over kill but it would accommodate the fact you said the reading are passed around 11 secs past the hour, without delaying the reading or adding either a randomness or fixed offset that may need adjusting at some point, especially if someone else uses your code.
Many moons ago, we made emoncms ready to accept “null” values in preparation for emonhub to pass null values when needed but I never got to implement this before Trystan took over. AFAIA you would need to rearrange your payload so that the gas reading is on the end so that both the short (no gas) and full (with gas) payloads are valid if using the bulk upload http interfacer, even if you are not, I would still recommend this approach to avoid a known problem popping up down the line should you (or someone else) choose to use the http interfacer eg to post to emoncms.org.
I’m not convinced this would work well since if there are no zero’s recorded (eg over night) then when there is a change (eg in the morning) the assumption will be that the use recorded in the last hour, is actually the amount used since the last valid (ie changed) value (eg the night before).
True enough - I was concerned about synchronism between the smart meter timing and emonHub processing the data, possibly leading to the result being an hour old.
Ok, I have looked at the emonhub code. The challenge for me is that my implementation here is already 6 years old, with the only update having taken place last year (after the SD card collapsed) and I am no real coder on top of that.
Therefore I would prefer not to fiddle with the emohub code if it can be avoided. It seems to be neater to me to add in the config file a second instance of my serial interfacer only that one running hourly, and saving the data to a different node. That should be easy enough. The only possible problem I can see is that the two interfacers might clash when they try to access the serial port at the same time.
Could this work? What do you guys think?
We don’t really know anything about your serial device/connection, but generally, and this is true of all the serially connected emon devices, the serial connection must be monogamous, 3 is a crowd and will cause issues, if not immediately, then soon.
The only real alternative is to do it externally in nodered or a custom script, but they involve some coding too, maybe less so with nodered but that’s a lot of overhead and work installing and running just for one task.
3 posts were split to a new topic: Emonhub and use of ThreadLock
Ok, I think I’ll try that one now. I did not realise that I can send 10s data as well as hourly data on the same node. If it works this will make most sense.